Reproducible Results
Benchmarks You Can Reproduce
Internal performance results with reproducible methodology. Every number below comes from a public workload. Run them yourself and verify.
Choose the workload path that matches your evaluation
Start with the methodology, then choose a workload path that matches your evaluation: inference, developer quickstart, or team pricing.
Reproducible Results
These benchmarks use our open methodology. Run them yourself: pip install epochly && python -m epochly.benchmark
JIT compilation (Level 2)
GPU acceleration (Level 4)
Overhead when not helping
GPU example: 100M-element array operation: 1,427ms → 21ms (68x)
Inference: per-request overhead and opt-in micro-batching
These are the measured inference results behind the homepage stat block, so “validated on real hardware, reproducible methodology” is true for inference too. Every figure below is cited exactly — medians and p95, no rounding — from the frozen measurement harness.
Per-request wrap overhead, every backend
Throughput via opt-in micro-batching (RTX 4070)
CPU cell
macOS arm64 (Apple silicon), CPU only, Python 3.11
torch 2.12.1 (CPU), onnxruntime 1.27.0, onnx 1.22.0, sklearn 1.7.2
GPU cell
NVIDIA GeForce RTX 4070 Laptop GPU, CUDA, Python 3.13
torch 2.12.1+cu130
Per-request overhead (N=200, 3 reps, warmup discarded)
| Backend / model | Raw median | Overhead median | Overhead p95 |
|---|---|---|---|
| torch_cpu small (~14us) | 0.0137 ms | +0.0010 ms | +0.0010 ms |
| torch_cpu big (15.34 ms) | 15.34 ms | -0.0184 ms | +0.0665 ms |
| onnx small (~5us) | 0.0050 ms | +0.0014 ms | +0.0016 ms |
| onnx big (18.15 ms) | 18.15 ms | -0.0674 ms | -0.3176 ms |
| generic (memo route) small (~16us) | 0.0164 ms | +0.0272 ms | +0.0288 ms |
| generic (memo route) big (11.00 ms) | 11.00 ms | +0.5880 ms | +0.5366 ms |
| torch_cuda small (~32us) | 0.0323 ms | +0.0013 ms | +0.0014 ms |
| torch_cuda big (17.24 ms) | 17.24 ms | +0.0561 ms | -0.1194 ms |
Every cell is under 1 ms absolute on median and p95, on both CPU and CUDA (torch_cpu +0.0010 ms, onnx +0.0014 ms, torch_cuda +0.0013 ms). The framework routes are a zero-overhead passthrough; the generic row is the sklearn memoization route measured on its worst-case cache-miss path.
Opt-in micro-batching throughput vs sequential (RTX 4070)
| Model | Sequential | Batch 4 | Batch 8 | Batch 16 | Batch 32 |
|---|---|---|---|---|---|
| launchbound_mlp (compute-light) | 16,250 req/s | 2.28x | 3.97x | 6.08x | 8.62x |
| membound_mlp (memory-bound) | 458 req/s | 3.84x | 7.66x | 14.71x | 28.87x |
At least 2.28x in every measured (model, batch-size) cell, up to 28.87x for the memory-bound model at batch 32. These are figures for the standalone, opt-in DynamicMicroBatcher component.
How to read these numbers
- Framework route is a passthrough proxy. The torch / onnx sub-millisecond overhead is not a speedup — it is the cost of a transparent passthrough that applies no optimization on that route today. Near-zero overhead is by design, not evidence of an inference speedup.
- Micro-batching is opt-in, not the wrap() path. The 2.28–28.87x figures are for an explicitly constructed
DynamicMicroBatcher. The two-lineepochly.wrap()call passes framework models through with zero added overhead and does not batch automatically. - No end-to-end GPU cost-reduction claim. A workload-level GPU cost reduction has not been measured and is not claimed here. The figures above are per-request overhead and component throughput, not a cost percentage.
Reproduce the inference benchmark
The inference suite reuses the same frozen probes that produced the canonical numbers, so the results reproduce rather than drift. One documented command per cell:
# CPU cells (any host with torch(CPU) + onnxruntime + sklearn)$ PYTHONPATH=src EPOCHLY_OFFLINE_MODE=1 python benchmarks/inference/run_inference_benchmarks.py --cpu# GPU cells (CUDA device, e.g. RTX 4070)$ PYTHONPATH=src EPOCHLY_OFFLINE_MODE=1 python benchmarks/inference/run_inference_benchmarks.py --gpu
Full methodology, the environment table, the traceability table (every number to its source JSON), and the mandatory scope caveats live in benchmarks/reports/INFERENCE_BENCHMARK_2026-07.md. A blocked GPU cell (no CUDA device) exits 2 and cites the archived RTX 4070 numbers — blocked is not failed.
Detailed Results
| Workload | Baseline | With Epochly | Speedup | Level |
|---|---|---|---|---|
| JIT polynomial evaluation (1M iterations) | 324.16ms | 1.68ms | 193x | Level 2 |
| JIT numerical loop (1M iterations) | 101.25ms | 1.15ms | 88x | Level 2 |
| JIT nested loop (10K iterations) | 66.54ms | 1.15ms | 58x | Level 2 |
| GPU elementwise ops (100M elements) | 1,427ms | 21ms | 68x | Level 4 |
| GPU reduction (100M elements) | 59ms | 1.6ms | 36x | Level 4 |
| GPU convolution (batch=16, 224x224) | 148ms | 7.6ms | 19x | Level 4 |
| GPU matrix multiply (4096x4096) | 200ms | 29ms | 7x | Level 4 |
| Parallel heavy CPU (16 cores) | 1,396ms | 166ms | 8x | Level 3 |
| Monte Carlo simulation (100M samples) | 3,639ms | 504ms | 7x | Level 3 |
| Unsuitable workload | measured | measured | ~1.0x | Disabled |
Test Environment
CPU
16-core CPU (all cores utilized for parallel benchmarks)
GPU
NVIDIA Quadro M6000 24GB, CUDA 12.1
Software
Linux WSL2 (x86_64)
Python 3.12.3
PyTorch 2.5.1+cu121, NumPy 1.26.4
Reproduce the Benchmark
We publish our benchmark suite so you can verify our results on your own hardware. Clone the repository and run the benchmarks yourself.
# Clone the benchmark repository$ git clone https://github.com/epochly/benchmarks$ cd benchmarks# Set up the environment$ python -m venv venv$ source venv/bin/activate$ pip install -r requirements.txt# Run all benchmarks$ python run_benchmarks.py --all# Run specific benchmark$ python run_benchmarks.py --workload numpy_matrix
Methodology
- 1Each benchmark runs 10 iterations with the first 2 discarded as warm-up
- 2Results show median execution time to minimize outlier impact
- 3Baseline measurements use vanilla Python with no optimization packages
- 4GPU benchmarks (Level 4) use PyTorch on CUDA. Parallel benchmarks (Level 3) use ProcessPoolExecutor with all cores. Pure NumPy is intentionally not intercepted (~1.0x) because it already uses optimized C code.
- 5System is idle during benchmarks with no other CPU-intensive processes