# Benchable Store and visualize arbitrary benchmarks — latency, build time, bundle size — with statistically tested verdicts, traces and budgets. Self-hostable. > Benchmarks in, verdicts out — currently v0.1.0. Benchable stores and visualizes arbitrary benchmarks — latency, build time, bundle size, throughput, token cost — plus the traces behind them. Metrics are never predeclared: the first run that mentions `api.latency_ms` creates it, infers `ms` and lower-is-better, and starts charting. Built on [Next.js](https://nextjs.org), [Postgres](https://www.postgresql.org) and Drizzle, and made by [linesofcode](https://x.com/linesofcode). ## Post a run, get the verdict Send metrics to `POST /api/v1/runs` with an `Idempotency-Key`, and the response names every metric that regressed — `X-Benchable-Regressions` lets CI fail the build without a second request. A metric is a bare number or an object with `value` and any percentiles; the first run that mentions a key creates it. Learn more in [sending a run](/reference#sending-a-run). ```sh curl -X POST http://localhost:3000/api/v1/runs \ -H "Authorization: Bearer $BENCHABLE_KEY" \ -H "Content-Type: application/json" \ -H "Idempotency-Key: $GITHUB_RUN_ID" \ -d '{ "branch": "main", "commitSha": "9f3c1ab", "environment": "ci-linux-x64", "metadata": { "runner": "github-actions", "node": "24" }, "metrics": { "build.time_ms": 4210, "api.latency_ms": { "value": 118.4, "p50": 110, "p95": 180, "p99": 260, "samples": 500 } }, "spans": [ { "id": "root", "name": "POST /checkout", "startMs": 0, "durationMs": 118.4 }, { "id": "db", "parentId": "root", "name": "SELECT orders", "startMs": 12, "durationMs": 40 } ] }' ``` ```json { "runId": "fhTNzgoqngNdETKAPYLkC", "branch": "main", "format": "benchable", "idempotent": false, "regressions": 1, "metrics": [ { "key": "api.latency_ms", "value": 210, "unit": "ms", "baseline": 120.6, "delta": 89.4, "deltaPct": 74.13, "verdict": "regressed" } ] } ``` ## The change has to be real When both runs carry a distribution the verdict comes from Welch's t-test rather than arithmetic on two means; send the raw `values` and it becomes Mann–Whitney U. Evidence can only ever downgrade a verdict, so nothing that gated your CI yesterday stops gating it today. Learn more in [is the change real?](/reference#is-the-change-real). ```json { "key": "api.latency_ms", "value": 106, "baseline": 100, "deltaPct": 6, "verdict": "neutral", "reason": "not significant (p = 0.287)", "significance": { "test": "welch", "p": 0.287, "effectSize": 0.52, "interval": { "low": -5.9, "high": 17.9, "level": 0.95 }, "significant": false } } ``` - metrics tested: 312 - flagged before control: 15 - flagged after: 2 ```json "fdr": { "applied": true, "tested": 312, "flaggedBefore": 15, "flaggedAfter": 2, "q": 0.05 } ``` ## Noise bands measure themselves Every metric carries a `stability profile` computed from its own history on the default branch, shown in Settings beside the configured band. Switch a band to **Measured** and the verdict uses the measured number instead of the ±5% default, floored at 0.5%. Learn more in [noise bands](/reference#noise-bands-the-metric-measures-for-itself). - api.latency_ms, measured noise: ±9.4% - runs of history: 46 - cold_start.p99_ms, worst move: ±10.5% ``` GET /api/v1/metrics -H 'Accept: text/plain' key name unit better noise band stability -------------------- ---------------- ---- ------ ---------- --------------------------------------- api.latency_ms Api Latency ms lower ±9.4% auto noisy, run-to-run noise ±9.4% over 46 runs bundle.main_kb Bundle Main KB lower ±5% stable, run-to-run noise ±0.0% over 40 runs cold_start.p99_ms Cold Start P99 ms lower ±5% flaky — too unstable to gate on, ±10.5% ``` ## Baselines compare like with like A baseline is searched from the narrowest scope outwards — same branch and environment first — and `baselineScope` in the response says which scope it landed on. A cross-environment comparison is still made, but labelled, so a hardware difference is never read as a regression. Learn more in [baselines](/reference#baselines-that-compare-like-with-like). | `baselineScope` | Meaning | | --- | --- | | `branch+environment` | Same branch, same environment — a like-for-like comparison | | `branch` | Same branch, another environment | | `default-branch+environment` | The default branch, same environment | | `default-branch` | The default branch, another environment | ## Import your tool's output Post the file your benchmark already writes to `POST /api/v1/import?format=auto` — no translation shim, the format is detected from the content. Metric keys are namespaced by adapter, so two tools never collide. Learn more in [importing your tool's output](/reference#importing-your-tools-output). ```sh go test -bench=. -benchmem ./... > bench.txt curl -X POST "$BENCHABLE_URL/api/v1/import?branch=main&commitSha=$(git rev-parse HEAD)" \ -H "Authorization: Bearer $BENCHABLE_KEY" \ -H "Accept: text/plain" \ --data-binary @bench.txt ``` ``` run WJ6S2WQYUr8NCdyWUSvLH on main (go-bench) metric value baseline change -------------------------------- ------------ -------- --------- go.BenchmarkEncode.ns_per_op 1.05µs — first run go.BenchmarkEncode.B_per_op 512 B — first run go.BenchmarkStream.MB_per_s 452 MB/s — first run No regressions. ``` | `format` | Produce it with | | --- | --- | | `benchable` | The native JSON payload | | `go-bench` | `go test -bench=. -benchmem ./...` | | `hyperfine` | `hyperfine --export-json bench.json './build.sh'` | | `pytest-benchmark` | `pytest --benchmark-json=bench.json` | | `google-benchmark` | `./bench --benchmark_format=json --benchmark_repetitions=5` | | `criterion` | `cargo criterion --message-format=json` | | `vitest-bench` | `vitest bench --outputJson=bench.json` | | `k6` | `k6 run --summary-export=summary.json script.js` | | `lighthouse` | `lighthouse https://example.com --output=json` | | `jmh` | `java -jar benchmarks.jar -rf json -rff bench.json` | | `prometheus` | `curl -s http://localhost:9090/metrics` | | `csv` | `name,value[,unit[,lower\ | higher]]` | | `otlp-trace` | OTLP/JSON `ResourceSpans` — becomes a trace waterfall | ## One regression, one record The first detection opens a record and that is the only time anyone is told; later detections bump `occurrences` silently, so one ongoing regression is one row in the queue. `wontfix` and `flaky` suppress a metric until someone reopens it. Learn more in [regressions are states, not events](/reference#regressions-are-states-not-events). ``` GET /api/v1/regressions -H 'Accept: text/plain' id metric status kind branch now was change seen since -------------------- ------------------ ------ ---------- ------ ----- ----- ------- ---- ---------- kf3ZcUxhw3I3EKIcBKJI api.latency_ms open regression main 155ms 100ms +55.0% 3x 2026-05-02 llT2cPh0mA26KPed4cQz bundle.main_kb open budget main 310KB 240KB +29.2% 7x 2026-04-28 ``` ## Which span moved Attach `spans` to a run and `GET /api/v1/compare` ranks the two traces by the change in each span's own work — the thing that attributes a regression. Spans are matched by path, not id, so an N+1 lines up one query for one. Learn more in [which span moved](/reference#which-span-moved). ``` trace 118.0ms → 196.0ms span self total change state ------------------ ------- ------- ------- ------- SELECT orders +78.0ms 118.0ms +195.0% changed INSERT audit_log — 12.0ms — added POST /checkout −12.0ms 196.0ms +66.1% changed Ranked by change in the span's own work, which is what attributes a regression. ``` ## Budgets a baseline cannot see Set `budgetMax` or `budgetMin` per metric in Settings and every run is checked against the absolute line — the two failures a relative comparison is blind to. The CLI exits `1` on a budget failure even when nothing regressed since yesterday. Learn more in [performance budgets](/reference#performance-budgets). ``` metric value baseline change budget -------------------- ------ -------- --------- ---------------- budget.bundle_kb 251 KB 240 KB · 4.58% over by 0% api.latency_ms 118ms 120ms · −1.67% 32% left ``` ## Find where it changed level `GET /api/v1/metrics/{key}/changepoints` runs binary segmentation over the series and returns each step with the commit range that contains the cause. Three percent a week never trips a 5% band — this is what catches it. Learn more in [change points](/reference#change-points). ``` search.throughput_ops on main — 40 runs analysed when level change from to commit range p ---------- --------- ------ ------------ ------------ ---------------- ------ 2026-09-10 regressed -6.30% 23.67k ops/s 22.18k ops/s aaaaaaa..aaa0aaa <0.001 2026-09-14 regressed -5.63% 22.32k ops/s 21.07k ops/s aaaf1da..a2aafab 0.002 The cause of each change is in the commits inside its range. ``` ## Agents read it in one fetch An MCP server at `/api/mcp` takes the same project key, and every tool answers in text rather than JSON. Every read endpoint honours `Accept: text/plain`, and `/llms.txt` describes the whole API in one fetch. Learn more in [agents](/reference#agents). ```json { "mcpServers": { "benchable": { "url": "https://benchable.example.com/api/mcp", "headers": { "Authorization": "Bearer bmk_..." } } } } ``` ## What holds, what is a guess Holds: - Evidence can only downgrade a verdict — nothing that gated CI yesterday stops gating today unless the data says the signal was never there. - Idempotent ingest: a retried CI step reuses its key and gets the original run back, verdicts included. - API keys are stored as SHA-256 hashes, and artifacts are never public. Judgements: - The ±5% default noise band is a guess nobody would set by hand — switch the metric to Measured. - False-discovery control runs at q = 0.05, the field default, not a tuned choice. - Stability needs six runs of history before it is reported — the line where guessing would start. Not here yet: - AI diagnosis needs an `AI_GATEWAY_API_KEY`; without it the endpoint answers `503 ai_unavailable` rather than pretending. - Benchable does not run benchmarks on a schedule — it ingests results, and CI owns the cadence. - Cross-environment baselines are labelled, not corrected — hardware is scoped, never normalized. ## Run it, then send one One Postgres URL and it runs anywhere Bun does; the seed script prints a demo workspace and an API key. The first run you send creates every metric in it — read [the README](/reference) for the rest. ```sh bun install cp .env.example .env # set DATABASE_URL and BETTER_AUTH_SECRET bun run db:migrate bun run db:seed # optional: a demo workspace with 54 runs across 3 branches bun run dev # picks the first free port at or after 3000 ``` ```sh export BENCHABLE_URL=https://benchable.example.com export BENCHABLE_KEY=bmk_... bunx benchable import --file bench.txt --branch main --commit "$(git rev-parse HEAD)" bunx benchable submit --metrics metrics.json --branch main bunx benchable summary bunx benchable report --run last --marker pr-42 # Markdown for a PR comment bunx benchable regressions --status live # the regression queue bunx benchable upload --run last --file flame.svg --kind flamegraph bunx benchable artifacts --run last bunx benchable download --run last --name flame.svg bunx benchable formats ``` ## Links - Full reference: https://benchable.dev/reference - Repository: https://github.com/TimMikeladze/benchmarker - Pricing: https://benchable.dev/pricing Benchable is a linesofcode project — self-hostable, Postgres, Bun and one build command. Every number on this page is read out of the README at render time, so the page breaks when the behaviour does. © 2026 linesofcode