Benchable — reference

The README, in full. The landing page demonstrates pieces of it; this page is the source. Back to the landing page.

Benchable

Store and visualize arbitrary benchmarks — latency, build time, bundle size, throughput, token cost — plus the traces behind them. Works for one person or a team.

Metrics are never predeclared. The first run that mentions api.latency_ms creates it, infers ms and lower-is-better from the key, and starts charting it.

What it does

• Takes your tool's own output. 13 ingest formats, auto-detected: Go benchmarks,

hyperfine, pytest-benchmark, Google Benchmark, Criterion, Vitest/tinybench, k6, Lighthouse, JMH, Prometheus, CSV, OpenTelemetry traces, and the native JSON payload.

• Baselines without config. Each metric is compared to the newest earlier run on its

branch that reported that metric, falling back to the default branch — so interleaving two benchmark suites on one branch does not break comparisons. The ingest response says what regressed, so CI can fail the build without a second request.

• Verdicts backed by statistics. Where the producer sent a distribution, a change is put

through Welch's t-test (or Mann–Whitney U over raw samples) before it is called a regression, and every metric measures its own noise band from its history.

• Idempotent ingest. A retried CI step reuses its key and gets the original run back.
• Traces beside the numbers. Attach spans to a run, read them as a waterfall, and diff

them between runs to see which span actually moved.

• Agents first-class. An MCP server, an llms.txt, and a plain-text digest on every read

endpoint.

• Optional AI diagnosis. Given the metrics that moved, the ones that did not, the trace

and the run metadata, it proposes ranked causes with the evidence for each.

• Discussion on the results. Comment threads anchored to a run, a benchmark, a single

measurement, or the project. CI and agents can post too, and are labelled as automated.

• Regressions with a lifecycle. One record per problem — open, acknowledged, resolved,

won't fix, flaky — so an ongoing regression is one row saying seen 14× rather than fourteen alerts.

• Webhooks, badges, retention. Signed JSON or Slack blocks when a regression opens or

recovers; a README badge; a nightly job that applies each project's retention.

• Scheduled reports. Tell a project how often it expects a run and it says so when the job

stops arriving — a dead benchmark job otherwise looks exactly like a healthy one. A daily or weekly digest says where the project stands rather than what the last run did.

• Share links. One opt-in, revocable URL publishes a project's results to anyone without

an account — the same charts and verdicts, minus the comments, diagnoses and artifacts.

• Suites and a health score. Metrics group themselves by namespace, and one number says

whether the project is getting faster or slower — from budgets, open regressions and drift, not from the last run's arrows.

• Teams or solo. Sign-up creates your own workspace; inviting someone makes it a team.

Running it

bun install
cp .env.example .env     # set DATABASE_URL and BETTER_AUTH_SECRET
bun run db:migrate
bun run db:seed          # optional: a demo workspace with 54 runs across 3 branches
bun run dev              # picks the first free port at or after 3000

bun run db:seed prints the demo sign-in and an API key.

Environment

The environment is validated at boot, so a missing variable fails the deploy with a readable message rather than surfacing as undefined on the first request.

Billing

Three plans, defined in one file — lib/billing/plans.ts holds every limit, feature flag and price, and nothing else in the codebase hardcodes a number. Free is the whole product with volume caps; Pro is $20 per member per month through Stripe Checkout; Enterprise is a conversation. The design and the enforcement points are in [docs/billing.md](docs/billing.md).

STRIPE_SECRET_KEY=sk_test_... bun run stripe:setup     # creates the products, prints the env vars
stripe listen --forward-to localhost:3000/api/stripe/webhook

**Without STRIPE_SECRET_KEY and STRIPE_PRO_PRICE_ID, billing is off entirely and every workspace is unlimited.** That is the intended state for a self-hosted deployment and for the test suite — a self-hosted Benchable is not a crippled one.

When billing is on, a refused action returns 402 plan_limit_exceeded from the API, with the limit and the current value in details, and a plain sentence from the UI.

Deploying

vercel.ts sets the build command to bun run db:migrate && next build, so every deploy applies pending migrations against that environment's DATABASE_URL before the build. A schema change ships with its code, and a build fails loudly rather than leaving the running app querying tables that do not exist yet.

Sending a run

Create an API key under a project's Settings tab, then:

curl -X POST http://localhost:3000/api/v1/runs \
  -H "Authorization: Bearer $BENCHABLE_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $GITHUB_RUN_ID" \
  -d '{
    "branch": "main",
    "commitSha": "9f3c1ab",
    "environment": "ci-linux-x64",
    "metadata": { "runner": "github-actions", "node": "24" },
    "metrics": {
      "build.time_ms": 4210,
      "api.latency_ms": { "value": 118.4, "p50": 110, "p95": 180, "p99": 260, "samples": 500 }
    },
    "spans": [
      { "id": "root", "name": "POST /checkout", "startMs": 0, "durationMs": 118.4 },
      { "id": "db", "parentId": "root", "name": "SELECT orders", "startMs": 12, "durationMs": 40 }
    ]
  }'

A metric is either a bare number or an object with value plus any of unit, direction (lower | higher), name, samples, min, max, mean, p50, p95, p99, stddev, values and labels.

**Send values if you have them.** The raw sample vector buys three things a summary cannot: the comparison uses Mann–Whitney U instead of a t-test, the percentiles are derived consistently rather than being each tool's idea of what p95 means, and the run page draws the distribution against its baseline — which is the only way to see that a tail got heavier while the median held still.

"metrics": { "api.latency_ms": { "value": 118.4, "values": [117.9, 118.2, 118.1, 402.7, …] } }

Up to 50,000 samples per metric are accepted; 2,000 are stored, evenly spaced so the tail survives. samples still reports the real iteration count.

The response classifies every metric against its baseline and carries X-Benchable-Regressions, so CI can gate on a header rather than parsing a body:

{
  "runId": "fhTNzgoqngNdETKAPYLkC",
  "branch": "main",
  "format": "benchable",
  "idempotent": false,
  "regressions": 1,
  "metrics": [
    { "key": "api.latency_ms", "value": 210, "unit": "ms",
      "baseline": 120.6, "delta": 89.4, "deltaPct": 74.13, "verdict": "regressed" }
  ]
}

verdict is improved, regressed or neutral. A change smaller than the metric's noise band (5% by default, editable per metric) is neutral. Direction is honoured, so rising throughput is an improvement while rising latency is a regression. The baseline is resolved per metric — the newest earlier run on the branch that reported *that* metric — so a project that posts a latency suite and a bundle suite as separate runs still compares like with like.

Is the change real?

A percent threshold alone calls a 6% move on a metric that swings ±12% a regression, which is how teams learn to ignore their own alerts. When a run and its baseline both carry a distribution — stddev and samples, which hyperfine, JMH, pytest-benchmark, Google Benchmark and Criterion all report — the verdict is decided by Welch's t-test rather than by arithmetic on two means. Send values (the raw sample vector) and it uses Mann–Whitney U instead, which assumes no distribution at all — the right test for the skewed, long-tailed timings micro-benchmarks produce.

{ "key": "api.latency_ms", "value": 106, "baseline": 100, "deltaPct": 6,
  "verdict": "neutral", "reason": "not significant (p = 0.287)",
  "significance": { "test": "welch", "p": 0.287, "effectSize": 0.52,
                    "interval": { "low": -5.9, "high": 17.9, "level": 0.95 },
                    "significant": false } }

The order is deliberate, and evidence can only ever *downgrade* a verdict:

1. No baseline → neutral.
2. Smaller than the noise band → neutral. A detectable 0.4% move is still not worth an alert.
3. Both sides have a distribution → the test decides, and p, the effect size and a 95%

confidence interval on the difference ride along.

4. No distribution → percent change against the band, exactly as before.

So nothing that gated your CI yesterday stops gating it today unless the data says the signal was never there.

Noise bands the metric measures for itself

Nobody can guess a good threshold, so nobody revisits the 5% default: a bundle-size metric that never varies by 0.1% needs a 50× move before it says anything, and a cold-start metric that swings ±15% on a shared runner cries wolf on most runs.

Every metric therefore carries a stability profile computed from its own history on the default branch — the run-to-run noise (two robust sigmas of successive percent deltas, so a metric on a clean trend reads as stable rather than as noise), the coefficient of variation, the worst single move, and a stable / noisy / flaky label. Settings shows it next to the configured band and says which way it is wrong:

GET /api/v1/metrics -H 'Accept: text/plain'

key                    name              unit  better  noise band  stability
--------------------  ----------------  ----  ------  ----------  ---------------------------------------
api.latency_ms        Api Latency       ms    lower   ±9.4% auto  noisy, run-to-run noise ±9.4% over 46 runs
bundle.main_kb        Bundle Main       KB    lower   ±5%         stable, run-to-run noise ±0.0% over 40 runs
cold_start.p99_ms     Cold Start P99    ms    lower   ±5%         flaky — too unstable to gate on, ±10.5%

Switch a metric's band to Measured and the verdict uses the measured number instead of the setting, floored at 0.5%. A flaky metric is labelled rather than silently gating your builds on its own jitter.

Deltas below six runs of history say unknown rather than guessing from three points.

Baselines that compare like with like

Hardware dominates every other variable in a benchmark — a laptop run and a CI Linux run of the same commit differ by more than most regressions do. A baseline is therefore searched from the narrowest comparison outwards, and the response says which scope it landed on:

A cross-environment comparison is still made — a rough baseline beats none — but it is labelled in the response, in the text digest and on the run page, so a hardware difference is never read as a regression. A run that reports no environment is compared only against other runs that report none.

Large suites: false-discovery control

A p-value of 0.05 means one test in twenty fires when nothing happened. A suite of 300 metrics therefore produces about fifteen "regressions" on a run where nobody changed anything — every run, forever — and the team learns to ignore all of them.

When a run has at least 25 metrics whose verdicts came from a test, those p-values go through Benjamini–Hochberg at q = 0.05. Anything that does not survive is downgraded to neutral with did not survive false-discovery control (adjusted p = 0.31 at q = 0.05), and the response reports both counts so nothing is hidden:

"fdr": { "applied": true, "tested": 312, "flaggedBefore": 15, "flaggedAfter": 2, "q": 0.05 }

Control only ever removes a flag. A metric decided by its noise band alone was never a hypothesis test and is left alone.

Performance budgets

Everything above answers *did this change?*. A budget answers the question a baseline structurally cannot: this number is allowed to be at most this, whatever it was yesterday.

Two failures are invisible to relative comparison and obvious to a budget. Twenty runs each 2% slower than the last: every verdict neutral, the bundle now 40% over what you can ship. And a regression that lands on a fast runner, where the delta looks small but the absolute number is past the line that matters.

Set budgetMax (at most) or budgetMin (at least) per metric in Settings, in the metric's own unit. A budget is then checked on every run:

metric                 value  baseline     change            budget
--------------------  ------  --------  ---------  ----------------
budget.bundle_kb      251 KB    240 KB     · 4.58%       over by 0%
api.latency_ms         118ms     120ms     · −1.67%      32% left

• The response carries budgetFailures and an X-Benchable-Budget-Failures header, and the

CLI exits 1 on either a regression or a budget failure — a build past the line is broken even when nothing regressed since yesterday.

• A violation reaches a webhook that only asked for regressions, as event run.over_budget.
• The badge turns red and says over budget, and amber within 10% of the limit.
• A metric with no budget reports —. Unbudgeted is not the same as passing.

Change points

Run-to-run comparison answers one question — *did this commit regress?* — and is blind to the two failures that cost the most. Three percent a week never trips a 5% band and doubles latency in six months. A real 12% step on a metric that swings ±10% lands as one alarming run followed by "recovered" runs that are actually the new normal.

GET /api/v1/metrics/{key}/changepoints runs binary segmentation over the series: the split that best separates a window into two levels, kept if it is significant (p < 0.01) and larger than the metric's noise band, then recursed into both halves.

search.throughput_ops on main — 40 runs analysed

when            level  change          from            to      commit range       p
----------  ---------  ------  ------------  ------------  ----------------  ------
2026-09-10  regressed  -6.30%  23.67k ops/s  22.18k ops/s  aaaaaaa..aaa0aaa  <0.001
2026-09-14  regressed  -5.63%  22.32k ops/s  21.07k ops/s  aaaf1da..a2aafab   0.002

The cause of each change is in the commits inside its range.

The commit range is the point: git log aaaaaaa..aaa0aaa is the set of changes that contains the cause. The metric page marks each change point on the chart and links its range straight into the comparison view, and the detect_change_points MCP tool gives an agent the same answer. One branch and one environment at a time — a series that mixes hardware steps every time the fleet changed, which is true and useless.

Idempotency. Send Idempotency-Key (or idempotencyKey in the body). A repeat returns the original run with 200 and "idempotent": true, including the same verdicts — a retried CI step gates the same way twice.

Importing your tool's output

Don't write a translation shim. Post the file:

go test -bench=. -benchmem ./... > bench.txt

curl -X POST "$BENCHABLE_URL/api/v1/import?branch=main&commitSha=$(git rev-parse HEAD)" \
  -H "Authorization: Bearer $BENCHABLE_KEY" \
  -H "Accept: text/plain" \
  --data-binary @bench.txt

run WJ6S2WQYUr8NCdyWUSvLH on main (go-bench)

metric                                   value  baseline     change
--------------------------------  ------------  --------  ---------
go.BenchmarkEncode.ns_per_op            1.05µs         —  first run
go.BenchmarkEncode.B_per_op              512 B         —  first run
go.BenchmarkStream.MB_per_s           452 MB/s         —  first run

No regressions.

?format=auto is the default. Pass an explicit ?format=go-bench to skip detection and get a real parse error instead of a silent fall-through. GET /api/v1/import lists the formats.

Metric keys are namespaced by adapter, so two tools never collide: go.BenchmarkEncode.ns_per_op, hyperfine.build.mean_ms, lighthouse.performance.score.

CLI

export BENCHABLE_URL=https://benchable.example.com
export BENCHABLE_KEY=bmk_...

bunx benchable import  --file bench.txt --branch main --commit "$(git rev-parse HEAD)"
bunx benchable submit  --metrics metrics.json --branch main
bunx benchable summary
bunx benchable report --run last --marker pr-42   # Markdown for a PR comment
bunx benchable regressions --status live          # the regression queue
bunx benchable upload --run last --file flame.svg --kind flamegraph
bunx benchable artifacts --run last
bunx benchable download --run last --name flame.svg
bunx benchable formats

Exit codes: 0 clean, 1 a metric regressed, 2 an error. Both --file and --metrics accept - for stdin.

See examples/github-actions.yml for a workflow that posts a run and fails on a regression.

Artifacts

A number tells you something regressed. It never tells you why. Attach the thing that does:

bunx benchable upload --run last --file flame.svg --kind flamegraph
bunx benchable upload --run last --file lighthouse.html
bunx benchable upload --run last --file heap.heapsnapshot

# or without the CLI — the body is the raw file, so there is no multipart encoder to install
curl -X POST "$BENCHABLE_URL/api/v1/runs/$RUN_ID/artifacts?name=flame.svg&kind=flamegraph" \
  -H "Authorization: Bearer $BENCHABLE_KEY" \
  --data-binary @flame.svg

An artifact belongs to one run and is deleted when that run is, so the evidence and the number expire together — a regression from three months ago still has its flamegraph.

kind is inferred from the file name when you leave it out: image, flamegraph, profile, trace, log, report, file. Re-uploading a name replaces that file rather than stacking a second copy, so a retried CI step leaves one flamegraph.

Getting them back out:

bunx benchable artifacts --run last                      # what is attached
bunx benchable download  --run last --name flame.svg     # writes ./flame.svg
bunx benchable download  --run last --name bench.log --out -   # or to stdout

download verifies the sha256 the server recorded at upload and exits non-zero on a mismatch, because a truncated download is otherwise indistinguishable from a small file.

Images show as thumbnails on the run page; everything else is a download. The run's Markdown report links whatever is attached, so the PR comment describing a regression also links the profile of it.

Artifacts are never public. GET /api/v1/artifacts/:id requires the same bearer token as the rest of the API (or a logged-in session, which is how the <img> tags on the run page work). Blob objects are written with private access for the same reason: a public URL is a bearer token that leaks through referrers and pasted links, and a flamegraph of a production service is not something to hand out on a guessable URL.

Where the bytes go depends on configuration:

The Postgres fallback is what makes bun run dev and the tests work against a bare database with no object store. It is not meant for a 200 MB heap snapshot, and the error you get past the cap says so and names the variable that fixes it.

Which span moved

"Checkout got 80ms slower" is where most investigations start and stop. The trace already knows the rest, if the two traces are put side by side. GET /api/v1/compare does that whenever both runs carried one:

trace 118.0ms → 196.0ms

span                   self    total   change    state
------------------  -------  -------  -------  -------
  SELECT orders     +78.0ms  118.0ms  +195.0%  changed
  INSERT audit_log        —   12.0ms        —    added
POST /checkout      −12.0ms  196.0ms   +66.1%  changed

Ranked by change in the span's own work, which is what attributes a regression.

Two decisions make it useful:

• Spans are matched by path, not id. Ids are regenerated every run;

POST /checkout / db / SELECT orders is stable, survives reordering, and repeated siblings get an occurrence suffix so an N+1 lines up one query for one instead of collapsing into a single row.

• Ranking is by self time. Total time propagates up the tree, so ranking by it reports the

root span on every trace where anything at all got slower — which is true, and useless.

The compare page shows the same table, and compare_runs hands it to an agent.

Reports on the pull request

The place a performance result is read is the pull request that caused it, and getting it there has to be one step in CI.

bunx benchable report --run last --marker "benchable-pr-$PR" > report.md
# or: curl "$BENCHABLE_URL/api/v1/runs/$RUN_ID/report?marker=benchable-pr-$PR"

### Benchmarks · payments-api

🔴 **1 regressed** — 41 metrics on `feature/checkout` at `9f3c1ab` · ci-linux-x64

| | Metric | Value | Baseline | Change | Signal | Budget |
|---|---|---:|---:|---:|---:|---:|
| 🔴 | `api.latency_ms` | 150ms | 100ms | +50.00% | p<0.001 | 32% left |
| 🟢 | `build.time_ms` | 3.80s | 4.21s | −9.74% | — | — |

<sub>39 metrics unchanged, hidden.</sub>

It shows what moved and hides what did not — a table of 300 unchanged metrics is how a bot comment gets collapsed and never expanded again. --full overrides that. Open regressions on the project ride along in a collapsed section, so a reviewer sees that this PR's neutral run is landing on top of something that has been broken for a week.

--marker embeds an HTML comment, and the workflow in examples/github-actions.yml uses it to edit its own comment rather than stacking a new one on every push. The endpoint carries the same X-Benchable-Regressions and X-Benchable-Budget-Failures headers as ingest, and benchable report exits 1 on either.

Sharing results

Everything above is behind sign-in. A share link publishes one project read-only, to anyone holding the URL:

https://benchable.example.com/s/bms_2f9c…                  overview, health, every metric
https://benchable.example.com/s/bms_2f9c…/runs             every run, filterable
https://benchable.example.com/s/bms_2f9c…/runs/{runId}     one run's measurements and trace
https://benchable.example.com/s/bms_2f9c…/metrics/{key}    one metric's history

Turn it on in Settings → Share link, copy the URL, paste it into a pull request, an issue, a customer email or a slide. The reader needs no account and can change nothing.

What travels with the link: metrics, suites, the health score, every run, each run's measurements with baselines, verdicts, signal and budget, sample distributions, the trace waterfall, change points and stability.

What does not: comment threads, AI diagnoses, attached artifacts and run metadata — internal, or in the case of metadata, arbitrary JSON your CI supplied. The regressions queue, the compare view and settings are not reachable either.

Two ways to revoke, and they differ:

• Stop sharing — every link dies immediately, but the token is kept, so turning it back on

restores the same URL.

• Rotate — issues a new URL and permanently breaks the old one.

The share token (bms_…) is deliberately not the badge read token (bmr_…). Badge URLs are already pasted into public READMEs on the documented promise that they fetch a badge and nothing else; reusing that token here would have published every project that has a badge.

Agents

Three surfaces, because agents arrive three ways.

MCP server at /api/mcp (Streamable HTTP, stateless), authenticated with the same project API key:

{
  "mcpServers": {
    "benchable": {
      "url": "https://benchable.example.com/api/mcp",
      "headers": { "Authorization": "Bearer bmk_..." }
    }
  }
}

Tools: project_summary, list_metrics, get_metric_series, detect_change_points, list_runs, get_run, compare_runs, list_branches, list_regressions, triage_regression, submit_run, import_run, explain_regression. Every tool answers in text, not JSON — an agent pays per token, and a table is a fraction of the cost.

Plain text. Every read endpoint honours Accept: text/plain or ?format=text.

**/llms.txt** describes the whole API in one fetch.

AI diagnosis

POST /api/v1/runs/{id}/analysis, and a button on the run page.

The useful question is not "summarize these numbers" — the table already does that. It is diagnosis: given which metrics moved, which stayed put, which trace spans moved with them and what the run metadata says, what is the likely cause and what should someone check first. The answer is structured — a headline, a confidence, ranked hypotheses each with its evidence, and concrete next checks — and cached per run, since the inputs are fixed once the run is written.

Without AI_GATEWAY_API_KEY the endpoint answers 503 ai_unavailable and the UI says which variable to set. It never pretends to be broken.

Regressions are states, not events

Comparing consecutive runs re-detects the same regression on every run that follows it, because every one of those runs is still worse than the baseline. Alerting on each detection means the channel fires forever for one problem — so people mute it, and the next real regression arrives in the same colour as the noise.

A regression therefore has an identity. One record per metric, branch and kind:

• the first detection opens it, and that is the only time anyone is told;
• later detections bump occurrences and lastSeenAt, silently;
• a run that brings the metric back to the level it regressed from resolves it, naming the

run that fixed it;

• wontfix (an accepted cost) and flaky (a metric nobody should gate on) suppress that

metric until someone reopens it.

The resolution rule is the subtle one. A record does not close because two consecutive runs stopped differing — otherwise the baseline ratchets up with the regression: run three is no worse than run two, the record closes, and the metric is left permanently 50% slower with nothing open against it. It closes when the value is back within the band of the value it started from.

GET /api/v1/regressions -H 'Accept: text/plain'

id                     metric              status  kind        branch    now    was   change  seen   since
--------------------  ------------------  ------  ----------  ------  -----  -----  -------  ----  ----------
kf3ZcUxhw3I3EKIcBKJI  api.latency_ms      open    regression  main    155ms  100ms   +55.0%    3x  2026-05-02
llT2cPh0mA26KPed4cQz  bundle.main_kb      open    budget      main    310KB  240KB   +29.2%    7x  2026-04-28

The Regressions tab is a work queue: take, acknowledge, resolve, won't fix, flaky. The list_regressions and triage_regression MCP tools give an agent the same queue — and the count in the tab is the number a stand-up actually needs.

Webhooks key off what is *new*: a hook fires when a record opens or resolves, not on every run where something is still bad. Events are run.regressed, run.over_budget, run.recovered and run.recorded.

Discussion

A regression is a conversation, so comments attach to the thing being discussed rather than to a generic feed. What a comment is about is expressed by which anchors it carries:

Threads are one level deep: a reply cannot have replies. Two levels keeps a conversation about one regression together; more turns a benchmark page into a forum.

Anyone on the team can resolve a thread — that is what resolving is for — but only the author can edit their own comment. Owners and admins can delete any.

CI and agents are participants. A comment posted with an API key is attributed to that key's name and rendered as *automated*, so a finding left by a pipeline is never mistaken for a teammate's. An AI diagnosis can be pinned into the run's thread with one button.

# CI leaves a note on the run it just posted
curl -X POST "$BENCHABLE_URL/api/v1/comments"   -H "Authorization: Bearer $BENCHABLE_KEY"   -H "Content-Type: application/json"   -d '{"runId":"'"$RUN_ID"'","metricKey":"api.latency_ms",
       "body":"p95 crossed the noise band twice in a row. @ada before the release cut?"}'

GET  /api/v1/comments?runId=&metricKey=&open=true
POST /api/v1/comments                      { body, runId?, metricKey?, parentId?, author? }
POST /api/v1/comments/{id}/resolve         { resolved: boolean }

MCP tools: list_comments, post_comment, resolve_comment. The server instructions tell an agent to read the thread before adding to it, so it contributes rather than repeats.

Suites and project health

Forty metrics in a flat list do not say which part of the system is slow. Their keys already do: the first namespace segment groups api.latency_ms with api.p99_ms and leaves bundle.main_kb alone. The project page groups by it and rolls each suite up — what regressed today, what is over budget, what has something open against it, what is drifting.

The header carries a health score, because "no regressions in the last run" is the wrong answer to *are we getting faster or slower?* The last run is compared against the run before it, so a project that regresses 4% a week reports a clean bill every single time. The score looks at what is true now instead:

health 88/100 (fair) over 17 metrics
  Within budget: No budgets set
  Nothing outstanding: 88% — 2 open regressions
  Not drifting: 88% — 2 metrics stepped the wrong way

suite       metrics  regressed  over budget  open  drifting
----------  -------  ---------  -----------  ----  --------
bundle            1          1            0     1         0
api               1          0            0     1         1

A component with nothing to measure is dropped rather than scored as perfect — a project that has set no budgets should not be rewarded for it. GET /api/v1/summary and the project_summary MCP tool return both.

Other endpoints

All take Authorization: Bearer <api key> and are scoped to that key's project.

Errors share one envelope: { "error": { "code", "message", "details"? }, "requestId" }. Codes are stable: unauthorized, forbidden, not_found, invalid_payload, invalid_query, payload_too_large, format_unrecognized, rate_limited, ai_unavailable, internal_error.

Rate limits are per API key, reported in X-RateLimit-Limit/-Remaining/-Reset; a 429 carries Retry-After.

Webhooks

Per project, fired after a run lands, optionally only on regression. Two shapes: signed JSON (X-Benchable-Signature: sha256=<hmac of the raw body>) or Slack blocks. Delivery is best effort with a 5s timeout and is never allowed to fail the ingest — a broken Slack URL must not turn a green CI run red. The last status and error are shown on the settings page.

Events: run.recorded, run.regressed, run.over_budget, run.recovered, and the two scheduled ones below, project.stale and project.digest. A webhook's reports switch governs the scheduled pair; onlyRegressions governs the run ones. They are separate because onlyRegressions means *only when a run was news*, and a staleness alert is the absence of runs — reading that flag would mute exactly the case it exists for.

Scheduled reports

Everything above speaks only when a run arrives, which leaves two silences.

A benchmark job that stopped running looks exactly like a healthy one. The last run is green, nothing is open, the health score is whatever it was in March. Set Expect a run every (hours) in project settings and the project's webhooks get a project.stale event once the newest run is older than that. One alert per outage, not one per hour: the flag is cleared by the next ingest, so a project quiet for a week sends one message and the recovery is implicit in that next run's own webhook. A project that has never reported a run is never stale.

Nobody opens a dashboard they have no reason to open. Set Digest to daily or weekly and a project.digest event goes out around 08:00 UTC (Mondays, for weekly) with where the project stands — health and its components, suites in trouble, open, new and resolved regressions, and the metrics that moved or sit over budget. Unlike a run webhook it is worth sending when nothing happened: "nothing moved" is the useful answer to *how are we doing*.

GET /api/v1/report -H 'Accept: text/plain'

Payments API (payments-api) — last 24 hours
18 runs · 4 open regressions (12 new, 9 resolved)
last run 2026-09-20T19:10:59.775Z

metric             value       change
-----------------  ----------  --------  -----------
bundle.main_kb     412 KB      ▲ +18.2%  over budget
cold_start.p99_ms  1.03s       ▲ +9.96%

health 58/100 (poor) over 6 metrics
  Within budget: No budgets set
  Nothing outstanding: 33% — 4 open regressions
  Not drifting: 83% — 1 metric stepped the wrong way

suite       metrics  regressed  over budget  open  drifting
----------  -------  ---------  -----------  ----  --------
cold_start        1          1            0     1         0
bundle            1          1            0     1         0

The open-regression count is distinct *metrics*, not records — the same metric regressing on three branches is one thing to fix, and it is the number the health block is computed from.

The same content is on demand at GET /api/v1/report and as the project_report MCP tool, so an agent asked "how are the benchmarks" gets it in one call.

Both are driven by an hourly cron (/api/v1/maintenance/reports, guarded by CRON_SECRET), hourly so a digest that missed its window goes out an hour late rather than a week late.

Benchable does not *run* benchmarks on a schedule — it ingests results. CI already owns that cadence; schedule: in your workflow plus the CLI is the supported answer.

How it is put together

Next.js 16 App Router with server components reading Postgres directly, Drizzle, Better Auth with its organization plugin, TanStack Charts, AI SDK over Vercel AI Gateway, mcp-handler.

Adding a format

Each adapter is a pure function — no I/O, fully testable. Add a module to lib/formats/adapters/, implement detect and parse, register it in lib/formats/index.ts (specific shapes before permissive ones), and add a fixture of real tool output under tests/fixtures/ with assertions in tests/formats.test.ts.

Charts

lib/charts/chart-definitions.ts is the only module that imports @tanstack/charts, because it pulls in a large slice of d3. Colors come from CSS custom properties (--ts-chart-*, --viz-*) in app/globals.css, so light and dark swap without rebuilding a chart. The categorical palette was validated for colorblind separation and contrast against both surfaces; its slot order is the safety mechanism, so don't reorder it.

Security notes

• API keys are stored as SHA-256 hashes. The plaintext is shown once, at creation, and the

ingest lookup is by hash — a leaked database row cannot be replayed.

• Keys are project-scoped and not tied to a user, so revoking one stops a CI job without

signing anybody out.

• The read token used by badges is separate and read-only; an ingest key is not accepted there.
• Share links use a third token (bms_…) and only resolve while the project's visibility is

link, so either flipping the switch or rotating the token revokes them. Shared pages import no server action and render no form, render no artifact, and are served noindex with referrer: no-referrer so the token in the path never leaves in a Referer header.

• The API carries no cookies, so CORS is permissive by design: a browser cannot use someone's

session against it.

• Invitations are stored, not emailed; accepting one requires being signed in as the invited

address.