# Using Benchable as an agent

Benchable stores benchmark results and answers whether a change is real. Point at an
instance, take a project API key from its Settings tab, and use the surfaces below. The
full API is described at https://benchable.dev/llms.txt and the full README at https://benchable.dev/reference.

## Read, do not scrape

Every read endpoint answers `Accept: text/plain` with a compact digest — a fraction of the
tokens of JSON and needs no parsing. Prefer it, and learn the whole API in one fetch at
`/llms.txt`. The endpoints an agent actually needs:

    GET /api/v1/summary            — every metric, latest value and change
    GET /api/v1/runs/{runId}       — one run's measurements with verdicts
    GET /api/v1/report             — where the project stands over a window

## Record a run

```sh
curl -X POST http://localhost:3000/api/v1/runs \
  -H "Authorization: Bearer $BENCHABLE_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $GITHUB_RUN_ID" \
  -d '{
    "branch": "main",
    "commitSha": "9f3c1ab",
    "environment": "ci-linux-x64",
    "metadata": { "runner": "github-actions", "node": "24" },
    "metrics": {
      "build.time_ms": 4210,
      "api.latency_ms": { "value": 118.4, "p50": 110, "p95": 180, "p99": 260, "samples": 500 }
    },
    "spans": [
      { "id": "root", "name": "POST /checkout", "startMs": 0, "durationMs": 118.4 },
      { "id": "db", "parentId": "root", "name": "SELECT orders", "startMs": 12, "durationMs": 40 }
    ]
  }'
```

The response classifies every metric against its baseline and carries
`X-Benchable-Regressions`, so a CI decision needs no second request. Always send an
`Idempotency-Key` — a retried step must return the original run, not create a duplicate.

## Metric payload options

| Field | Meaning |
|---|---|
| `value` | The number. Required; a bare number is also accepted. |
| `unit`, `direction` | `ms`, `KB`, … and `lower` (default) or `higher` for is-better. |
| `samples`, `stddev` | A summary distribution — switches the verdict to Welch's t-test. |
| `values` | The raw sample vector — switches it to Mann–Whitney U. 2,000 stored, evenly spaced. |
| `p50` … `p99`, `min`, `max`, `mean` | Percentiles and extents, charted on the run page. |
| `labels` | One label per sample, kept beside the distribution. |
| `spans` | A trace for the run: `{ id, parentId?, name, startMs, durationMs }` each. |

## MCP

```json
{
  "mcpServers": {
    "benchable": {
      "url": "https://benchable.example.com/api/mcp",
      "headers": { "Authorization": "Bearer bmk_..." }
    }
  }
}
```

Every tool answers in text, not JSON. Read `list_comments` before `post_comment`, and
resolve a thread with `resolve_comment` when it is settled.

## Three mistakes that break it

1. **No `Idempotency-Key`.** Every retry of a CI step becomes a new run, baselines ratchet
   forward, and the regression you were hunting becomes the new normal.
2. **No `environment`.** A laptop run compared against CI Linux looks like a regression or
   an improvement, depending on the weather. Send an environment label; the baseline search
   honours it and labels anything it has to widen to.
3. **Gating on percentages alone.** A 6% move on a metric that swings ±12% is noise; the
   verdict already knows this when you send `samples`/`stddev` or `values`. Do not
   reimplement the threshold client-side — send the distribution and read `verdict`.