Model bake-off
Pro.
A model bake-off compares one OpenRouter model against one Fireworks model on the same representative sample of files from a BNLRepo, and reports latency, tokens, cost and a deterministic quality score side by side.
It exists because choosing a summarisation model by reading provider marketing is a bad way to spend money. Run it on your code and look at the numbers you actually get.
It is telemetry only
A bake-off never commits an alternate .llm file and never changes which model a BNLRepo is paired to. It is a way to decide what to point a pairing at, not a way to run one.
The generated summaries themselves are not retained. What is stored per result is a SHA-256 of the output, the quality score and its reasons, fact counts, token counts, cost and latency, and any failure message. There is no commit or push path anywhere in the bake-off code.
What is compared
Per contender:
- Succeeded and failed file counts
- Average quality score
- Average latency, in milliseconds
- Input and output token counts
- Cost, in micro-USD
Plus an overall best-quality contender.
Guardrails
| Guardrail | Value |
|---|---|
| Files sampled | 12 by default, 24 maximum |
| Runs per person, per hour | 3 |
| Runs per repository, per hour | 5 |
| Cost ceiling per run | $0.25 (on your provider account) |
| Mirror repositories | Refused — bake-offs run against source repos only |
Rate limits are enforced under a lock rather than checked-then-acted-on, so two simultaneous requests cannot both slip through. Exceeding a limit returns HTTP 429.
Every run requires explicit cost confirmation before any provider call is made — see Managed worker and spend controls. Requests accept an Idempotency-Key header, so a client retry after a timeout cannot double-bill you.
Running one
From the UI, or:
POST /api/v1/repos/{owner}/{repo}/bnlgit/model-bakeoffs
Authorization: token <your token>
Content-Type: application/json
Idempotency-Key: <any unique string>
{ "confirm_cost": true }Or from the command line on the instance:
bnlgit summaries bakeoffThe quality score is a deterministic signal computed from the generated summary — how well-formed it is, how many facts it anchored, whether its spans validate. It is a useful comparator between two models on the same files. It is not an absolute measure of correctness, and we do not publish cross-model leaderboards derived from it.
Something here wrong or missing? Mail docs@bnlgit.com — these pages are the single public source of truth for BNLGit and we would rather fix them than leave you guessing.