Part 4 of 5
From Vibes to Benchmarks
The fourth spine page replaces vibe-based model choice with measured tradeoffs that can survive repeated runs on one machine.
Broken intuition
The biggest model, the fastest first token, the largest context, and the model with vision support can all be different candidates. The repo treats that as a benchmark problem instead of a taste problem: sweep the same contexts, keep cooldown and probe-series policy explicit, record deferred axes instead of silently dropping them, then let a human lock the daily driver after reviewing the tradeoffs. src/coding_agent/main.clj:93-176 README.org:20-31
Experiment
Choose the priority, then inspect which metric answers it and why that still does not auto-pick the winner.
A measured tradeoff here means prompt throughput, generation throughput, draft acceptance, memory, and sustained generation are compared under one explicit cooldown policy instead of one lucky run. src/coding_agent/main.clj:101-142 src/coding_agent/metrics.clj:20-26
Revealed constraint
Comparable rows need normalized metrics because the runtimes do not all report the same raw shape. llama-server and mlx_vlm.server expose timings directly, while for mlx_lm.server prompt throughput is unavailable client-side: prompt-tps is nil, and generation throughput comes from the streamed usage window. That is why the same page has to teach prompt throughput and generation throughput as different questions, and why draft acceptance only appears when MTP is actually configured. src/coding_agent/bench.clj:56-60 src/coding_agent/metrics.clj:20-26
Named concept
The useful unit is a measured tradeoff: one benchmark row that records the request model id the server expects, the context that was tested, prompt throughput when the runtime reports it, generation throughput, draft acceptance when present, memory, and the cooldown policy that kept the run comparable. That bundle makes the final lock a reviewable Pareto decision rather than a vibe. src/coding_agent/bench.clj:46-53 src/coding_agent/bench.clj:56-60 src/coding_agent/metrics.clj:20-26 src/coding_agent/lock.clj:55-88
Repo implementation
bb bench writes a dated runbook with the machine, context grid, probe series, eval tier, and cooldown policy in provenance, plus deferred axes for anything that the first sweep intentionally leaves out. That makes repeatability visible. The next command, bb lock --candidate <id>, does not auto-pick a winner; it materializes the human-reviewed choice into a reviewable {:locked ...} block that preserves the chosen runtime, model, server, and spec. src/coding_agent/bench.clj:46-53 src/coding_agent/runbook.clj:38-60 src/coding_agent/main.clj:118-176 src/coding_agent/lock.clj:55-88 README.org:20-31
Recognition
Pick the statement that matches the repo's benchmark discipline.
Select a statement to see whether it matches the measured tradeoff loop.
Sources
src/coding_agent/main.clj:93-176 src/coding_agent/bench.clj:46-53 src/coding_agent/bench.clj:56-60 src/coding_agent/metrics.clj:20-26 src/coding_agent/runbook.clj:38-60 src/coding_agent/lock.clj:55-88 README.org:20-31