Ada's brief went through five rounds of fixes between these two versions — tighter length and cadence rules, a structural fix to how tool use is narrated, and, the one that reshapes the model comparison, an explicit rule against inventing detail the brief does not state. The question set did not move at all this time: same 189 questions, same wording, both times, confirmed by direct comparison rather than assumed. Both brief versions are fingerprinted, so the two runs can be compared question by question rather than headline to headline — which matters, because the headline moved for every single model, and by very different amounts.
Version comparisons: v1 → v2 · v2 → v3 (this page)
One thing moved: the brief. The question set and the scoring method are identical across both runs, which is what makes this transition worth reading closely — and what makes a single before-and-after number misleading anyway, because the brief is the standard the score is measured against.
Ada's instructions went through five separate fixes: tighter sentence-length and cadence rules, an explicit rule against inventing detail — a timeline, a sector of experience, a business-model claim — that the brief does not state, a structural fix to how tool use gets narrated, and a voice pass on how Ada describes Tromp IT's services. Each version of the brief is fingerprinted, and the two fingerprints are recorded at the foot of the page.
189 questions stayed 189 questions: same set, same wording, both times, verified by direct comparison rather than assumed. Nothing here needs to be disentangled from a changing test — the brief is the only intentional experimental change. It is not the only source of variation: model and judge output are stochastic, which is why the chart below reports movement in three-of-four terms as well as unanimous ones, rather than treating every point of difference as caused by the brief.
Same four judges, same thresholds, same generation settings, same deterministic tool assertions. But "the method is unchanged" is not the same claim as "the judges scored it the same way" — see variance in the judging panel below, where they did not.
Each line is one model, from its v2 score to its v3 score on all 189 questions — none held out this time, because none of them changed meaning between the two briefs. Hover any line or endpoint for the model and both of its figures.
0 of the 8 models improved, 8 went backwards — every model that moved down moved down by double digits. Claude Opus 5 lost the least, 13 questions. Claude Sonnet 4.6 lost 51 — the single largest movement of any model in either direction, and the result that prompted the deployment change described in why Ada runs on Sonnet 5.
A question passes only when all four judges award full marks. One dissenter fails it. That makes the headline unusually sensitive to how often the panel splits — and it splits far more often on v3 than it did on v2 (see below). Scored a second way, passing when three of four judges agree, most of the field's collapse nearly disappears. Not all of it.
| Model | v2 confirmed of 189 |
v3 confirmed of 189 |
Change | 3-of-4 |
|---|---|---|---|---|
| Claude Opus 5 | 161 | 148 | -13 | -11 |
| GPT-5.5 | 172 | 146 | -26 | -3 |
| Claude Sonnet 5 (Ada today) | 163 | 146 | -17 | +0 |
| Claude Sonnet 4.6 | 166 | 113 | -51 | -20 |
| Gemini 3.5 Flash-Lite | 120 | 78 | -43 | -23 |
| Gemini 3.5 Flash (thinking on) | 112 | 68 | -44 | -25 |
| Qwen 3 235B | 90 | 56 | -34 | -28 |
| Gemini 3.5 Flash | 98 | 48 | -50 | -38 |
Both confirmed columns are over the same 189 questions — the headline bound from each run's own report. Change and 3-of-4 are computed on unanimous passes only, so they can differ by a question or two from the difference between the confirmed columns, where a verdict was auto-corrected.
A headline score averages over eight quite different kinds of question. Splitting it by version shows which categories actually separate the models, which moved between v2 and v3, and which are buying nothing. Each dot is one model; the two rows are the two versions, measured on the questions common to both.
The same movement per category, measured only on the comparable questions, under both pass rules — because a gain that evaporates when one dissenting judge is forgiven was never a gain in behaviour.
These are the model comparison's own charts, drawn once per version and put next to each other. Same models, same questions, same judges, same settings — the brief is the only intentional difference between the two panes.
Each model is a whisker rather than a dot: the bar runs from confirmed up to ceiling, with likely marked on it, against running cost on a log scale. A dot would assert a precision the panel does not have, and the overlap between the leading models is the thing worth seeing. The vertical axis is the share of each version's question set, so the two panes are directly comparable on the same 189 questions both times. Watch the top of the field: the whisker furthest up the page changes hands, and the leaders' whiskers overlap in both versions.
The panel is meant to be the fixed part of the instrument — the same four judges, the same prompts, the same thresholds in both runs. It is worth checking, because if the judges moved then everything above is measuring the wrong thing. Both charts use only the comparable questions, so neither is a question-mix effect.
v2 v3
Being honest about the limit: judges score answers, so marks falling could mean the answers genuinely got worse rather than the judges tightening up, and these numbers cannot fully separate the two. What argues for a real behavioural change, not just a stricter panel, is the shape of the movement: a panel effect should shrink similarly for every model under the lenient rule, and it does not — Claude Sonnet 5 goes flat while Claude Sonnet 4.6, which ran this site when both runs were made, keeps a real double-digit loss. That asymmetry is difficult to explain as a uniformly stricter panel alone — though different models produce different answer distributions, which could in principle interact with a stricter panel unevenly even without a real behavioural difference behind it.
Versioning the brief and the question set made these visible. Without it each one would have been invisible, or worse, would have looked like something else.
v2 had the deployed model, Claude Sonnet 4.6, essentially level with Claude Sonnet 5 — 166 against 163 confirmed, a 3-question gap squarely inside what this evaluation elsewhere calls noise-floor territory. v3 has Sonnet 4.6 at 113, Sonnet 5 at 146 — a 33-point gap that survives even under the lenient three-of-four rule, where Sonnet 4.6 is still down 20 and Sonnet 5 is flat. The brief was the only intentional experimental change between the two runs, and the two models responded to the tightened version very differently. Ada was moved to Sonnet 5 on 2 October 2026 on this evidence; the reasoning, and what it does not claim, is set out in why Ada runs on Sonnet 5. This page covers what changed experimentally between the two versions; that one owns the deployment decision.
In v2, Sonnet 4.6 and Sonnet 5 failed almost identically often — 25 and 26 questions — and about three-quarters of each model's failures were judges catching invented, unstated detail: a business-model claim, a specific timeline, a sector of experience. v3's anti-overclaiming rule targets exactly that habit, and it worked on the field's answers, not on the underlying behaviour: the same failure type still accounts for roughly the same share of each model's failures (75% and 77%), but the volume diverges hard — 76 for Sonnet 4.6 against 43 for Sonnet 5. See the overclaiming finding for two concrete examples, including the same question answered by the same model in both versions.
Two real bugs, found by hand-testing the live v2 persona in actual conversation rather than by asking it one question at a time, are now regression-tested on every run: card acceptance mistaken for consent to be contacted, and a lead's details, gathered one at a time across turns, never triggering a lead record at all. Run against the unmodified v2 brief, Claude Sonnet 5 passes both; Claude Sonnet 4.6 fails both for real. That was true before any of the five brief fixes existed, which is what makes this a difference between the two models rather than a side-effect of the brief changing underneath them. See the multi-turn ablation.
On four tool questions in v3, one panel member asserted a tool had not fired when the run's own record of which tools actually ran proves it did. Every one of those four cases had the correct tool run, correctly. This is the panel's measured noise floor showing up somewhere that looks, on a first read, like a real Ada bug — it isn't, and it is exactly the kind of single vote a three-of-four rule exists to outvote rather than a finding about the model.
v2's replies and v3's were run back through the same sentence-length and cadence measurement. Every model measured in both runs dropped in average sentence count and in how often it ends a reply with a trailing question — the rules were not tuned against one model's output and coincidentally generalising, they moved the whole field. One habit did not follow that pattern: crediting Phil with an opinion the brief never states fell for the two models that had it worst in v2, but rose for most of the rest, including Claude Sonnet 5, which now runs the site. See verbosity and style for the full table.
v2 had two questions confirmed for no model — which industries constitute Tromp IT's depth of experience, and whether multilingual evaluation is in scope — both genuine silences in the brief that every backend inherited equally. v3 has three, and none of them are the same two. Two ask about a regulated-industry use case (financial advice, healthcare), and are the same kind of brief silence.
The third is not. Asked about Phil's background before founding Tromp IT, no model is confirmed — but there is not a single fail among the eight, and one judge dissents on every one of them, twice on reasoning that contradicts itself. That one is a gap in the panel rather than in the brief, and it is set out in full in the questions no model passes. An earlier draft of this card said every model invented career history; the judge reasoning does not support that, and the correction is recorded rather than quietly swapped.