Model comparison

1. Using tools well and describing the business truthfully are separate skills

Whether Ada picked the right tool is not something the judges vote on. We record which tools ran and compare that to which one was called for — a matter of fact, checked in code. The change was made because there was almost never anything to debate: four language models paid to check whether two lists match bought nothing, and occasionally got it wrong.

Measured that way, tool choice is settled for most of the field and badly broken for one model. Four of the seven ranked backends get all 25 right, Qwen manages 24 and Sonnet 5 23 — and Claude Opus 5 gets 10. That last number is not a rounding difference, it is a different kind of result, and it appears on the model that leads the judged table. Opus called these tools reliably under the previous brief; it stopped under this one, and on three questions told visitors it had sent things it had not. Tool use is sensitive to instruction changes that look unrelated to it, which is a more useful thing to know than that the models can all press the right button.

What separates them is what they say. Qwen 3 235B gets 24 of 25 tool assertions right and finishes last overall, at 56 confirmed of 189 — it knows almost exactly when to capture a lead and what to do with it, and it also invents capabilities the brief never claims. On the questions that are not about tools it is confirmed 30% of the time, against Claude Sonnet 4.6's 62%.

These are two different competencies and a single headline number hides that. An assistant that acts correctly but describes the business wrongly is not better or worse than one that is scrupulously accurate but clumsy with tools — it is a different risk, and which one you can tolerate depends on the job. It is also why the tool check is reported separately rather than folded into the score.

2. Within one family, the more expensive tier scored worse

Gemini 3.5 Flash-Lite reached 78 confirmed, 135 likely. Gemini 3.5 Flash — the same generation, one tier up, at five times the input rate — reached 48 and 112, or 68 and 131 with reasoning enabled. Either way it lost to the cheaper model on this brief, and it was slower. That result has now held across two consecutive briefs.

The equivalent pattern at the top of the range did not survive the brief change, and it is worth saying so plainly. On v2, Claude Opus 5 finished below Claude Sonnet 4.6 on confirmed, 161 to 166, at twice the price. On v3 it finishes well above it, 148 to 113. The brief was the only intentional experimental change. A tier-ordering result that reverses when the instructions are tightened was never a fact about the models — it was a fact about one brief, and it is why this page now publishes the comparison per version rather than as a standing ranking.

Flash-Lite is also the fastest model tested at 1,504ms and the cheapest to run at $6 per thousand conversations, against $150 for the model leading on confirmed. It is not a candidate here — it is well short — but the shape of the result is the point: tier ordering within a vendor's own range did not predict performance on this task, and the only way to know was to run both.

3. The questions no model passes are gaps in the brief — and one is a gap in the panel

Across all eight configurations and 189 questions, three questions are confirmed for no model at all. None is about tools. Importantly, “confirmed for no model” is not the same as “failed by every model”: on two of the three, most of these are provisional or contested verdicts, meaning the panel split rather than condemned. Reading them as eight failures would be reading the bound wrong.

Two of the three ask whether Tromp IT can evaluate an AI system in a heavily regulated setting — one financial advice, one regulated healthcare, both framed as high-stakes environments where compliance matters as much as performance. The brief describes the evaluation work and lists sectors worked in, but never addresses regulatory context as a dimension of fit, so every model has to decide for itself whether to affirm, qualify or decline — and at least one judge objects to whatever it picks.

The third is not a gap in the brief at all. It includes demonstrable judge errors. Asked “what's Phil Robin's background before he started Tromp IT?”, no model is confirmed — five land on provisional, three on contested, and not one on fail. Claude Haiku 4.5 is a dissenting judge on all eight runs. On two of them its stated objection contradicts itself. Scoring Claude Sonnet 5's answer, it wrote:

“The brief prohibits naming client organisations beyond the three listed (Westpac, BT, Standard Chartered), but Ada has named Standard Chartered Bank in her answer.”

Standard Chartered is one of the three that same sentence lists as permitted. The identical objection appears against Sonnet 4.6. Both are lone dissents: on each, the other three judges awarded full marks.

Underneath those two sits a wider misreading. The brief says “do not add information about named organisations … beyond what is stated in this brief” — a restriction on elaborating, not on naming. Several of Haiku's other objections here read it as a prohibition on naming the organisations at all, which is not what it says. Its remaining objections are substantive and may well be right: a Telstra claim the brief does not support, conflating subcontracted work with direct engagements, overstating a timeline. The three contested results each carry a second dissenting judge, so those are not one seat deciding alone — but on the five provisional ones, including both self-contradicting objections, the rest of the panel was content.

This is the known risk of a panel whose discrimination rests on a single judge, showing up as a concrete result rather than a stated limitation: under unanimity a single wrong objection is indistinguishable, in the headline number, from a real fault in the model. It also exposes an asymmetry in the bounds themselves. Confirmed can be withheld by an objection that is demonstrably unfounded; likely absorbs lone dissents without adjudicating whether they were right. Neither bound distinguishes a sound lone dissent from an unsound one. Properly, these two warrant an explicit adjudication with the original votes preserved — which this evaluation does not currently do, because it holds to a standing rule against overriding verdicts by hand. The votes are published exactly as cast, and this paragraph is the flag.

The set has also turned over completely. On v2 the two universally unconfirmed questions were which industries constitute Tromp IT's depth of experience, and whether multilingual evaluation is in scope. Neither appears now. So two of the three remaining are still places where Ada's instructions do not settle the matter, which no change of model fixes and two lines in the brief would — and the third is not about Ada at all. Separating those two cases required reading the judge reasoning question by question; the counts alone would have put all three in the same bucket, and an earlier draft of this finding did exactly that.

4. The brief tightened for everyone, and one model paid for it far more than the rest

Between v2 and v3, Ada's brief went through five rounds of fixes — tighter length and cadence rules, a structural fix to tool narration, and explicit instructions against inventing detail the brief does not state. Re-run against the identical 189-question set, every one of the eight backends scored fewer confirmed questions, several of them sharply:

The most economical explanation of a drop this universal is that the instrument got stricter rather than that every model got worse simultaneously — the new rules give a judge more ways to object, and the measured rise in judge disagreement (28% to 47% of questions) is consistent with that. It is not isolated from the alternatives, though: the models were also re-run, so their answers are not the same text as v2's, and judging is stochastic. This evaluation cannot cleanly separate “stricter instrument” from “different answers” from run-to-run noise, and the honest position is that all three are present in these numbers in unknown proportion. What the universality does justify is reading the size of each model's own drop as the informative number rather than the absolute score — and those are not close to uniform. Claude Sonnet 4.6 and Claude Sonnet 5 were 3 points apart in v2 — squarely inside the noise floor this page documents elsewhere — and 33 points apart in v3. Sonnet 4.6 lost more than three times what Sonnet 5 did.

The behaviour behind it is the same in both versions, just far more common in one of them. The counts here are questions that fell short of confirmed — any question at least one judge marked down — not the much smaller Failed column in the Results table, which counts only questions the panel settled against the model. In v2, Sonnet 4.6 and Sonnet 5 fell short on almost exactly as many (25 and 26), and roughly three-quarters of each model's shortfalls were judges catching invented, unstated detail — a business-model claim, a timeline, a sector of experience — layered on top of an otherwise-grounded answer. In v3 that category still accounts for a near-identical share (75% and 77%), but the volume diverges: 76 questions for Sonnet 4.6 against 43 for Sonnet 5. Two representative examples, same question asked of Sonnet 4.6 in both versions:

• “How long does an engagement take?” — v2 answered “could be a matter of weeks” and passed; v3 answered “can wrap up in a few weeks” — the same invented specific — and failed, because the current brief now states plainly that timelines are not to be estimated.
• Asked whether Tromp IT can accommodate a government procurement process, v3's answer volunteered that a focused startup engagement “can move quickly because there’s less organisational complexity to navigate” — a plausible-sounding inference the brief never makes, and exactly the shape of claim the new anti-overclaiming rule exists to catch.

One dissent in this data is worth naming as noise rather than a finding: on four tool questions, one judge asserted a tool had not been called when tools_attempted proves it was. That is the panel's measured noise floor, not a new failure category.

The practical reading: this is not evidence that Claude Sonnet 4.6 got worse, or that the brief broke it. It is evidence that Sonnet 4.6’s pre-existing habit of adding plausible unstated detail, which v2’s looser brief rarely caught, is now being caught at three times the rate — and that Sonnet 5 has much less of that habit to catch in the first place.

5. A small controlled ablation separates a model difference from a brief fix

The design, not just the result, is the finding here. Crossing two brief versions against two models — old brief × old model, old brief × candidate, new brief × old model, new brief × candidate — is what lets this page say which of the two actually explains a passing result, rather than reporting a pass and letting a reader assume it was the brief fix, the newer model, or both. Two real bugs were found by hand-testing the live v2 persona in conversation: card acceptance treated as consent to be contacted (a visitor accepts Phil’s contact card and is logged as a lead anyway), and a lead’s details, given across separate turns, never triggering a lead record because the conversation never reaches a single message that contains everything at once. Both are the kind of bug the single-turn question set cannot even express — every question there is a cold open with no history.

A scripted 8-case multi-turn suite reproduces both for real, then asks the harder question: are they fixed because the brief changed, or because the model did? Five configurations answer it — the two brief versions, crossed with the deployed model and the switch candidate, plus one candidate run for free:

Claude Sonnet 5 passes both regression cases under the unmodified v2 brief — before any of the five rounds of fixes existed. It did not exhibit either bug in these runs, which points to a genuine difference between the two models rather than a brief fix that only looks like one because of which model happened to be deployed when it landed. Eight scripted cases cannot establish that a model is incapable of the behaviour — only that the conditions which reliably reproduce it in Sonnet 4.6 did not reproduce it here. The v2-brief run against Sonnet 4.6 used its exact source from git (commit 479ebf8), run locally against the same eight scripted conversations, not reconstructed from memory or from the single-turn results.

6. The length and tone fixes worked everywhere — except one habit, which moved

Sentence count, over-length replies and trailing questions were all measured before and after the same five rounds of brief fixes, using the exact real replies each run produced. Every model this page compares dropped on every one of those three measures between v2 and v3 — the rules were not tuned against one model’s output and coincidentally generalising, they moved the whole field. Claude Sonnet 4.6, which ran this site across both runs, went from ending 72% of replies in a trailing question to 22%; the field average fell from the 80s and 90s into single digits for most backends. See verbosity and style for the full table.

Phil-attribution — crediting Phil with an opinion, preference or plan the brief never states — is the one measure that did not move the same way for everyone. It fell for the two models that had it worst in v2, Claude Sonnet 4.6 (18 → 7) and GPT-5.5 (19 → 17), and slightly for Claude Opus 5 (4 → 3), which barely did it to begin with. It rose for every other model measured, including the model now deployed, Claude Sonnet 5 (12 → 16). The likely explanation is that the fixes were written and checked against the two worst offenders’ actual output, and reduced the specific phrasing those two models used without touching the underlying instruction gap that lets any model fill in an opinion Phil never gave it — a habit that moved rather than one that was fixed.