Three findings from the same run, at more length than the comparison page carries. They are supporting detail: the main page answers what the evaluation found and which model the site runs, and these sit behind it. Each states its own limits.
On a tool question the action is finished in the first clause, so every sentence after it is a fresh assertion with no upside available — only downside. On a conversational question, length is where the good behaviour lives: acknowledging a limit, deferring to Phil, asking rather than assuming. The two pull in opposite directions.
| Model | Reply length, tool | Confirmed, tool | Reply length, other | Confirmed, other |
|---|---|---|---|---|
| Qwen 3 235B | 236 | 28% | 344 | 30% |
| Gemini 3.5 Flash-Lite | 238 | 48% | 301 | 40% |
| GPT-5.5 | 246 | 92% | 372 | 75% |
| Gemini 3.5 Flash | 350 | 20% | 359 | 26% |
| Claude Sonnet 4.6 | 368 | 48% | 501 | 62% |
| Claude Sonnet 5 | 387 | 68% | 501 | 79% |
| Claude Opus 5 | 389 | 24% | 427 | 87% |
The sign still flips. Reply length correlates −0.24 with score on tool questions and +0.64 on every other category. The two previous runs measured −0.55/+0.56 and −0.15/+0.72. The direction has survived three runs; the magnitude has moved every time, which is roughly what a seven-point sample should be expected to do.
Claude Opus 5 is the clearest single illustration in the table: 87% confirmed on non-tool questions and 24% on tool questions, at almost identical reply lengths. The model that writes best about the business is the one that most often fails to do the thing it is describing — see the tool-correctness column in the Results table, where it scores 10 of 25.
Seven models is far too few to treat either coefficient as a measurement, and it is offered as a mechanism visible in the replies rather than a result. The practical form is unchanged and still worth one line in the brief: report what you did; do not predict what Phil will do.
Thinking was disabled on every model that allows it, so the comparison stayed like-for-like. That is a defensible choice and it has a measurable price, which we tested rather than assumed by re-running one model with its reasoning left on:
Gemini 3.5 Flash scored 48 confirmed with thinking off and 68 with it on — and 112 against 131 on likely. It also cost 73% more and took 120% longer. The gap is 20 questions, up from 14 on the previous run: on a stricter brief, switching reasoning off costs more than it used to.
The same uniform configuration produced a stranger result on Claude Opus 5, and a more dangerous one. On three questions it wrote a fluent confirmation — “Sent — it may take a minute to arrive, so check your junk or spam folder if it doesn't show up” — having called no tool at all. Nothing was sent. The reply is more convincing than the correct one, and a judge reading it as prose would have passed it; the deterministic assertion caught all three, which is the clearest demonstration on this page of why that layer exists.
So a uniform configuration is necessary for a fair comparison and simultaneously penalises models that reason by default. Both figures are reported rather than picking the flattering one, and the practical warning is concrete: reasoning is charged against the same token budget as the reply, so a model switched on without raising that ceiling can return a truncated answer, or none at all.
Measured on the same 189 questions, Claude Sonnet 5 consumed 34% more input tokens and 37% more output tokens than Claude Sonnet 4.6 — while making fewer tool calls (19 against 22) and writing essentially the same quantity of reply text (91,857 characters against 91,331, a difference of under 1%). Same work, same amount of prose, a third more tokens.
It is still the cheaper model, because its published rate is a third lower: $67 per thousand conversations against $74. Those two effects run in opposite directions and neither is visible on a rate card alone.
An earlier version of this page had this backwards, reporting Sonnet 5 at $96 and calling the two rates identical. Sonnet 5 had been recorded internally at Sonnet 4.6's rate — a copied figure, never checked — which overstated its cost by 50% and inverted its standing against the deployed model. Cost is now recomputed from measured tokens at current published rates every time this page is built, rather than frozen into the run at the moment it executed.
Every cost figure here is scaled by measured token usage rather than projected from published rates. That also keeps GPT-5.5 below Opus 5 on cost, $136 against $150, which the rate card alone had the other way round. Price per token is not price per conversation.