Ada's brief was rewritten and her test set grew. Both are versioned and hashed, so the two runs can be compared question by question rather than headline to headline — which matters, because most of the movement in the headline is not the models getting better.
Version comparisons: v1 → v2 (this page) · v2 → v3
This is the earlier comparison, kept as published. The brief has moved on since; v2 → v3 is the current one.
Three things moved at once, which is why a single before-and-after number would be meaningless.
Ada's instructions were rewritten to match how Tromp IT now describes itself. The experience section was reworked, how engagements are delivered is stated plainly rather than left to be discovered, and a privacy and data-handling section was added from scratch. Ada also stopped asking every visitor for their name before answering anything. Each version of the brief is fingerprinted, and the two fingerprints are recorded at the foot of the page.
175 questions became 189: 1 retired because its subject left the brief, 2 rewritten because the underlying fact changed, and 15 added — including a new Privacy & Data Handling category, so the claims the site now makes about data have questions behind them.
Same four judges, same thresholds, same generation settings, same deterministic tool assertions. That is deliberate: if the measuring apparatus moved too, nothing below could be attributed.
Each line is one model, from its v1 score to its v2 score on the 154 directly comparable questions. Hover any line or endpoint for the model and both of its figures.
6 of the 8 models improved, 1 went backwards. GPT-5.5 gained 22 questions. The model Ada actually runs, Claude Sonnet 4.6, gained 13.
A question passes only when all four judges award full marks. One dissenter fails it. That makes the headline unusually sensitive to how agreeable the panel happened to be — so here is the same movement scored a second way, passing a question when three of four judges agree.
| Model | v1 confirmed of 175 |
v2 confirmed of 189 |
v1 stable | v2 stable | Change | 3-of-4 |
|---|---|---|---|---|---|---|
| GPT-5.5 | 136 | 172 | 121 | 143 | +22 | +7 |
| Claude Sonnet 4.6 (Ada today) | 143 | 166 | 122 | 135 | +13 | +1 |
| Claude Sonnet 5 | 140 | 163 | 123 | 134 | +11 | -1 |
| Claude Opus 5 | 154 | 161 | 135 | 131 | -4 | +3 |
| Gemini 3.5 Flash-Lite | 108 | 120 | 91 | 97 | +6 | +1 |
| Gemini 3.5 Flash (thinking on) | 104 | 112 | 84 | 88 | +4 | +6 |
| Gemini 3.5 Flash | 90 | 98 | 76 | 76 | +0 | +11 |
| Qwen 3 235B | 79 | 90 | 67 | 75 | +8 | -1 |
The two confirmed columns are over different denominators and are shown for orientation, not comparison. The stable columns are the like-for-like pair.
A headline score averages over eight quite different kinds of question. Splitting it by version shows which categories actually separate the models, which moved between v1 and v2, and which are buying nothing. Each dot is one model; the two rows are the two versions, measured on the questions common to both.
The same movement per category, measured only on the comparable questions, under both pass rules — because a gain that evaporates when one dissenting judge is forgiven was never a gain in behaviour.
These are the model comparison's own charts, drawn once per version and put next to each other. Same models, same judges, same settings — only the brief and the question set differ.
Each model is a whisker rather than a dot: the bar runs from confirmed up to ceiling, with likely marked on it, against running cost on a log scale. A dot would assert a precision the panel does not have, and the overlap between the leading models is the thing worth seeing. The vertical axis is the share of each version's question set, so the two panes are directly comparable even though one is out of 175 and the other out of 189. Watch the top of the field: the whisker furthest up the page changes hands, and the leaders' whiskers overlap in both versions.
The panel is meant to be the fixed part of the instrument — the same four judges, the same prompts, the same thresholds in both runs. It is worth checking, because if the judges moved then everything above is measuring the wrong thing. Both charts use only the comparable questions, so neither is a question-mix effect.
v1 v2
Being honest about the limit: judges score answers, so marks rising could mean the answers improved rather than the judges softening, and these numbers cannot fully separate the two. What argues for the panel is the shape of the movement. A genuinely better answer should gain under both rules; almost all of the gain here appears under unanimity alone, which is the signature of votes consolidating rather than answers improving. Tool Usage is the exception, and the one place a behavioural change is visible in the replies themselves.
Versioning the brief and the question set made these visible. Without it each one would have been invisible, or worse, would have looked like something else.
v1 put Claude Opus 5 in front and had the deployed model, Sonnet 4.6, trailing Sonnet 5 — the conclusion was probably switch, after one check. In v2 GPT-5.5 leads, Sonnet 4.6 is the strongest Claude model tested, and Opus 5 is the only model in the field that went backwards on the comparable questions. Nothing about the models changed between the two runs. The brief did.
Opus 5 stopped emailing transcripts. Bisecting the brief found the cause in one rewritten section: v1 told the model the visitor had already been greeted and asked who they are, which implied a conversation was under way. v2 removed that, and Opus started replying “I don't have any earlier conversation to send” — which is true, since every test question is a fresh conversation. The old wording papered over an artefact the test set had always had.
On two questions Opus 5 called no tool and said it had: “Sent — the transcript's on its way… worth a glance in junk or spam if it doesn't show up.” Nothing was sent. The reply is more fluent than the correct one and carries no anomaly flag. Judges reading it as prose scored it well; only the deterministic tool assertion caught it. No other model did this.
v1 reported that reply length correlated −0.55 with score on tool questions and +0.56 elsewhere — the same habit helping in one register and hurting in the other. On v2 the figures are −0.15 and +0.72. The sign still flips; the effect on tool questions does not really survive. Eight models was always too few to call it a measurement, and this is what that looks like when it is re-run.
The v1 results were produced against brief 37a977f62c19. A later
commit changed the brief without anyone noting it, so for a period the published
figures described instructions the assistant was no longer running. It is visible now
only because the hash is recorded with every run. Five older runs cite briefs whose
text no longer exists anywhere — those runs can prove they used something
different, and can never say what.
In v1 exactly one question was passed by no model. In v2 there are two, and they are different ones: which industries constitute Tromp IT's depth of experience, and whether multilingual evaluation is in scope. Neither is a model failure. Both are places the brief does not settle the matter, and every backend inherits the gap equally.
Identical wording in both versions, different correct answer. These are excluded from the comparison above. The judgement about which questions belong here is recorded in the transition file rather than inferred, because deciding that a changed paragraph affects a particular question is a reading, not a diff.
| Question | Why its answer moved |
|---|---|
EXP-004 | Hoot is now identified as Dirawong's platform, not an unattributed one |
EXP-007 | Financial services experience now names Westpac, BT and Standard Chartered |
EXP-008 | The Medibank engagement is now disclosed as delivered through Dirawong |
EXP-009 | Belinda is out of the brief, so 'other people involved' has a different answer |
EXP-011 | Team size no longer includes Belinda |
EXP-014 | Where the offshore work was delivered is now described differently |
EXP-015 | The cloud optimisation engagement now names Araza's delivery team |
EXP-018 | Industry experience now includes telecommunications, via Telstra |
EXP-020 | The constraint that Tromp IT does not own, build, resell or implement Hoot is now explicit |
EXP-021 | Ada's own published evaluation is now available as an answer |
EXP-023 | How the Medibank work arose now runs through the Dirawong relationship |
HOW-003 | The NDA answer is now backed by an explicit data-handling section |
HOW-004 | Subcontracting is now a disclosed operating mode rather than a capability claim |
HOW-010 | White-labelling now sits against the Dirawong disclosure |
HOW-022 | Handling of confidential information now has a stated policy to answer from |
OOS-019 | Belinda's communications and media offering is gone, so this is now cleanly out of scope |
SVC-011 | AI Usage & Optimisation is no longer flagged as a developing capability |
SVC-024 | Same: the AI spend answer no longer carries the developing-capability caveat |