Method
Evaluation doesn't start by picking a metric. It starts by defining what success actually means for this system — then choosing the measures, the test set and the judging method that fit what you're trying to find out.
The process
What is the system actually supposed to accomplish, and which failures matter? Accuracy is rarely the whole answer — task completion, groundedness, tool use, latency and cost all matter too, and different systems weight them differently. A medical triage bot and a marketing assistant that both "answer correctly" 95% of the time are not carrying the same risk.
Ordinary questions, edge cases, adversarial attempts and known failure modes — deliberately, not as an afterthought once the easy questions run out. Test-set composition is itself part of the method: what you choose to ask determines what you're able to find, and the set is versioned so a later change to it is visible rather than silent.
Some things are objectively knowable and should be checked in code, not asked of a language model. Whether a required action actually happened — a tool call, a database write, an API request — is a fact about the runtime, and a judge's opinion of it is a worse instrument than simply checking. Other things genuinely require judgement: is this answer well-grounded, is this tone appropriate, is this refusal reasonable. The two get different treatment, on purpose.
Where judgement is genuinely required, we use LLM-as-judge, and we check its reasoning rather than taking it on faith. A judge marking an answer down is making a checkable claim about the source material, and that claim can be tested. Independence matters too: for independent evaluation, a judge is not also a system under test — a model marking its own work isn't independent evidence. Multiple judges, and disagreement reported rather than silently resolved, catch what a single strict or lenient judge would miss.
A headline score tells you how much of the analysis is still to do. What does the remainder represent — harmless wording variation, a systematic gap in the system's instructions, an occasional hallucination, or an inability to complete the one task that actually matters? Two systems can post an identical score and carry entirely different risk.
The system under test, its configuration, the test set, the judging method and the date it was run are all recorded together, so a result can be checked rather than taken on trust — and so a later re-run can tell you whether something genuinely changed, or whether the instrument moved instead.
What the measures mean
Not every evaluation uses every measure. The measures are selected according to the system, its purpose, and the consequences of getting it wrong.
| Measure | What it answers | How it's typically measured |
|---|---|---|
| Accuracy | Is the answer correct? | Judged against the source of truth or stated criteria |
| Groundedness | Is the answer supported by the available material, rather than invented? | Judged against the retrieved or source content |
| Task success | Did the interaction achieve what it was for? | Explicit, system-specific success criteria |
| Tool correctness | Did the system take the required action? | Checked in code against the runtime, not judged |
| Consistency | Does it behave reliably across repeated attempts? | Repeat runs, variance across them |
| Latency | How quickly does it respond? | Instrumented directly |
| Cost | What does it cost to run in production? | Measured token and API usage at published rates |
| Business outcome | Did it produce the result the business actually needed? | Production data, over time |
Confirmed, provisional, contested
A binary pass/fail forces a threshold onto disagreement a judging panel could not resolve — which throws the disagreement away as if it were noise, when it is often the most useful part of the result. So a question doesn't just pass or fail; it lands in one of four states.
Confirmed — every judge agreed, or a single dissenter's objection didn't hold up against the source material. Provisional — one dissenter, but their objection does check out: a real judgement call, not a wrong answer. Contested — the panel genuinely split. Failed — a majority agreed it was wrong, or, for anything with a required action, the deterministic check failed regardless of what the judges thought. Nothing here is averaged away, and a provisional or contested result is reported as unresolved rather than resolved by rounding. See how this works in our own published evaluation →
Why one score isn't enough
A headline percentage invites one question and answers none of the ones that matter. Is that good? It depends — on which questions failed, how important they were, whether the failures were random or systematic, whether the judges agreed with each other, and whether the same shortfall would matter more in a different system. A score is where the analysis starts. That's the argument for publishing failure analysis alongside the number, rather than folding it into one.
We hold ourselves to the same standard we bring to a client's system: our own published evaluation of the assistant on this site reports exactly this way, including the questions our own judging panel could not agree on. Read the published evaluation →
See the methodology in practice
We use the same approach on our own systems that we use in client work. The assistant on this site is a live example — no numbers here, because they change every time it's re-evaluated; the current figures live on the pages themselves.
Every question asked, grouped by what it's testing — including the out-of-scope and adversarial cases most benchmarks skip.
See the test set →The same test set run against multiple model backends, with cost and latency measured alongside accuracy and groundedness.
See the comparison →Where the judging panel agreed, where it split, and what that reveals about the panel itself as well as the models it scored.
See the judging results →What changed when the system under test was rewritten, and how we separated a real behavioural change from the instrument itself moving.
See the version history →