What Counts as Good, and How to Check It

Someone asks whether the new version is better. Answering requires something that was supposed to exist before the change: a written account of what better would mean for this task. Without it the question can only be settled by impression, and impressions of fluent text are unreliable in a specific direction: fluency is easy to perceive and correctness is not.

What evaluation compares

Evaluation compares observed behaviour against an expectation that was written down before the run.

Both halves carry weight. Observed behaviour, rather than intended behaviour. And an expectation that already existed, because an expectation written after seeing the output has a way of describing the output.

A benchmark is one instance of this shape, with the expectation supplied by whoever built the benchmark and the population fixed in advance. That makes it useful for comparing models with each other. It answers a different question from whether this system does this job, which is why a model that scores better on a benchmark can land worse in the product.

Name the property before choosing the method

There is no way to check output quality. There is only a way to check a property.

• Does the response parse, and carry the fields something downstream depends on?
• Is the answer supported by the document it cites?
• Does it decline the things it is supposed to decline?
• Does it hold the register the product committed to?

Those fail in different ways, get checked by different means, and carry different consequences. Collapsing them into a single quality score produces a number that moves without telling you which of them moved, which is the opposite of what a measurement is for.

Four ways to check, and what each one costs

Deterministic assertions. Did it parse, is the required field present, is the value inside the permitted range, does the cited identifier correspond to a document that exists. These are exact, cheap, and narrow. They cannot tell you whether an answer is any good, and they catch a class of failure that otherwise gets reported as the model behaving strangely. Well-formedness itself is structured output work, applied here after the fact rather than before an action.

Heuristics. Length bounds, presence of a required phrase, overlap with a reference answer. Word-overlap measures such as BLEU and ROUGE sit here. They are quick and they have no access to meaning, so a close paraphrase scores badly and a fluent restatement of the wrong fact can score well. Useful as a cheap tripwire, unsafe as a definition of correct.

Model-based graders. A model scoring output against a rubric. It extends judgment to volumes nobody can read by hand, and it is doing the same probabilistic thing as the system under test. Three consequences follow. A grader that returns a score without a reason gives you a number you cannot audit. Graders can reward confident, well-formed writing, which may be a property the rubric never asked for. And a grader has to be compared against human labels before its scores mean anything, which makes the grader a component you also have to evaluate.

Human review. The expensive one, and the one the other three are standing in for. Every cheaper check approximates a judgment someone made first. An assertion about citations encodes a decision about what counts as supported. A rubric row encodes a decision about what counts as the right tone. The judgment does not disappear when it is automated. It gets fixed at the moment it was encoded, and revisiting it becomes separate work that nothing prompts.

Rubrics

A rubric is the written form of that decision, broken into criteria specific enough that a second person can apply them and land near the first.

That agreement is the test of a rubric, rather than how thorough it looks. Where two reviewers diverge on the same output, the disagreement is worth reading before it is resolved. Two things are worth checking first: whether the criterion is carrying two ideas that pull apart under pressure, and whether it is written in terms of a quality rather than an observable, which leaves each reviewer to supply their own definition.

Keep the criteria to the ones that would change what you do. A rubric that scores nine dimensions produces nine numbers, and attention gets spread across all of them rather than concentrated on the ones that would change a decision.

A fixed set of things that already went wrong

Keep the bad outputs. Sort them by failure mode: wrong fact, wrong tone, missed refusal, malformed structure, quiet truncation, confident answer from thin evidence. Re-score that same collection after a prompt change, a model change, or a change in what gets retrieved.

The value lives in the word same. A fresh sample each time tells you how the system did on that sample. A fixed set tells you what moved, because the only thing that changed between the two runs is the thing you changed.

It is worth being clear about what this set is not. It is not representative of production traffic and should not be read as an estimate of how often anything happens. It is a collection of failures that have occurred at least once, kept so that their return is noticed immediately rather than reported by a user.

Closing the loop

Evaluation and the record of what happened are two halves of one cycle, and the handoff between them is where it comes apart.

A run fails in production. Its record is inspected and the failure mode is named. The named failure becomes a case in the fixed set. The set is rerun. The change ships.

The first two steps depend on what the system preserved while it ran. The rest belong here. The step that gets skipped is the third: a failure is diagnosed, discussed and fixed without ever becoming a case, which leaves nothing to prevent its return except whether someone remembers.

Metrics that change a decision

A retained metric should say what is measured, on what population, over what period, how it was produced, and which decision changes if it moves.

The last of those does the filtering. A number attached to no decision still costs something: it occupies a dashboard, it gets reported, and it competes for attention with the two or three numbers that would actually change what happens next.

This is also the discipline that keeps a score honest when it is quoted later. A figure repeated without its population and its method hardens into a fact about the system, and the conditions it was true under stop travelling with it.

What this changes

Before the next change, write down what would count as worse. Not a target number. A description of behaviour you would not accept, concrete enough that someone else could look at an output and agree it had happened.

Everything on this page is machinery for checking that description quickly and repeatedly. Without the description, the machinery has nothing to check against, and the honest answer to whether the new version is better stays the one you started with.