Pruning My Pothos
← StoryboardsStoryboard · 10 frames · Stage 06, Evaluation & evidence

What Counts as Good, and How to Check It

Every check is standing in for a judgment someone wrote down first. Ten frames on naming the property before choosing the method.

Download PDFRead the article →

01 / 10

  1. A sweating trophy engraved BEST, followed by a blank nobody filled in, and an executive holding a scorecard that reads: vibes. Graded on vibes, shipped on hope. Evaluation compares behaviour against an expectation written down before the run, and every faster, cheaper check approximates that written judgment.

  2. A single quality score of 7.3 is crossed out: it moves without saying which part moved. Check properties instead. Does it parse, with the fields and types right? Is it supported by the source it cites? Does it decline what it is meant to? Does it hold the register the product committed to?

  3. Goalposts, relocated post-kick. Evaluation compares observed behaviour against an expectation that was written down before the run.

  4. Four ways to check: deterministic assertions, exact, cheap and narrow; heuristics, a tripwire rather than a definition of correct; model-based graders, judgment at volume from the same probabilistic kind of system; and human review, the one the other three stand in for.

  5. Model-graded. Model-approved. A+. Model-based graders are one of four ways to check, each an approximation of the written judgment. The expensive one, people, is the one the other three stand in for.

  6. A jar of past failures: wrong fact, wrong tone, missed refusal, malformed, truncated, sure on thin evidence. Change something, rerun the same set, see what moved; only your change moved. Fresh samples are crossed out in favour of the same set. It is not a traffic estimate, just failures kept so their return gets noticed.

  7. Tracked everything. Decided nothing. A retained metric should say what is measured, on what population, over what period, how it was produced, and which decision changes if it moves.

  8. A retained metric should say what is measured, on what population, over what period, how it was produced, and which decision changes if it moves. If no decision would change, drop the metric.

  9. Five thumbs up. Zero written criteria. Evaluation only becomes possible once someone writes down what good means for this task. Every faster check approximates that written judgment.

  10. A person writes a note: worse means a confident answer with no source. Write it before the change, concretely enough that someone else could agree it happened. Everything else is machinery for checking that description.