Pruning My Pothos
← StoryboardsStoryboard · 6 frames · Stage 06, Evaluation & evidence

What Counts as Good, and How to Check It

Every check is standing in for a judgment someone wrote down first. Six frames on naming the property before choosing the method.

Download PDFRead the article →

01 / 06

  1. PRUNING MY POTHOSSYSTEMS · STAGE 06A STORYBOARD IN 6 FRAMESWhat counts asgood, and howto check itBEST?engrave the blank firstbetter at what,exactly?
    Evaluation compares behaviour against an expectation written down before the run. Every faster, cheaper check approximates that written judgment.
    PRUNINGMYPOTHOS.COM01 / 06

    A trophy engraved BEST, followed by a blank nobody filled in. Better at what, exactly? Evaluation compares behaviour against an expectation written down before the run, and every faster, cheaper check approximates that written judgment.

  2. PRUNING MY POTHOSNAME THE PROPERTY FIRSTCheck a property,not “quality”.QUALITY: 7.3moves without sayingwhich part movedDoes it parse?fields present,types rightIs it supported?by the sourceit citesDoes it decline?what it'smeant toRight register?the one the productcommitted to
    Each property fails differently and is checked by different means. One blended score hides which of them moved.
    PRUNINGMYPOTHOS.COM02 / 06

    A single quality score of 7.3 is crossed out: it moves without saying which part moved. Check properties instead. Does it parse, with the fields and types right? Is it supported by the source it cites? Does it decline what it is meant to? Does it hold the register the product committed to?

  3. PRUNING MY POTHOSFOUR WAYS TO CHECKFour checks,one judgment.1Deterministic assertionsexact, cheap, narrow2Heuristicsa tripwire, not a definition of correct3Model-based gradersjudgment at volume, same probabilistic kind4Human reviewthe one the other three stand in for
    Every cheaper check approximates a judgment someone made first. The judgment doesn't disappear when it's automated; it gets encoded.
    PRUNINGMYPOTHOS.COM03 / 06

    Four ways to check: deterministic assertions, exact, cheap and narrow; heuristics, a tripwire rather than a definition of correct; model-based graders, judgment at volume from the same probabilistic kind of system; and human review, the one the other three stand in for.

  4. PRUNING MY POTHOSA FIXED SET OF FAILURESKeep the badoutputs.the fixed setwrong factwrong tonemissed refusalmalformedtruncatedsure, on thin evidencechange somethingrerun the same setsee what movedonly your change movedfresh samples→the same set
    It isn't an estimate of production traffic. It's failures that happened at least once, kept so their return gets noticed instead of reported by a user.
    PRUNINGMYPOTHOS.COM04 / 06

    A jar of past failures: wrong fact, wrong tone, missed refusal, malformed, truncated, sure on thin evidence. Change something, rerun the same set, see what moved; only your change moved. Fresh samples are crossed out in favour of the same set. It is not a traffic estimate, just failures kept so their return gets noticed.

  5. PRUNING MY POTHOSMETRICS THAT CHANGE A DECISIONA number needsa decision.1What is measuredthe property, named2On what populationwhose runs, which cases3Over what periodfrom when, to when4How it was producedthe method, not only the result5Which decision changesif none would, drop the metric
    A figure quoted without its population and method hardens into a fact, and the conditions it was true under stop travelling with it.
    PRUNINGMYPOTHOS.COM05 / 06

    A retained metric should say what is measured, on what population, over what period, how it was produced, and which decision changes if it moves. If no decision would change, drop the metric.

  6. PRUNING MY POTHOSWHAT THIS CHANGESWrite down what“worse” looks like.worse =a confident answerwith no sourcebefore the change, not after
    Before the next change, describe behaviour you wouldn't accept, concretely enough that someone else could look at an output and agree it happened. Everything else checks that.
    Read the whole explanationpruningmypothos.com/systems/evaluation-is-a-human-problem/→PRUNINGMYPOTHOS.COM06 / 06

    A person writes a note: worse means a confident answer with no source. Write it before the change, not after, concretely enough that someone else could agree it happened. Everything else is machinery for checking that description.