What to Record So You Can Explain What Happened

A week after launch, someone asks why the system did that last Tuesday. The uncomfortable part is not that the answer is complicated. It is that the answer no longer exists anywhere, because the run left behind an output and nothing else.

What observability is

Observability is the ability to investigate system behaviour from signals the system preserves while it runs.

That framing puts the emphasis in an awkward but accurate place. Observability is not a volume of logs and not a product you install. It is a decision, taken before anything goes wrong, about which signals will still be there when a question arrives that nobody anticipated. Logs are one input to that. The property being described is what you can find out afterwards.

For systems that generate text, the gap is specific. The output is the part everyone sees, and it carries very little about how the run reached it. A fluent answer looks the same whether the right document was retrieved or a plausible one was, whether a permission check ran or was skipped, whether a tool returned what was wanted or something that was never examined. What distinguishes those cases is not in the text.

Four words that are not synonyms

These get used interchangeably, and each answers a different question.

Monitoring watches signals chosen in advance and reports when one of them moves. It is good at telling you something changed and structurally unable to tell you about a failure nobody thought to watch for.

Observability is the property above: whether an unanticipated question can be answered from what was kept.

Audit preserves evidence under a defined control model. The distinguishing part is the control model rather than the evidence. Who may read it, how long it is retained, whether it can be altered after the fact and whether that alteration would be detectable. A trace that engineers can edit may be entirely adequate for debugging and still not satisfy audit.

Replay rebuilds a past execution from the events that were recorded of it. It is worth being exact here, because the word invites a stronger reading than it can carry. Replay can reconstruct a recorded execution only to the extent that relevant events and state were preserved. It does not guarantee the same model output, the same external state, or reconstruction of information that was never recorded.

Three separate limits sit inside that. Rerunning a probabilistic model need not reproduce the same output. The systems a run touched may have changed since, so external state at replay time is not guaranteed to be the state the run met. And a field nobody captured cannot be recovered afterwards by any amount of replaying, which makes replay a consumer of the recording decision rather than a substitute for it.

What the record has to answer

A record earns its place if it can answer three questions without anyone reconstructing them from memory.

1. What was decided?
2. What evidence supported the decision?
3. What changed since the last run that behaved correctly?

The third depends on something the other two do not: knowing which earlier run behaved correctly, and having kept enough of it to compare against. It is also the one that turns an investigation into a bounded search, because without it the search starts from the whole system.

A minimum record

The decision itself and the verification of its effect are made elsewhere. Authorization asks whether an action may proceed, and effect verification checks whether the intended change actually occurred; both belong to governed execution. What this page adds is that their outcomes are worth keeping in a form you can query across many runs, because a single run tells you what happened and a series tells you what is happening.

For each governed step, a compact record holds up well:

• run_id, step_id, parent_step_id
• the policy decision and the version of the policy that produced it
• the selected tool or action, and the parameters it was scoped to
• the verification result, and identifiers for whatever artifact proved it
• timing, and retry count
• a reason code from a fixed vocabulary when something failed

Prompt text and model output are deliberately not in that list. They are among the most sensitive material a run touches and the most expensive to retain at volume, and the three questions above can be answered without them. Keep them if you need them, but keep them as a separate decision with its own retention rule, rather than by default because they were easy to capture.

The reason codes deserve the fixed vocabulary. Free-text failure descriptions cannot be counted, so the same failure appearing forty times reads as forty separate incidents.

A specimen, and what it does not show

This repository carries a small artifact worth reading as a design specimen. This repository keeps a historical operations ledger of five records, written on one day inside a span of about 91 minutes, in which the human decision is stored as a separate field from the publish status. It documents a predecessor site built on a different stack, not the current publication, so nothing in it describes how anything runs today.

Two things in its schema are worth copying. Separating the human decision from the publish status means a record can show work completed and approval still outstanding, which is exactly the state that gets lost when one field is made to carry both. And each record holds the undo path for that specific change, written at the time rather than reconstructed later under pressure.

One thing in it is worth pausing on, and it is the more instructive half. The evidence fields describe produced artifacts or outcomes in prose, while information about validation checks is recorded separately in checks_run. Two of the five go further and describe the outcome in evaluative terms, calling a layout stunning and a contrast perfect.

Nothing here says the schema is wrong, and this is one small specimen rather than a lesson about schemas in general. The narrow point is about the label. A field called evidence holding the word stunning is a value whose name asserts more than its content supports.

A record of what happened is not the same thing as a judgment about whether it was good. The moment those two are stored in the same field, the record starts asserting the judgment, and whoever reads it later inherits an opinion formatted as a fact. Whether the behaviour was good is a separate question with its own machinery.

What changes underneath you

For teams running their own trained models, drift has a settled meaning involving retraining and held-out sets. For teams calling a hosted model, the sources of change are different: the provider updates the model, someone edits a prompt or a template, the retrieved corpus changes, or the shape of incoming requests shifts.

Observability does not explain any of that. What it does is make the change locatable, by keeping the policy version, the template version and the model identifier next to the behaviour they produced. Without those, a change in behaviour can be observed and not attributed, and the investigation ends in a plausible story.

Signals worth watching, without treating any of them as a diagnosis:

• retries rising while the inputs stay stable
• more actions permitted, with no matching rise in verified success
• tool calls succeeding and users correcting the result afterwards
• behaviour shifting soon after a policy or template change
• one workflow segment falling back to manual handling repeatedly

Each of those is a reason to go and look, and the looking is only possible if the record was there first.

What this changes

Decide what a run leaves behind while you are designing the run, not after the first incident. The instinct to add logging later is reasonable and it arrives too late for the specific thing you will want, because the question that matters is the one nobody anticipated.

Start with one flow that would be expensive to get wrong, record what the list above names for every governed step in it, and check that you can answer the three questions from the record alone. What that gives you is the ability to reconstruct what happened. Whether what happened was any good is the next question, and it needs a different apparatus.