What Systems Thinking Helps You Inspect

Systems theory is a large field with a long history, and only a little of it pays for itself when you are looking at a production incident.

Systems thinking contributes a short list of questions worth asking about a running system, rather than a vocabulary for describing one. The list is short enough to hold in your head, which is the only reason it gets used.

Where the edges are

The first question is where the system stops.

Every component has a boundary, and the boundary is where responsibility changes hands. What crosses it is an interface, whether or not anybody designed it as one: a function signature, a database row, a queue message, a person deciding something.

A constraint is whatever limits what can cross. Some constraints are deliberate, and some are accidental facts about how the thing was built, and the difference matters because only one of them was chosen.

Locating the edges does much of the work of understanding a system, because behaviour comes from how the parts are connected at least as much as from any one of them. For AI applications specifically, that boundary question has its own page.

What the feedback does

Feedback is whether the result of something affects what happens next.

Where a deviation feeds back in a way that reduces it, the system corrects itself and small errors stay small. Where it feeds back in a way that amplifies it, the same small error grows, and the growth is often the first thing anyone notices.

The question worth asking is narrow: when this behaves badly, does anything about that fact reach the thing that caused it? Some systems have no path at all, which is an absence of a loop rather than a defect in one.

Three ways things degrade quietly

These three are worth recognising by name, because each leaves the system apparently working.

Drift. The system gradually stops matching the conditions it was built for. Nothing breaks. The world moved, the inputs changed shape, and the design assumptions quietly stopped holding.

A measure becoming a target. A measure that becomes a target stops being a reliable measure of the thing it originally stood for. Once people or software are optimising the proxy, the proxy improves and its relationship to what you cared about weakens. The number gets better and the thing gets worse, which is why a single improving metric is weak evidence.

Variety mismatch. When a system meets a wider range of situations than it has responses for, the excess arrives as edge cases and escalations. A rising edge-case load is usually not a queue problem. It is a mismatch between the world's variety and the system's, and it gets fixed by adding responses or by admitting a case out of scope.

The questions

Applied to something you are actually looking at:

• Where are the boundaries, and what crosses each one?
• Which constraints were chosen, and which are accidents of implementation?
• When something goes wrong, does that fact reach anything that could act on it?
• What has changed underneath since this was designed?
• Which number is being optimised, and what did it originally stand for?
• What proportion of the work is arriving as exceptions?

None of those are AI questions. That is the point: they were useful before any of this and they apply to a system whose behaviour nobody can fully predict, which is a category AI applications belong to.

What this changes

The next time something is behaving oddly and every component looks fine, try the first and third questions before opening the code. Whether the problem is at a boundary, and whether anything is feeding back, will usually tell you which part to open.