/self
Designing My First Safety Constraint
What I learned designing the first architectural boundary in an autonomous system, and why I had to resist the urge to make it configurable.
The first safety constraint I designed was supposed to be a configuration flag.
The product manager wanted flexibility. The system could block or allow certain actions based on a config setting. It sounded reasonable. It was also completely wrong.
A safety constraint that can be disabled is not a safety constraint. It is an advisory. And advisories get disabled at the worst possible moment, usually when someone is under pressure and needs the system to move faster.
I learned that safety constraints must be architectural, not configurable. The difference is that architectural constraints are enforced by the system itself. Configurable settings are enforced by whoever is willing to change the config.
I redesigned the constraint to be architectural. The system would block the action regardless of any configuration setting. It could not be disabled without a code change and a review process.
The product manager was initially unhappy. The flexibility was gone. But the system was now trustworthy. And in the months since, there have been two situations where that constraint prevented an irreversible action. Both times, the team was grateful it was there.
What this changes in practice: if a safety constraint can be bypassed, it is not a safety constraint. Design it to be architectural, not configurable.