A safety gate on our own agents had not run for four months, so we made it fail safe
Our own system · 2026 Q3
The situation
Our development framework runs AI agents that change our own software. Every change is ranked by risk, and a highest-risk change must pause, notify a person on Slack, and wait. We were preparing to run it unattended.
What it looked like
The configuration said that if Slack could not be reached, the run would halt and print to the terminal. The status screen reported no open escalations.
What was actually true
Slack had been off since the start of May, and the sending step returned quietly, which is why nobody noticed for about four months. The halt setting was read by no code: an escalation became one line in a log file, and the run carried on. The table for open escalations had never been written to, so the status screen could only say zero. That day, two highest-risk changes had merged: both reviewed by a person, neither through the gate.
How we found it
A handoff note from earlier that day stated the problem; we checked it against the code first. That narrowed it: the merge step did refuse highest-risk changes, but held them back silently and moved on. Our own first fix had an ordering fault that only running it exposed: a halted run still did start-up housekeeping, including a step that can close tickets. The halt is now the very first step.
What changed
We made the halt binding rather than switching Slack back on, because a delivered message nobody reads still lets the run proceed. Now a blocking escalation that cannot be delivered opens a record and stops the run until a person releases that one escalation by hand: no timeout, no release-all.
Where it ended
Done and verified. The real runner, on a copy of the database, halted, released on acknowledgement, and ran again. The tests caught deliberately broken versions of the gate. The fix was merged only after the approval was written on the ticket, not given in chat.
What we did not claim
That notification works. Slack is still off and the rules were not rewritten; the fix makes the existing rule true without Slack.
Why it matters beyond this case
Anything that runs continuously needs its boundaries written down before it runs — what may be settled without asking, and what has to come back to a person.