What a self-healing system can't do
By Daniel Farré Manzorro ·
First the model could write the code. Then the agent could run it — call the API, run the migration, move the money.
Now the promise is a system that maintains itself, with no one watching.
Well, a system that changes itself but fails to explain each change, and why it took place, will eventually become unaccountable: the Nth iteration may appear to have fixed an issue, as if it self-healed from some unexpected external hits, and maybe use-case behavior is just fine at that point, but at what cost? How do you know whether it really improved the internals and stabilized the system, or just applied some ill-justified workarounds? Most engineering leads have said some version of this before going on leave: bother me on holiday rather than hand me a snowball of issues when I get back.
Self-healing complex systems are feasible today - I won't pretend otherwise - but, as long as they are prone to erratic failures, they need to be continuously auditable, accountable: on the bad morning after something moved that shouldn't have, the system must succeed in answering who did that, on what evidence, and who signed off; and if some of the internals were compromised, that must show up on reading the audit trail. Ideally, the system should notice any internal issue that it couldn't solve by itself and notify immediately.
AI agents do not emit audit trails of their activities. At most, they can explain the language and coding steps they took to proceed, including some evidence and reasoning that, when accumulated, become unreadable. The same applies at the codebase version-control level: maybe that lead who went on vacation will correct, on return, a series of commits without any chronological examination, so that developers won't get their feedback to confront similar future situations, and the technical director won't have a detailed explanation of the causes. An audit trail must be end-to-end-consistent with the system, and permanent — it is evidence that an organization can trust.
Audit trails, though, do not stop a bad action, or decide whether a migration should run or the money should move — that judgment sits where it always sat, with a person who can be held to it. What the system can do is narrower, and it's the part usually skipped: making sure that when someone asks what happened, a complete answer is provided, and it's readable, reliable, and doesn't depend on who still remembers.
Proof is not prevention, but the floor prevention has to stand on. And that floor is what gets scarce; the action is getting cheap; the account of it is not. So that's the layer worth building — and the only one that survives the morning after.