The robot had an answer

By ·

A restaurant robot in Cupertino stops serving, pounds on a table, and carries on dancing.

Three servers drag it out by a strap around its neck. Nobody seems to know where the switch is.

I went looking for who taught it that. The answer was already published: the company that built it says the routine is preprogrammed, and the space that night was tighter than the robot is meant to work in. Two sentences, and the behaviour has a history — a model, a version, a routine.

Ask an agent the same question and there is nothing at the other end. That is a missing input, and reliability engineering runs on it.

We have known how to build reliable things out of unreliable parts since von Neumann wrote it down in 1956, and every construction since works the same way: you take parts that fail, and use what you know about how often they fail to arrange them into something that does not. The arrangement is arithmetic. It runs on that rate. And the rate comes from provenance. You know how often a part fails because you know what it is.

There are measured error rates. There is no rate you can put in the arithmetic. The one it needs is for this component, on this input, today, and it moves with the prompt and the context. A rate measured on another distribution does not enter the calculation. It resembles it.

Ask the agent and it answers, fluently, and the number is worth nothing you can build on: the confidence is produced by the same process that produced the claim. Run three and take the majority, and three models trained alike on the same context are not three independent parts. Their mistakes arrive together, which is the one thing redundancy cannot survive.

Ours did it to us. A number our own system served said nine deploys out of nine were clean. The arithmetic was right; the table those deploys were read from had stopped being written. An empty window scores the same as a clean one, and the rate could not tell which, because it never knew where its input came from.

So the interesting problem stops being how good the agent is and turns into plumbing. When it cannot resolve something, is there anywhere to send it — a probe, a service, a person — and does the answer come back from something that never read the same input? That last clause is the whole test. Find the place where your system checks itself: if what checks it read the same context, you do not have a check. You have the same component answering twice.

Most of what is being built has no such place. The agent is handed a task and left to finish it, and more of what it is handed every month came from another agent a moment earlier. Same question, no better answer, one layer down.

The robot is the case where the answer exists, and it still went badly: three people and a strap, while a fourth looked for the control on her phone. Whatever worked, worked after the plates. That is interruption, not escalation.

An agent with nowhere to ask is not autonomous. It is unsupervised.