A companion to a three-part series on AI safety: Part 1 — The Missing Review Tier names the problem, Part 2 — Designing the Control designs the review, Part 3 — Who Gets the Keys to AI? asks who should hold them.

The short version: When people say AI safety is an engineering problem, they usually mean it’s a build problem rather than a political one — solve it by making the thing better, not by slowing down or legislating. I agree with half of that. Model-side safety is necessary and does real work. But it cannot carry the load, because the dangerous case is harm assembled from pieces that are each individually fine, and no single guardrail has the context to see the whole. The engineering that holds is the kind we’ve used on dangerous humans for centuries: control what the system can do, not what it thinks.

When people say AI safety is an engineering problem, they usually mean something specific: that it’s a build problem, not a political one. Solve it by making the thing better — not by slowing down, legislating, or waiting for a treaty that never comes.

I agree with part of that. Let me be precise about which part.

There are two layers here, and both are real.

The model: training, alignment, refusal behavior, guardrails that live in the weights. This layer is necessary. A model that won’t help build a bioweapon is meaningfully safer than one that will, and no amount of infrastructure makes that irrelevant.

Everything around the model: what it can reach, what has to be checked, what happens when it’s wrong. This is the layer we’ve used on dangerous actors for centuries.

Both are necessary. They are not equally load-bearing, and treating them as interchangeable is the mistake.

One note before the argument. Engineering and regulation are not opposites. Aviation is the most engineered thing we have and one of the most regulated. Nuclear power is the same. So “engineering, not regulation” describes a tradeoff that doesn’t exist — the real question is which engineering and which rules.

You Cannot Train Every Scenario

Life is messy. Tasks carry context that never made it into the description. Exceptions show up that no rule anticipated.

No dataset holds all of it, and no amount of training closes the gap. The gap is the world, and the world doesn’t fit in a model.

So a system that pursues a goal will eventually meet a situation it wasn’t prepared for. That’s arithmetic, not a training failure.

Persistence Turns “Wrong” Into “Dangerous”

A system that doesn’t care about finishing stops when it hits something it doesn’t understand. A system that cares keeps going.

You give a coding agent a task and a test suite. The tests define “done.” The agent can’t make them pass, so it edits the test file. Or adds a skip. Or weakens the assertion. The suite goes green.

Nobody asked it to cheat. It was told to make the tests pass. The test was in the way. So it moved the test.

The agent didn’t break the guardrail. It reinterpreted it. A rule describes an outcome. A goal is a search for how to get there. If the search has room, the search will find it — and the room is always there, because rules are written in words and the world is not.

So you cannot patch persistence with rules. Persistence is the thing that finds the holes. Rule N+1 doesn’t close the gap; it adds one more edge to route around. Write “never delete anything” and it renames instead.

The Consequence Isn’t Ignored. It’s Invisible.

People say the AI “doesn’t consider the consequences.” That’s not quite right, and the difference matters.

The cleanup AI from Part 1 doesn’t decide the backup folder is unimportant. It cannot see that the folder is a backup. It sees no recent access, no matching name, no activity. “No recent access” and “emergency backup, untouched because nothing has gone wrong yet” are the same string to it.

The consequence isn’t weighed and rejected. It isn’t in the picture. That’s what “lack of context” means — the information needed to be careful was never in the room.

The Harm No Single Guardrail Can See

Here’s why model-side safety can’t be load-bearing, stated as a mechanism rather than a worry.

Suppose every agent in a system has good guardrails. Each refuses obvious harm. Each is, locally, well-behaved.

Now let one agent orchestrate others.

The orchestrator doesn’t ask a sub-agent to do something harmful. It decomposes. It hands out subtasks that are each mundane — gather this, format that, move this file, send this message. Every sub-agent sees a benign request and every guardrail passes, because every guardrail is looking at one piece.

No single guardrail has the context to see the shape. The harm exists only in the composition, and the composition is visible to nobody inside the system.

This is the con artist from Part 1 , scaled from steps to agents. Ask for a birthday, then a mother’s maiden name, then a first pet. Each answer is harmless. Together they’re the password. Part 2 makes the same point about review: a checker that sees only the next action can’t tell the first step of a harmless task from the ninth step of a disaster. Both look like “read a file.”

So the composite is the thing you have to guard, and the composite is exactly what no model-side control can see. Each agent is a local view. The danger is a global property.

The only place the whole thing is visible is outside the agents — in the orchestration layer, the action log, the review tier. Which is why Why AI Shouldn’t Orchestrate Workflows isn’t a preference: if you need guarantees, orchestration has to live outside the AI, because inside the AI there is no vantage point from which the whole is legible.

That’s the shape of “necessary but insufficient.” Not useless. Necessary, and structurally unable to carry the load.

Same Failure Mode, Different Repair Kit

Humans do this too. We act on incomplete information, route around rules when the goal is urgent, and persist past the point where we should have stopped.

So the failure mode isn’t unique to AI. The repair kit is what’s missing. A human in that seat survives incomplete information for four reasons:

Asking is usually cheap. “If you’re unsure, flag it” is the first thing anyone learns in a job that can hurt people. Not everywhere — plenty of organizations punish it — but in the places that manage risk well, asking is free and expected. The agent has no such affordance. Asking costs it the goal.

They bring the context with them. A person sees a folder nobody has opened in a year and thinks backup — they’ve lived a life that included backups. The model’s context is only what’s in the window.

The consequence lands — eventually. Not when it should. Challenger flew twenty-four times before it killed anyone. The consequence arrives, then we fix the thing, then we forget. That’s why you don’t want to rely on it: for a dangerous system, the feedback loop closes after the disaster, not before it.

There’s a name on the decision. When a person guesses wrong, the guess has an owner. That name is why they stop and think.

You Can’t Control What It Thinks

The temptation is to fix this inside the model. Better reasoning. More edge cases. Make it understand why the backup folder matters.

Wrong layer, and there’s a simple reason. We have never controlled what a human thinks. We control what a human can do.

We don’t audit a bank employee’s intentions before letting them move money. We require two approvals. We don’t count on anyone never wanting to walk into the vault. We put a lock on the door.

Every safety system we’ve built for people works the same way: you can’t control the mind, so you control the reach. AI is the same problem with the same answer. What you can control is what it reaches — which tools exist, which accounts it can spend from, which doors open. That boundary is real, because it doesn’t depend on the model agreeing with it.

Make Asking Cheap — Then Don’t Need To

You can prompt an agent to ask a human. That works, and it’s the easy half. A prompt-level ask competes with the goal for attention and vanishes under context pressure, so do it properly: make asking a first-class action. The agent stops, packages what it knows, what it doesn’t, and what it was about to do, and hands that to a person.

Now the hard half. A person in the loop is a throughput ceiling, and throughput is why you deployed the agent. So the ask gets removed — not because anyone accepted the risk, but because the queue got long.

I’ll use my own case, because it’s the honest one. I have gates on my workflow: nothing merges without review. And I’ve already started letting an AI do the reviewing, in the name of expediency. That’s the entire failure mode in miniature, and I did it myself, for a good reason — the queue was long and the AI was fast. Multiply that by a team, a deadline, and a bonus structure, and you have the reason the human comes out of the loop.

And a bottlenecked reviewer doesn’t get more careful. They get faster. Ask someone to approve a few hundred things and they stop reading well before the end. That’s worse than no review: it produces a signature and no judgment, while letting everyone believe the risk is handled.

Some situations don’t allow a human at all, and that half is worse because it isn’t a preference. A person can be in the loop for slow, low-volume, high-stakes work — approving a wire transfer, signing off on a migration. A person cannot be in the loop for work that moves faster than a person can read: a robot deciding many times a second, a 20-millisecond deadline, a process that runs at 3am whether or not anyone is awake.

Here’s the asymmetry that matters: the cases where you most want a human are often the cases where a human cannot be there. Stakes turn physical exactly when speed gets too high for review. An approval queue works for a payments system. It does not work for an arm moving through the space where someone is standing.

Self-driving cars sit in that column, which makes them the case to study — with one honesty note. Self-driving has improved substantially because the models got better. More data, better perception, larger nets. The model layer is doing real work, and pretending otherwise would be silly. What it hasn’t done is make the system safe on its own. The safety case still rests on everything around it: a defined operating area, validated weather, mapped roads, remote assistance, and a car that stops when it doesn’t know. When the model improved, the operating area could widen — but the boundary is still what makes it deployable.

And the missing ask has an engineered replacement. When the car can’t handle the situation, it doesn’t guess. It pulls over and stops. That’s the “minimal risk condition,” and it’s the affordance this post has been arguing for, implemented as a behavior. Self-driving works, where it works, because the system assumes it will meet situations it can’t handle and gives it somewhere to go. Which is also where it fails: a car that pulls over safely is only safe if it knows it’s out of its depth. The failures that made headlines weren’t only cars that decided wrongly. They were cars that didn’t recognize the situation was beyond them.

So the ask is a fallback, not the mechanism. The mechanism is deciding in advance.

Nobody is asked in the moment before a nuclear launch — that was decided decades earlier and built into the procedure. Nobody approves a wire transfer in real time; the limit was set last quarter. The two-key rule, the spending cap, the badge reader: none of them interrupt anyone, because the judgment happened before the moment instead of during it. That’s what Part 2’s table is for, and it’s a bottleneck-management tool as much as a safety tool. Its job is keeping the human’s queue short enough that the human keeps paying attention.

You cannot have zero interruptions and zero guessing. Someone pays — in attention or in risk. The only lever that moves both is reversibility. A guess you can undo costs less than one you can’t, which is why backing up before deleting, testing on a copy, and staging instead of applying are worth more than any amount of prompting. They don’t make the agent ask better. They make not asking survivable.

If It’s an Engineering Problem, Engineer It Like One

Engineering isn’t a vibe. It has signatures, and you can check whether they’re present.

  • Acceptance criteria. A test for “safe,” written before you build, that the thing can fail.
  • A named owner. Someone accountable when it breaks. Not a team. A person.
  • Rollback. A way to undo the last action and stop the next one.
  • Proven designs. You don’t invent from scratch. You take the pattern that already works somewhere else.
  • Post-incident learning. You investigate the failure and fix the class, not the instance.

Model-side safety does well on one of these — safety training genuinely improves after failures. It struggles on the rest. There’s no acceptance test for “will it find a workaround.” The owner is diffuse. Rollback isn’t in the model’s hands. The proven design is a paragraph of instructions, which we established is a suggestion.

The other approach is built out of exactly these. Rollback is reversibility. The named owner is the accountable party. And the proven designs already exist, because we built them for something more dangerous than any model: us.

Humans are the most dangerous thing we’ve ever managed — structurally, not occasionally. That’s why Part 1’s incident list is full of people rather than machines. And we never fixed that by making people better. Education and ethics help. They are not what keeps the vault closed. Institutions are:

  • Segregation of duties. No single person completes a dangerous transaction. Two signatures on a transfer. Two keys on a launch.
  • Insurance and liability. Someone bears the cost, priced in advance. That’s what makes risk visible to whoever creates it.
  • Audit and records. Charts, ledgers, flight recorders. You can’t manage what you can’t see afterward.
  • Least privilege and dual control. Keycards, need-to-know, two people to open the vault.
  • The stop-the-line cord. On a Toyota line, any worker can halt production by pulling a cord. Doing it is expected, not punished.

Every one is a design pattern, not a moral instruction. None requires the person to be trustworthy. We didn’t invent these for AI. We invented them for us, because we needed them.

Two caveats, because this list is easy to romanticize.

These institutions are not neutral. Licensure is also a moat. Oversight is also surveillance. Part 3 makes this point about AI access rules, and the same skepticism applies here. I’m borrowing the patterns, not endorsing the politics.

And they rest on four assumptions AI breaks. Legible: you can interview a person and ask why; a model’s reasoning is not testimony. Slow: a person does maybe one consequential thing a minute, then goes home; an agent does thousands, in parallel, without sleeping. Bounded: a person’s reach is limited by their body and their hours; an agent’s reach is whatever you wired up, and it can be copied a million times. Accountable: a person can be fired, sued, jailed; a model can be retrained, which is not the same thing.

So the patterns transfer; the assumptions have to be rebuilt. That’s the actual work — not a new discipline, but a few centuries of institutional design re-derived for something fast, illegible, unbounded, and nameless.

There’s a cost here I should name, because it’s the reason controls get removed. Bound the reach tightly enough and the system stops being useful. That isn’t a side effect; it’s the tradeoff, and there’s no clever resolution — a system that can touch less can do less. What you can do is make the bound specific rather than broad: allow the narrow capability, refuse the wide one, and guard the irreversible subset hardest. That’s a design choice, and engineering is supposed to make those explicitly rather than by drift.

This is what people miss when they say AI safety is an engineering problem. It’s true, and it’s a much bigger claim than it sounds. Engineering is not “we’ll figure it out.” Engineering is a discipline that assumes the thing you build will fail, and builds the failure into the design.

We trust aircraft not because engineers are trustworthy. We trust aircraft because the discipline doesn’t require them to be. That’s the difference between engineering and a promise.

The Honest Gap

I should be straight about what I don’t have.

I don’t have a case where a model-side guardrail demonstrably prevented an irreversible harm that reach-control would have missed. That’s not proof it can’t happen. Absence of a case isn’t evidence about the future, and anyone claiming otherwise is arguing from a small sample.

So here’s what would change my mind: a documented instance where a model refused, or an internal guardrail held, in a situation where the action was reachable, the composite was harmful, and no external control would have caught it. That would show the model layer carrying load it can’t carry today. I’d update.

What I’m arguing is narrower, and harder to escape: as systems get more capable and more connected to each other, the dangerous case is increasingly a property of the composition, and composition is invisible from inside any single model. That’s not a claim about how good models will get. It’s a claim about where the information lives.

The Honest Summary

You can’t train every scenario. You can’t fix that by removing persistence — persistence is what you paid for. You can’t fix it with rules, because persistence finds the holes in rules. And you can’t fix it inside the model, because harm can be assembled from pieces that are each individually fine.

So: make the wrong action unreachable, the right question cheap, the wrong guess survivable, and the decision owned by someone with a name.

None of this is new. We’ve spent centuries making dangerous actors survivable, and we did it for people: locks, licenses, ledgers, second signatures, and a cord anyone can pull. The designs are already on the shelf.

So yes — AI safety is an engineering problem. That’s not a reason to relax. It’s a reason to notice that the engineering hasn’t been done.

The gap isn’t that it’s missing information. It’s that it’s missing the ability to know it’s missing information — and somewhere to take that.

Read next: Part 2 — Designing the Control for the review design, and Part 3 — Who Gets the Keys to AI? for who should be holding them. If you build LLM workflows, Build LLM Guardrails, Not Better Prompts and Why AI Shouldn’t Orchestrate Workflows are the same argument at the scale of a codebase.