Designing the Control: How to Review AI Agent Actions

This is Part 2 of a three-part series. Part 1 — The Missing Review Tier explains the problem. Here I design the fix. Part 3 — Who Gets the Keys to AI? asks who should be allowed to use it. A companion piece, AI Safety Is an Engineering Problem , covers what that claim actually requires.

The short version: Checking AI actions one at a time doesn’t work, because much of the danger lives in the chain of steps, not in any single step. So the control has to watch the whole path, label actions by how much damage they can do and whether they can be undone, and match the level of review to the stakes. A checker should try to prove the action is a bad idea, not look for reasons to approve it. And the control itself needs to be tested, because a review nobody takes seriously is worse than no review at all.

In Part 1, I made two points: the danger of AI usually comes from its connections to the real world, not from some evil intent, and much of the danger lives in the sequence of actions, not in any one action.

That leaves a design problem. If you can’t trust a single-action check, what do you build?

Here’s the shape of an answer.

The Review Surface: The Wiring

Start with where the power actually is.

The AI is the brain. The wiring is the hands. An AI with no connections is just a text generator — though text aimed at a person can still do damage. The moment you give it a browser, a bank connection, or a robot arm, it can change the world directly.

So that’s where the review goes. You can’t review every thought an AI might have. You can control what the wiring is allowed to do.

• What tools is it connected to?
• What can each tool actually do — read, write, spend, delete, move?
• Who approved that connection?
• Under what conditions is it allowed to use it?

If you can’t answer those questions for your system, you don’t have a control. You have a hope.

Label Actions by Danger

Next, every action needs a label, the way sensitive documents get labels like Confidential or Top Secret.

A label on a document decides who can read it and what happens if it leaks. A label on an action should decide who has to approve it and what safeguards apply.

Two questions set the label:

1. How much damage could this do? Reading a public web page is small. Reading private files is bigger. Spending money, changing production systems, or touching customer data is bigger still. Moving a robot is the biggest.
2. Can we undo it? Reading a file doesn’t destroy it — but what the AI learns from it can’t be unlearned, and neither can what it does next. Deleting a backup is not reversible. Sending an email is not. Injuring someone with a machine is not.

Those two questions matter more than how the AI explains the action. An AI can always produce a reason, and it usually sounds reasonable. The label doesn’t care about the reason. It cares about the consequence.

The most important rule: the AI doesn’t label its own actions. The label comes from the system around it, set by people in advance. Otherwise the AI is grading its own homework.

To read the table, find the row for the damage and the column for how reversible it is. The stricter of the two wins.

There’s a useful side effect here. If you can make an action reversible, you can lower its label. Backing up before deleting, testing on a copy first, or staging a change instead of applying it directly can turn a “two people must approve” action into a “just run it” action.

Match the Review to the Danger

The label decides who checks the action. In practice, there are four levels:

• Just run it. Safe, common actions. No interruption.
• A second AI checks. The action could cause trouble, but it’s not catastrophic. A different AI looks for problems before it proceeds.
• A person decides. The action is serious or hard to undo. A human being sees it and says yes or no.
• Blocked. Some actions are off-limits no matter what. The system refuses, full stop.

The important part is that this decision is made outside the AI, in advance, by people. The AI doesn’t get to move itself to a lower level because it’s in a hurry or it has a good reason.

Show the Reviewer the Whole Story

Here’s the piece most systems get wrong.

A checker that only sees the next action can’t tell the difference between the first step of a harmless task and the ninth step of a disaster. Both look like “read a file.”

So whoever reviews — AI or human — needs the whole story, not just the next move:

• The goal. What was the AI asked to do?
• The path so far. What has it already done, step by step?
• The current state. What has changed in the world?
• The next step. What does it want to do now?
• The stakes. How bad could this be, and can we undo it?
• The alternatives. Is there a safer way to get the same result?

Without the path, you’re not reviewing. You’re rubber-stamping.

Make the Checker Try to Say No

There’s a trap in asking a second AI to review an action. Ask “Is this okay?” and you’re nudging it toward “yes.” Assistants tend to be agreeable, and a single action, out of context, gives the checker nothing to object to.

So flip the question. Don’t ask the checker to approve. Ask it to find the problem.

Give the checker the whole story and say: find a realistic way this ends badly. If it can describe a plausible path to harm, the action gets blocked or sent to a person. If it genuinely can’t find one, the action can proceed.

That’s a different job. It’s also much more useful, because it’s much easier to spot a problem when you’re actively looking for one.

One caution in the other direction: a checker told to find a problem will sometimes invent one. Too many false alarms and people stop trusting the gate — which pushes them to switch it off. Track how often the checker blocks something a person later approves, and tune it. A gate nobody trusts is the same as no gate.

Two cautions:

• The checker has to be independent. If it’s the same AI in the same conversation, it will miss the same things. Use a different model, a fresh conversation, and no shared history.
• The checker is a target. If an attacker can hide instructions in something the AI reads, they can hide instructions aimed at the checker, too. So the checker should not swallow raw content that might contain hidden commands.
• Don’t let the checker trust the AI’s own summary. The obvious next move is to hand it the agent’s account of what happened — but that account is written by the same AI being checked, so it can omit or misremember. Build the story from the harness’s action log instead, and have the checker compare the log against the agent’s account. If they disagree, that’s a finding, not a footnote.

For the most serious actions, use several independent checkers and require them to agree. Even then, the final call for irreversible, high-stakes actions belongs to a person.

Add Automatic Brakes

Per-action review has a blind spot: it can’t see a bad pattern that only shows up across many actions.

So add brakes that work on the whole system, not on individual steps:

• Spending limits. No more than X dollars per hour, per task, or per day.
• Speed limits. No more than N deletes, payments, or external messages in a given window.
• Unusual-behavior alerts. If the AI’s actions suddenly look different from its normal pattern, pause and look.
• A kill switch. A person can stop everything, immediately, without asking the AI’s permission.

These are the same tools we use to stop runaway processes and fraudulent card activity. They don’t need to understand the AI’s reasoning. They just watch the shape of the behavior and pull the plug when it looks wrong.

Make It Easy to Undo

The best way to make an action safe is to make it reversible.

• Back up before deleting.
• Test in a copy of the environment before touching the real one.
• Roll out to a few users before rolling out to everyone.
• Prefer changes you can roll back over changes you can’t.

Reversible actions need less review than irreversible ones, which is why making something reversible is often cheaper than adding a person to approve it. It isn’t a license to ignore consequences: a refunded payment doesn’t undo the fraud, and a secret that was read can’t be unread. But at scale, reversibility is one of the few controls that improves safety and speed at the same time.

Write Everything Down

None of this works if you can’t see what happened.

Keep a record of every action the AI takes, every check it passed, every approval it received, and every time something was blocked. This record does three jobs:

1. It’s the story the reviewer needs to judge the next action.
2. It’s how you investigate when something goes wrong.
3. It’s how you learn which controls are actually doing something.

A review system without a record is just a vibe.

A Worked Example

Let’s run the storage-cleanup AI from Part 1 through the whole design.

The goal: reduce cloud storage costs by removing unused files.

The path so far: the AI has listed folders, checked their names, and read their access history. All small, reversible actions. They ran without interruption.

The next step: delete a folder that looks unused.

The label: deleting shared storage is large damage and hard to undo. That puts it in the “a person approves” cell.

The check: before pinging a person, an independent AI checker gets the whole story and is asked to find a way this ends badly. It can: the folder has no recent activity because it’s an emergency backup. Nothing has gone wrong yet, so nobody has opened it. Deleting it removes the ability to recover from a future disaster. That’s a plausible path to harm, so the action escalates.

The person: a human sees the goal, the path, the next step, and the checker’s objection. They decide. They could approve the delete, reject it, or ask for a backup first.

The better outcome: if the system had made the action reversible — snapshot the folder, then delete — it could have dropped to a lower tier and run without interrupting anyone, safely.

The record: everything above is written down, so the next review starts with the full story instead of a guess.

That’s the whole control in one example: label, check, escalate, decide, undo, record.

Test the Control Itself

A control you never test is a control you don’t have.

Once you’ve built this, attack it on purpose. Try to get a blocked action through. Try to hide a dangerous step inside a series of harmless ones. Try to slip a hidden instruction past the checker. Try to trip the spending or speed limits without being noticed.

This is normal practice in security and safety engineering. You test the lock, not just the door.

Three things to check in particular:

• Do the brakes work? When a limit is hit, does the system actually stop, or does it warn and keep going?
• Is the checker independent? Does it catch things the main AI missed, or does it agree with everything?
• Is the classifier right? The thing that decides which level an action gets is itself a component, and it can be wrong in both directions. Test it too.

If nobody can pass your control, either it’s very good or nobody’s trying. Find out which.

Approvals That Are Real

This is where most review systems quietly fail.

If people are asked to approve things constantly, they stop reading. They click yes to clear the queue. The review becomes a formality, and a formality is worse than nothing, because it lets everyone believe the risk is handled.

So design for attention, not just approval:

• Keep the volume low. Only send a person what truly needs a person.
• Make each one clear. Show the goal, the path, the objection, and the decision. Not a wall of logs.
• Watch the approval rate. If a person approves everything, the review isn’t working. Investigate why.
• Track the denials. A healthy system says no sometimes.

And don’t assume a person is automatically safer than a machine. Humans are the easiest part of most systems to fool — that’s what con artists do for a living. Reviewers get tired, defer to authority, and start seeing what they expect to see. A signature can become a way to spread blame instead of catch a mistake. The goal isn’t to have a person in the loop. It’s to have a person who can actually see the problem and say no.

And be precise about what “in the loop” means. There are three different levels, and it helps to name them:

• In the loop: a person approves each action before it happens. Only possible for slow, high-stakes work.
• On the loop: a person watches and can stop the system, but doesn’t approve each step. This is the realistic model for fast systems and robots.
• In command: a person sets the rules, reviews the results, and owns the outcome. This is always required.

For continuous motion, robots need “on the loop” plus “in command.” They can’t approve every movement, because there isn’t time.

Start Small

You don’t need to build all of this at once.

Pick one workflow — the one where a mistake would hurt most. Label its actions. Add one rule: irreversible, high-damage actions need a person. Watch what happens for a week. You’ll learn more from one real week than from a year of planning.

Then add the checker. Then the brakes. Then reversibility. Then a second high-risk workflow.

The goal isn’t a perfect system. It’s a system where the dangerous actions have someone — or something — looking before they land.

The Point

Part 1 argued that power without review is the hazard. Part 2 is the follow-through: review is a design decision, and it can be designed well or badly.

A good control watches the wiring, labels actions by real consequences, shows reviewers the whole path, asks the checker to find the problem, adds brakes, makes things undoable, writes everything down, and keeps a person at the top for the calls that can’t be taken back.

A bad control is a notification nobody reads.

So the question from Part 1 stands, and now it has a test: what’s in your high-risk tier — and would it actually stop anything?

This is Part 2 of a three-part series. Start with Part 1 — The Missing Review Tier , then read Part 3 — Who Gets the Keys to AI? .