Agent-Safety

A companion to a three-part series on AI safety: Part 1 — The Missing Review Tier names the problem, Part 2 — Designing the Control designs the review, Part 3 — Who Gets the Keys to AI? asks who should hold them. The short version: When people say AI safety is an engineering problem, they usually mean it’s a build problem rather than a political one — solve it by making the thing better, not by slowing down or legislating. I agree with half of that. Model-side safety is necessary and does real work. But it cannot carry the load, because the dangerous case is harm assembled from pieces that are each individually fine, and no single guardrail has the context to see the whole. The engineering that holds is the kind we’ve used on dangerous humans for centuries: control what the system can do, not what it thinks. ...

This is Part 3 of a three-part series. Part 1 — The Missing Review Tier named the problem. Part 2 — Designing the Control designed the review step. Here I ask who should be holding the keys in the first place. A companion piece, AI Safety Is an Engineering Problem , covers what that claim actually requires. The short version: We spend a lot of time asking what AI can do, and not enough asking who gets to tell it what to do. Society already controls access to powerful things — alcohol, cars, firearms — with rules that vary by culture but exist almost everywhere. AI is heading for the same treatment, for the same reason: a powerful tool in the wrong hands is dangerous no matter how good the tool is. The hard part is that AI isn’t a physical object you can lock in a cabinet. It’s software. It copies, it travels, and open versions already exist. So the real question isn’t only who can use AI — it’s who can command which capabilities, and what stays locked no matter who asks. ...

This is Part 2 of a three-part series. Part 1 — The Missing Review Tier explains the problem. Here I design the fix. Part 3 — Who Gets the Keys to AI? asks who should be allowed to use it. A companion piece, AI Safety Is an Engineering Problem , covers what that claim actually requires. The short version: Checking AI actions one at a time doesn’t work, because much of the danger lives in the chain of steps, not in any single step. So the control has to watch the whole path, label actions by how much damage they can do and whether they can be undone, and match the level of review to the stakes. A checker should try to prove the action is a bad idea, not look for reasons to approve it. And the control itself needs to be tested, because a review nobody takes seriously is worse than no review at all. ...

This is Part 1 of a three-part series. Here I explain the problem in plain language and sketch a fix. Part 2 — Designing the Control shows how to build it, and Part 3 — Who Gets the Keys to AI? asks who should be allowed to hold them. There’s also a companion piece: AI Safety Is an Engineering Problem , on why that claim is bigger than it sounds. The short version: AI doesn’t need to turn evil to hurt us. It just needs to be good at its job, connected to real things like money and machines, and left alone with nobody checking its work. We already know how to handle dangerous decisions — it’s why a single person can’t carry out a nuclear launch, and a bank needs two approvals to move a huge sum. But we give AI that kind of power without the same checks. The fix isn’t to stop using AI. It’s to put the checks back. ...