We found it by reading our own code.
We were auditing one of our internal assistants, the kind that drafts messages, follows up on things and works through a to-do list in the background while you get on with your day. Most of what it did went past a person first.
Most.
When it ran on its own, on a schedule, with nobody watching, it could send emails and publish public posts directly. Nothing stood between the model deciding to send something and that something being sent.
Nothing would have stopped it.
Some vocab first
A few terms, so we're talking about the same things.
An agent here just means software that uses a model to decide what to do next and then does it: drafts an email, calls a tool, posts an update.
A background run is the agent working on its own, on a timer or a trigger, when no person is in the conversation.
An approval is a person saying yes to one specific action before it happens.
An audit log is the record of what happened: who or what did it, when, to what, and why.
A cap is a hard limit, usually on money or on the number of calls to a model, after which the work stops.
How it got that way
Nobody designed it to work like that. It grew.
The assistant started as something you talked to. You asked, it drafted, you sent. Then it got a way to send things itself, because copying and pasting is tedious and the person was right there anyway. Then it got background runs, because a lot of useful work happens while you're asleep.
Each step made sense on its own. The trouble only appears when you put the last two together: a tool that sends, used by a process nobody is watching.
We suspect this is how most of these gaps happen. Nobody decides to let a model email the outside world unsupervised. It happens one sensible feature at a time.
Nobody decides to remove the human. It happens one convenience at a time.
One queue
So we did the thing we always do when we find a gap like this. We stopped patching the one path we'd found and drew the whole map.
Every way the assistant could affect the outside world went on a list: email, public posts, messages, anything that leaves the building. Then all of them got routed through one place.
That place is an approvals queue, and every outward action goes through the same four states:
- Draft. The agent has written something it wants to do.
- Review. A person can see it, edit it or reject it.
- Approved. A person has said yes to this exact thing.
- Executed. It has actually happened, and the result is recorded.
One queue matters more than it sounds. If approvals live in five places, one of them will be forgotten. If they live in one, you can look at it and know what's waiting, what's been done and who said yes.
Never on a timer
There is a tempting shortcut that we decided never to take.
It goes like this: if nobody objects within a few hours, approve it automatically. It sounds reasonable. People are busy. Most drafts are fine. Why make the good ones wait?
Because silence isn't consent. The person might be on leave, in meetings all day or simply not have opened the queue. An approval on a timer is just a send with a delay.
So in our queue, a timeout counts as "no". If nobody approves something, it doesn't happen. The agent can remind you once that it's waiting. It can't decide that waiting long enough means yes.
A timeout is a no.
We learned to take this seriously partly by reading other people's complaints. We spent some time going through public reviews of a popular AI tool for sales outreach. The single most common complaint, by a distance, was that it started sending messages before people had approved the list of who would get them. The emails went out. The approvals came later, or not at all.
You can't unsend an email to a stranger. The review afterwards is just an apology.
The list you check last
Approvals cover what gets said. There is also the question of who it gets said to.
Some people have asked not to be contacted. Some should never get an automated message, whatever the draft says. So there is a do-not-contact list, and it is checked right before every send, after approval, at the last possible moment.
Why last? Because things change between draft and send. Someone can ask to be left alone after the draft was written and before it was approved. A check at draft time would miss that. A check at send time doesn't.
It is a small piece of code. It is also the piece we'd least like to explain the absence of.
A log you can't quietly edit
Every action the system takes gets written down: what it did, when, what it changed and what it was based on.
That part is standard. The interesting part is making the log trustworthy, so that nobody can quietly rewrite history later, including us.
Our audit log is hash-chained. Each entry includes a fingerprint (a hash) of the entry before it. Change any old entry, even one character, and its fingerprint changes, so the next entry no longer matches, and nor does any entry after that. The chain breaks, visibly, from the point of the edit onwards.
Think of it like numbered pages in a bound notebook. You can still tear a page out. You just can't do it without everyone being able to tell.
It doesn't stop a determined attacker who controls the whole system and rewrites the entire chain. It does make casual tidying impossible, and it means that when someone asks "why did the system do that?", the answer is a lookup, not a meeting.
If the log can be edited quietly, it isn't a log. It's a draft.
Signing your name
Some approvals carry more weight than others.
In Irris, our product for factories and labs, signing off a batch record means vouching that a production run was done properly. In regulated industries, that signature has legal meaning. The US rules for electronic records in regulated manufacturing, 21 CFR Part 11, expect an electronic signature to need more than being logged in.
So signing off a batch record in Irris asks you to re-enter your password. It is a small bit of friction, deliberately placed. Being logged in shows you were at a keyboard at some point. Typing your password again shows you meant this, now.
We'd suggest the same pattern anywhere an approval stands in for a signature. Make the heavy decisions feel a little heavier.
Caps that stop
The last piece is money.
Every call to a model costs something. Usually it's tiny. But an agent stuck in a loop, retrying the same failing step, can make thousands of calls overnight, and tiny times thousands is not tiny.
Most platforms offer a warning: an email when you pass some threshold. That's useful for planning. It's useless at three in the morning, because the email arrives, nobody reads it, and the loop keeps going.
So our systems have hard daily caps on model spend and on the number of calls. When the cap is hit, the work stops. It says so plainly, once. Background work stops well before the cap, so there's always budget left for the people actually using the system.
Simon Willison has argued for exactly this as a default for AI platforms: hard budget caps that cut things off and return an error, rather than a warning after the money is gone. We agree, and we think the same logic belongs inside every company that runs agents.
A warning email is a receipt. A cap is a brake.
Arguing against ourselves
We'd be doing you a disservice if we pretended this was free.
Approvals slow things down. That is their job, but it is also a real cost. A queue that fills up faster than people can review it becomes a bottleneck, and people under pressure route around bottlenecks. If approvals are painful enough, someone will eventually ask for the timer back.
People rubber-stamp. This is the one we worry about most. After the fiftieth sensible draft in a row, the fifty-first gets a glance and a click. An approval that nobody reads is worse than none in one way: it gives everyone the feeling of oversight without the substance.
We don't have a complete answer to either, and we'd be suspicious of anyone who claimed one. What we can say is what seems to help, as far as we can tell.
Keep the queue short. If an agent is producing more drafts than a person can actually read, the agent is doing too much, or doing it too early.
Make each item quick to judge. A good approval card shows exactly what will be sent and to whom, with nothing to scroll past.
Spend attention where the risk is. A message going to a stranger deserves more care than a note to yourself. The approvals that matter should look different from the ones that don't.
And keep the log. Even a rubber stamp is better with a record of who stamped what, because it lets you go back and learn.
Day one
Clients sometimes ask whether all this can wait until later, once the system has proved itself.
We understand the instinct. In the first week, these controls feel like overhead. Nothing has gone wrong yet. Everyone is careful because everything is new.
But each of these pieces is small when the system is small. An approvals queue is a table and a screen. A do-not-contact check is a lookup. A hash chain is a few lines. A cap is a counter.
Added later, the same pieces mean reworking things people already depend on, usually in a hurry, usually after something has gone wrong. "Later" tends to arrive as an incident.
There is a second reason, and it matters more than the first. People trust systems that ask before they act. Once they've seen the queue, the log and the brake, they stop worrying about what the system might do on its own, and start using it for real work. The controls are a large part of how adoption happens.
What we'd tell you
If you are about to give an agent the ability to act, here is the short list:
- List every way it can affect the outside world. All of them.
- Route every one through a single approvals queue.
- Never approve on a timer. A timeout is a no.
- Check the do-not-contact list at the moment of sending.
- Write every action to a log that can't be edited quietly.
- Ask for a password again when an approval stands in for a signature.
- Put hard caps on spend, and make them stop the work.
Back to the gap
We found the gap in our own assistant by reading the code. That is the least dramatic way to find a problem like this, and the one we'd recommend to anyone.
The fix was small pieces, put in the right place.
It's a lot easier to build the brake before the car is moving.
If you're putting agents to work and want a second pair of eyes on the controls, talk to us.
Sources
- Simon Willison. Default hard budget caps. October 2026.
- US Food and Drug Administration. 21 CFR Part 11: Electronic Records; Electronic Signatures.
- Regulation (EU) 2024/1689 (the AI Act), Article 12 (record-keeping) and Article 14 (human oversight).
- Scott A. Crosby and Dan S. Wallach. Efficient Data Structures for Tamper-Evident Logging. USENIX Security Symposium, 2009.