We wave through a third of the malicious requests (external link)
Alex Wauters built a browser game. You get sixty seconds and a stream of simulated permission requests from a coding agent, and you approve or deny. Forty thousand runs later, across roughly 409,000 commands, the results are in and they are not flattering.
Players approved about a third of the malicious requests. Scope violations —
the agent reaching for AWS credentials or Kubernetes config when it had no
business there — were missed 35% of the time, the worst category. The single
most-approved dangerous command was npm run analyze, waved through nearly 65%
of the time, because it looks like something you have typed a thousand times and
you cannot see what is in the script without going and looking.
The commands people caught reliably were the theatrical ones. rm -rf / gets
denied. Of course it does. It looks like what it is.
The number that should worry you is not from the game at all. Wauters cites Anthropic's own telemetry: users approve around 93% of permission prompts. That is the whole problem in one figure. A control that is exercised 93% of the time in one direction is not a control, it is a formality with a keystroke attached. And the more prompts you see, the less each one means — which makes the safety mechanism degrade precisely as the thing it guards gets busier.
This is the sharpest evidence yet for something I keep going on about here. We have told ourselves that the human in the loop is the answer to agents doing alarming things. This measures what that human actually does, at speed, under volume, and the answer is: clicks yes. Not because anyone is careless — the conditions guarantee it. Time pressure, thin context, and no way to check what a script contains without stopping to read it, which is the one thing the workflow is designed to stop you doing.
A game is not a workplace, and forty thousand self-selected players are not a sample of professional practice. But it is pointing at the right thing, and the direction of the error is the one that costs you.