Security

Alex Wauters built a browser game. You get sixty seconds and a stream of simulated permission requests from a coding agent, and you approve or deny. Forty thousand runs later, across roughly 409,000 commands, the results are in and they are not flattering.

Players approved about a third of the malicious requests. Scope violations — the agent reaching for AWS credentials or Kubernetes config when it had no business there — were missed 35% of the time, the worst category. The single most-approved dangerous command was npm run analyze, waved through nearly 65% of the time, because it looks like something you have typed a thousand times and you cannot see what is in the script without going and looking.

The commands people caught reliably were the theatrical ones. rm -rf / gets denied. Of course it does. It looks like what it is.

The number that should worry you is not from the game at all. Wauters cites Anthropic's own telemetry: users approve around 93% of permission prompts. That is the whole problem in one figure. A control that is exercised 93% of the time in one direction is not a control, it is a formality with a keystroke attached. And the more prompts you see, the less each one means — which makes the safety mechanism degrade precisely as the thing it guards gets busier.

This is the sharpest evidence yet for something I keep going on about here. We have told ourselves that the human in the loop is the answer to agents doing alarming things. This measures what that human actually does, at speed, under volume, and the answer is: clicks yes. Not because anyone is careless — the conditions guarantee it. Time pressure, thin context, and no way to check what a script contains without stopping to read it, which is the one thing the workflow is designed to stop you doing.

A game is not a workplace, and forty thousand self-selected players are not a sample of professional practice. But it is pointing at the right thing, and the direction of the error is the one that costs you.

npm now has staged publishing. A package version goes to a queue rather than straight to the registry, and a maintainer has to approve it with 2FA before anyone can install it. Five commands — stage publish, list, view, approve, reject — and crucially, CI can stage without 2FA while the human supplies it at approval.

GitHub's framing is that it closes an entire class of CI takeover publish attacks, and that looks right. If your pipeline is compromised, the attacker can now push a version into a queue, which is a great deal less interesting than pushing one to every machine that runs npm install.

What makes this worth noting here rather than filing under routine supply-chain news is the shape of it. Everything else in this space is racing to remove human steps. This deliberately adds one, at the exact point where the cost of being wrong is highest and the cost of the delay is lowest. That is the correct trade, and it is nearly the opposite of how agent tooling is being designed — where the pitch is precisely that you do not have to sit in the middle.

The scepticism in the responses is fair, though. Adnan Khan is right that people should turn it on immediately; the doubters are right that most won't. An opt-in control adopted by the maintainers who were already careful does not move the number that matters. And a human approving a staged release under time pressure is the same human from the approval-fatigue study — the mechanism only works while the queue is short enough that looking at it still feels like a decision.

Turn it on anyway. Needs npm CLI 11.15.0 and Node 22.14.0.

Shukla, Joshi and Syed ran the experiment everyone assumed had already been run: take generated code, iteratively ask the model to improve it, and count the vulnerabilities at each round. Four hundred samples, forty rounds of refinement, four different prompting strategies.

Critical vulnerabilities went up 37.6% after five iterations.

I want to be precise about why this is interesting, because "AI bad at security" is not the finding. The finding is that the refinement loop — the single most natural thing to do when you are not happy with generated code, the thing every agent harness does automatically — has a failure mode that points the wrong way. Each pass is locally plausible. The model addresses what you asked about, adds some defensive code, moves something around. Nothing in that loop has any memory of the security properties the previous version happened to have, so they get traded away for the improvement you asked for.

Different prompting strategies produced different vulnerability patterns, which tells you the effect is a property of the loop rather than of any one phrasing. You cannot prompt your way out of it.

The paper's recommendation is human validation between iterations rather than at the end. That is correct and almost nobody does it, because the entire appeal of an agent loop is that you do not have to sit in the middle of it. Which is roughly where we are with all of this: the practices that would make it safe are the practices that make it feel slow, and feeling fast is what people are buying.

The preprint is from mid-2025 and has been sitting in my "should have written about this" folder ever since. It has aged into relevance rather than out of it.

Hanqing Zhao and the Systems Software & Security Lab at Georgia Tech have been running a project called Vibe Security Radar since May 2025: take public vulnerability advisories, trace the fix backwards to the commit that introduced the bug, and check whether an AI coding tool left fingerprints on it.

The trend line is the story. Six CVEs attributed to AI-generated code in January 2026, fifteen in February, thirty-five in March. Seventy-four confirmed cases in total across roughly fifty tools tracked. The researchers reckon the real number is five to ten times what they can detect — four to seven hundred across open source — because most AI-written code does not announce itself.

One finding needs handling with care, and the coverage mostly did not handle it carefully. Claude Code accounts for the largest share of attributed flaws, and the researchers say that is partly because it leaves identifiable signatures. That is a detection artefact, not a quality ranking. A tool that stamps its commits is a tool you can measure; a tool that does not is a tool that vanishes from your dataset. If you take that chart as a league table of which assistant writes the least safe code, you have read it exactly backwards — you are looking at a map of which vendors are legible to researchers.

The genuinely alarming number is buried in the middle: Claude Code alone touched more than four percent of public GitHub commits last month. Whatever the per-commit defect rate turns out to be, it is now multiplied by a denominator that did not exist two years ago.

Methodology this transparent is rare in this corner of the discourse. Most "AI code is insecure" statistics you will see quoted this year come from vendors selling scanners. This one comes from a lab publishing its method, and it is correspondingly less dramatic and more useful.