Asking the model to "make it more secure" five times makes it less secure↗ (external link)

Shukla, Joshi and Syed ran the experiment everyone assumed had already been run: take generated code, iteratively ask the model to improve it, and count the vulnerabilities at each round. Four hundred samples, forty rounds of refinement, four different prompting strategies.

Critical vulnerabilities went up 37.6% after five iterations.

I want to be precise about why this is interesting, because "AI bad at security" is not the finding. The finding is that the refinement loop — the single most natural thing to do when you are not happy with generated code, the thing every agent harness does automatically — has a failure mode that points the wrong way. Each pass is locally plausible. The model addresses what you asked about, adds some defensive code, moves something around. Nothing in that loop has any memory of the security properties the previous version happened to have, so they get traded away for the improvement you asked for.

Different prompting strategies produced different vulnerability patterns, which tells you the effect is a property of the loop rather than of any one phrasing. You cannot prompt your way out of it.

The paper's recommendation is human validation between iterations rather than at the end. That is correct and almost nobody does it, because the entire appeal of an agent loop is that you do not have to sit in the middle of it. Which is roughly where we are with all of this: the practices that would make it safe are the practices that make it feel slow, and feeling fast is what people are buying.

The preprint is from mid-2025 and has been sitting in my "should have written about this" folder ever since. It has aged into relevance rather than out of it.