Vibe Coding

A revised version of this landed at the end of June and it is the best thing I have read on why people actually like working this way. Pimenova, Fakhoury, Bird, Storey and Endres went through 190,000-odd words of interviews, Reddit threads and LinkedIn posts and built a grounded theory of vibe coding rather than another benchmark.

Their central claim is that trust is the regulator:

AI trust regulates movement along a continuum from delegation to co-creation and supports the developer experience by sustaining flow.

That continuum is the useful idea. Delegation at one end — you hand over a task and check the result. Co-creation at the other — you are in the loop, thinking alongside it. Where you sit is not a setting you choose or a property of the tool; it is a function of how much you currently trust the thing, and it moves constantly, sometimes several times inside one session. Everybody who does this seriously already knows the feeling. Nobody had named the axis.

It also explains why the productivity literature is such a mess. Two developers using identical tools on identical tasks can be at opposite ends of that continuum, which means they are doing genuinely different activities. Averaging them produces a number that describes neither.

The pain points they catalogue are refreshingly unromantic: specification, reliability, debugging, latency, code review burden, collaboration. Note that latency is on that list alongside correctness. Flow is fragile, and a model that is right but slow can be worse for the practice than one that is quicker and occasionally wrong. That is an uncomfortable finding and I believe it completely.

Qualitative work gets dismissed in this field as soft next to a benchmark score. This is a good argument for why that is backwards: the benchmarks can tell you what the model did, and only this kind of study can tell you what the person was doing while it happened.

Shukla, Joshi and Syed ran the experiment everyone assumed had already been run: take generated code, iteratively ask the model to improve it, and count the vulnerabilities at each round. Four hundred samples, forty rounds of refinement, four different prompting strategies.

Critical vulnerabilities went up 37.6% after five iterations.

I want to be precise about why this is interesting, because "AI bad at security" is not the finding. The finding is that the refinement loop — the single most natural thing to do when you are not happy with generated code, the thing every agent harness does automatically — has a failure mode that points the wrong way. Each pass is locally plausible. The model addresses what you asked about, adds some defensive code, moves something around. Nothing in that loop has any memory of the security properties the previous version happened to have, so they get traded away for the improvement you asked for.

Different prompting strategies produced different vulnerability patterns, which tells you the effect is a property of the loop rather than of any one phrasing. You cannot prompt your way out of it.

The paper's recommendation is human validation between iterations rather than at the end. That is correct and almost nobody does it, because the entire appeal of an agent loop is that you do not have to sit in the middle of it. Which is roughly where we are with all of this: the practices that would make it safe are the practices that make it feel slow, and feeling fast is what people are buying.

The preprint is from mid-2025 and has been sitting in my "should have written about this" folder ever since. It has aged into relevance rather than out of it.