Confident and wrong looks exactly like confident and right (external link)
The finding is one sentence and I have been chewing on it for two days.
Auditing a deployed, developer-accessible on-device foundation model — the abstract does not name it, so neither will I — the authors ask a question that is not "is it accurate" but "can anyone tell when it is wrong?" Then they answer it: a classifier over 15 user-visible features separates the model's confident-correct outputs from its confident-wrong ones at AUROC 0.55, and they confirm the equivalence rather than just failing to find a difference.
Nothing visible on the surface distinguishes them. Not hedging, not length, not any of the tells you think you have. The thing you have been doing — reading the output and judging whether it feels solid — has been measured, and it is approximately a coin flip.
The self-reported confidence is worse than useless. AUROC 0.47, which is below chance, at ECE 70, the worst calibration among comparable small models they tested. Asking it how sure it is returns a number with no information in it, delivered in a format that looks exactly like information.
The failure modes go in opposite directions at the same time, which is the part that makes it a design problem rather than a tuning problem. It confabulates on 69% of false-premise questions — ask about something that does not exist and two times in three it will confidently furnish details. It also refuses 18% of entirely benign prompts. One dial does not fix both. The authors call it task-asymmetric miscalibration, and it means the guardrails are simultaneously too loose and too tight depending on which task you walked in with.
Then the useful half. No cheap single-generation signal flags any of this — everything they tried tops out at 0.68 AUROC. But a black-box consistency wrapper, requiring no access to the model at all, takes confident confabulation from 75% to 3% and selective accuracy from 43% to 83%. Generate several times, compare, and treat disagreement as the signal you could not get from any single answer.
That is n× the inference cost, which the authors call tunable and which on device means battery and latency, and it is why nobody ships it by default. But note what it is not: it is not a better read of the answer. It works because it never looks at the surface at all — the disagreement between samples carries information that no amount of scrutinising one sample can recover.
Which puts it in the same family as the other escape route this site has covered, from the opposite end. The formally verified polygon library replaced human reading with a machine checking code against eighty-seven lines of specification. This replaces human reading with a machine checking the model against itself. Both give up on the eyeball entirely, and this paper is the measurement of why they had to.
Two things I am taking from this beyond the paper's own scope, flagged as extrapolation because they are.
This is a small on-device model, and the numbers do not transfer to whatever is in your terminal. What plausibly does transfer is the shape: confidence in the text is a stylistic property, produced by the same process as the content, and there is no reason a bigger model would suddenly couple it to correctness.
And the audit protocol matters as much as the result. They released the code and frozen evaluation items so anyone can run this against a model they did not build, from outside, with no privileged access. Hundreds of millions of devices are running inference with no server-side moderation and a configuration that, as the authors put it, is rarely audited independently. Somebody has to be able to check, from the outside, cheaply, and this is what that looks like.
The uncomfortable version, for those of us who read agent output all day: your review of a confident answer is not a check on the model. It is a check on the prose. Those turned out to be different things, and one of them is measurable at 0.55.