39% of the score belongs to the backend↗ (external link)

Masoudian, Shafaei, Swain and Schedl ran a fully-crossed experiment — three instruction-tuned models, five inference frameworks, six benchmarks, four generation modes — and found that roughly 39% of the observed variability in benchmark performance comes from the backend, not the model.

Not from sampling noise. They checked: even under greedy, sampling-noise-free decoding, changing from HuggingFace to vLLM to Ollama significantly alters measured performance. The divergence is worse on factual benchmarks than on social-bias ones.

Take a moment with what that does to the last two years of discourse. Every league table, every "model A beats model B by four points", every procurement decision justified with a benchmark chart — a large share of that gap is plumbing. Two labs can evaluate the same weights, follow the same protocol, publish different numbers, and both be right. Nobody was cheating. They used different serving stacks.

It also quietly indicts a lot of self-hosted deployment advice. If you evaluated a model on one backend and shipped it on another, you did not deploy the thing you measured. I have never once seen a serving framework version in a model-selection writeup.

The recommendation is unglamorous and correct: disclose the backend, its version, and the full generation configuration alongside any benchmark result, and use deterministic decoding for cross-backend comparison. That is the sort of proposal that gets nodded at and ignored, because adopting it means your historical numbers are no longer comparable to anyone else's — including your own.

The underlying assumption this breaks is that a benchmark score is a property of a model. It is a property of a model and an implementation, and we have been reporting one while measuring the other.