Forge, and the number that moved↗ (external link)

Forge is a reliability layer for self-hosted tool-calling. Small models are perfectly capable of deciding what to do and then fumbling the mechanics — malformed JSON, a tool that does not exist, the right call in the wrong dialect. Rather than returning that to the caller as a failure, Forge validates each call against the tool list, rescues calls buried in code fences or Mistral bracket syntax or Qwen's XML, and re-runs inference up to three times with corrective feedback.

The idea is good and slightly unfashionable: assume the model will be sloppy at the interface and engineer around it, instead of waiting for a model that is not. Most of the value in a tool-calling loop is not intelligence, it is plumbing that refuses to give up on the first malformed brace.

Now the awkward part. This reached me as "guardrails take an 8B model from 53% to 99%". The README says something different: from single digits to 84% on its 26-scenario suite, and Sonnet 4.6 from 85% to 98%. Those are good numbers. They are not the numbers in the headline that carried it around.

I do not think anyone is being dishonest — the eval suite moved from v0.6.0 to v0.7.0, the README says outright that the Anthropic figures were not re-run because the cost is non-trivial, and that kind of restatement is normal. But it is a clean example of how a benchmark claim travels. The headline number detaches from the repository, gets repeated, and by the time it reaches you it is a fact about the world rather than a measurement of one project's 26 scenarios against its own harness.

Which is the standing problem with every improvement figure in this field, and it is worse than it looks — see today's other link on how much of a benchmark score belongs to the backend rather than the model.

Read the README rather than the headline. It is honest about scope: this hardens a single agentic loop, it is not an orchestrator, and the proxy mode trades depth for ease of integration.