GitHub Trending #2: Self-Verification Framework Lets DeepSeek V4 Flash Surpass Fable 5 at 1/11 the Cost

LLM-as-a-Verifier is an open-source framework led by Jacky Kwok, developed with scholars from Stanford University, UC Berkeley, and NVIDIA Research, reaching #2 on the GitHub trending list. Its core idea: without training additional reward models or verifiers, directly use existing LLMs to perform fine-grained verification and ranking of an agent's complete execution trajectories.

On the Terminal-Bench 2.1 benchmark, DeepSeek V4 Flash's native success rate is only 78.7%, rising to 86.5% with Best-of-3 and 88.0% with Best-of-5, surpassing the closed-source frontier model Claude Fable 5. Oracle (ideal selection upper bound) reaches 96.6%, and the original paper reports up to 98.9% when aggregating different configurations. This means correct trajectories are often already in the candidate pool; the bottleneck is selection.

In terms of cost, the team estimates DeepSeek's self-verification total cost per task is about $0.11, while Fable 5 is about $1.3, less than one-tenth the cost. Some netizens questioned whether the 11x gap might come solely from per-token price differences, but the paper's first author responded that the cost already includes all overhead for generating candidate trajectories and verification, and despite higher token consumption, the extremely low unit price of open-source models still yields a total cost advantage.

Traditional LLM-as-a-Judge requires the model to output integer scores from 1 to 5, with granularity too coarse, causing trajectories with subtle quality differences to often be judged as 4 or 5 simultaneously. The original paper measured a tie rate of up to 26.7% for single discrete scoring, meaning over a quarter of candidates end up tied, preventing the system from optimal selection.

LLM-as-a-Verifier's breakthrough lies in intercepting the underlying log-probability distribution (logprob) when the model outputs its evaluation, computing a continuous verification score. Even if two trajectories are both judged as '4', as long as the model's probability distribution over adjacent score levels differs, they can still be differentiated.

The team further extends verification computation along three directions: increasing scoring granularity, repeatedly evaluating the same trajectory, and splitting complex evaluations into multiple criteria judged separately. Experiments show that as the verification budget increases, verification accuracy continues to improve, providing important evidence for defining Verification as an independent scaling axis.

As the number of candidates increases, the computational complexity of exhaustive pairwise comparison explodes. The team proposes Probabilistic Pivot Tournament (PPT) ranking to complete sorting with fewer comparisons; on the engineering side, prefix caching reduces the cost of repeated input for long agent trajectories.

In the past, Test-time Scaling focused on more sampling and more search; LLM-as-a-Verifier adds the verification side: after candidate generation, verification itself is worth investing computation in. For developers, model selection should not only look at per-token price, but at the total cost per successful task. Running a cheap model multiple times and then filtering may be more cost-effective than directly calling an expensive flagship model.

This is especially important for open-source models: previously, high-quality verification relied on training additional Reward Models, PRMs, or calling stronger models as judges, but this framework proves that off-the-shelf LLMs can handle fine-grained verification themselves, even serving as both generator and verifier. Verifiers have the opportunity to enter the Agent Harness and become a runtime foundational capability.