GitHub Trending #2: Self-Verification Framework Lets DeepSeek V4 Flash Surpass Fable 5 at Less Than 1/11 the Cost

An open-source framework called LLM-as-a-Verifier has sparked discussion in the community and climbed to second place on the GitHub trending list. Led by Jacky Kwok, in collaboration with scholars from Stanford University, UC Berkeley, and NVIDIA Research, the core idea is to have the model itself act as a verifier, selecting the best solution from multiple candidate trajectories.

On the Terminal-Bench 2.1 benchmark, DeepSeek V4 Flash achieves a native success rate of 78.7% without verification; with Best-of-3 it rises to 86.5%; with Best-of-5 it reaches 88.0%, surpassing the closed-source frontier model Claude Fable 5. The Oracle (ideal selection upper bound) reaches 96.6%, and the original paper reports up to 98.9% when aggregating different agent configurations.

In terms of cost, DeepSeek self-verification costs about $0.11 per task, while Fable 5 costs about $1.3, making the cost less than one-tenth. Some netizens questioned whether the 11x gap might be solely due to per-token price differences, since self-verification consumes more tokens; first author Jacky responded that the published cost already includes the full overhead of generating N candidate trajectories and subsequent verification, and thanks to the extremely low per-token price of open-source models, the total per-task cost still has an advantage.

Traditional LLM-as-a-Judge requires the model to output integer scores from 1 to 5, which is too coarse, causing two trajectories of similar quality to often receive the same score. The original paper measured a tie rate as high as 26.7% for single discrete scoring, preventing the system from selecting the optimal solution.

The breakthrough of LLM-as-a-Verifier lies in intercepting the underlying log-probability distribution (logprob) when the model outputs its evaluation, using the probability distribution corresponding to different score tokens to compute a continuous verification score. Even if two trajectories are both judged as 4, they can still be differentiated if the probability distributions of adjacent score levels differ.

The team extended verification compute in three directions: increasing score granularity, repeatedly evaluating the same trajectory, and decomposing complex evaluations into multiple criteria for separate judgment. Experiments show that as the verification budget increases, Verification Accuracy continuously improves, which serves as important evidence for defining Verification as an independent scaling axis.

As the number of candidates increases, the computational complexity of full pairwise comparisons explodes. The team proposed Probabilistic Pivot Tournament (PPT) to rank candidates with fewer comparisons; on the engineering side, prefix caching reduces the cost of repeated input for long agent trajectories.

In cost estimation, the DeepSeek side uses DeepSeek V4 Flash Max, priced according to OpenRouter quotes at the time, and the costs of generating multiple candidate trajectories and subsequent verification are all included.

For developers, model selection should no longer focus solely on per-token price, but on the total cost per successful task. Running a cheap model multiple times and then filtering through a Verifier may be more cost-effective than directly calling an expensive flagship model.

In the past, Test-time Scaling mostly discussed multi-sampling and multi-search; LLM-as-a-Verifier fills the other half: after candidate generation, verification itself is worth investing compute in. Verification signals can also be used for progress tracking and reinforcement learning, providing smooth, dense reward signals for complex agents.

This is especially important for open-source models: previously, high-quality verification relied on additional trained Reward Models, PRMs, or calling stronger models as judges; this team demonstrates that off-the-shelf LLMs can handle fine-grained verification themselves, even serving as both generator and verifier simultaneously.

Framework v0.2.0 adds DeepSeek V4 Flash verification support and a Terminal-Bench 2.1 self-verification benchmark. The research has been validated in multiple domains including coding, robot control, and medical diagnosis.