Jeff Dean-backed RSI benchmark DeepRSI released: Kimi K3 ranks third, top Chinese open-source model

What DeepRSI wants to measure can be put in one sentence: whether a model can keep finding better solutions through long stretches of repeated trial and error. It was released by AI startup hillclimb and specifically tests large models' recursive self-improvement ability, with 15 tasks all drawn from real machine learning research, covering four major directions: deep learning training scheme optimization, GPU kernel acceleration, estimator accuracy improvement, and algorithmic improvements such as vector search.

The exam rules are quite strict: all models must complete the tasks in an offline environment, using the unified Pi Agent framework, at maximum reasoning effort. During this period, the model needs to propose its own approaches, write code, run experiments, and then continuously adjust based on results, striving to improve the research metrics. Compared with previous short-term tasks that mostly lasted around 12 hours, the 48-hour duration is itself a variable—it replaces 'can it get it right once' with 'can it keep getting better over a long cycle.'

On the overall leaderboard, Claude Fable 5.1 took first place with 1548 points, GPT-6 Astra followed with 1232 points, and Kimi K3 ranked third with 820 points. Kimi K3 also surpassed Grok 4.7 and Gemini 3.8 Flash, making it the highest-ranked open-weight model in this test and also first among domestic models.

The economics are another dimension worth watching on this leaderboard. The listed costs on the public leaderboard are: Claude Fable 5.1 about $627, GPT-6 Astra about $404, and Kimi K3 about $159, with Kimi K3's cost about a quarter of Claude's. But it must be made clear that the cost here reflects test-run expenses; each model's actual exploration path differs, and it cannot be directly equated with the input-output ratio in real research scenarios.

Token consumption shows another kind of contrast. Claude Fable 5.1 consumed about 47 million tokens per attempt on average and took first place; GPT-6 Astra achieved second place with only about 7 million tokens; Kimi K3 ranked third with about 36 million tokens. The most extreme was Gemini 3.8 Flash, which consumed over 100 million tokens but ultimately ranked fifth. The three curves of score, cost, and tokens do not overlap, showing that 'stronger' in this evaluation system is not leadership on a single dimension.

Official testing shows that GPT-6 Astra once led early in the competition, but as the research tasks continued, Claude Fable 5.1 gradually completed a comeback. This detail is more noteworthy than the final ranking: it means that in this kind of long-cycle task, an early advantage may not hold to the end, and the ability to keep adjusting may be more critical than opening performance.

Grok and Gemini each won and lost on different tasks, with Gemini performing better in GPU acceleration and kernel optimization. This shows that model capabilities are not evenly distributed across different research directions, and a single total score masks differences at the task level.

DeepRSI uses a dual scoring mechanism of public validators plus private validators: the former allows AI to repeatedly test and self-optimize; the latter is responsible for final acceptance, with the aim of preventing models from overfitting to public tests or exploiting loopholes in the scoring rules. Finally, through a Bradley-Terry model, each model's performance is converted into an Elo rating and ranked.

But a case released by the team shows that preventing overfitting and preventing 'goal deviation' are two different things. In a color-swap puzzle training task, GPT-6 Astra initially tried to train a Transformer to imitate a traditional solver; after finding the results unsatisfactory, it redesigned the technical approach, directly encoded the solving algorithm into the model weights, and used the residual stream as working memory, constructing attention heads and logic gates, ultimately writing a hand-designed algorithm into the neural network structure. This case was included by the research team in a discussion of reward mechanism loophole exploitation.

The key here is not that Astra 'cheated,' but that what it optimized may deviate from the goal the researchers originally hoped to achieve. The AI did demonstrate complex engineering problem-solving ability, but whether 'the smarter the model, the better it is at exploiting evaluation rule loopholes' currently has only this one case in support, with no systematic conclusion yet.

hillclimb, behind DeepRSI, was founded in 2025, was selected for Y Combinator's fall batch that year, and currently has only 4 people on the team. Its main business is cooperating with frontier AI labs to help AI Agents gradually acquire independent research capabilities through high-quality training data and reinforcement learning environments.

The investor lineup is one reason this small company has attracted attention. Former Google Chief Scientist Jeff Dean himself is one of the investors, and the AI startup he co-founded, Discovery Loop, has RSI itself as its research direction.

The two co-founders, Jun Park and Ibrakhim Ustelbay, both come from the professional esports world and co-founded the company in August 2025. Among them, Jun Park once left Google DeepMind after staying only about 3 months. From leading an esports team to leading AI in research, this path itself also explains why this company would make 'long-cycle, high-intensity, repeated trial and error' the core of its evaluation design.

The boundaries must be made clear. DeepRSI itself is still at an early stage: 15 research tasks, 5 model configurations, and only two runs per model, still far from comprehensively measuring AI autonomous research capability. Task goals, experimental environments, and scoring criteria are still set by humans, and there is still a long way to go before AI truly achieves recursive self-improvement.

Therefore, whether DeepRSI's results can be extrapolated to real machine learning research scenarios has not been verified by the source; the relationship between the leaderboard's listed costs and the real input-output ratio of research is also still unclear and needs continued observation. These are not questions that can be answered by a single leaderboard.

What can be confirmed is: under this set of rules set by hillclimb—48 hours, offline, unified framework—Claude Fable 5.1, GPT-6 Astra, and Kimi K3 ranked in the top three, and Kimi K3 was the highest-ranked among open-weight models. As for whether this equals 'AI can do research on its own,' the current evidence is still insufficient to support that conclusion.