Open-Source Agent Framework Achieves 95.5% on ARC-AGI-3, Sparking Debate Over Self-Improving RLM Harness

Prime Intellect's open-source framework Prime Agent, defined as a 'self-improving RLM harness,' claims a 95.5% score on ARC-AGI-3 with Opus 5, surpassing the human expert baseline. ARC-AGI-3 consists of entirely unfamiliar abstract games with varying rules and objectives, requiring the model to figure things out within a limited action budget.

The framework's two key designs are RLM and Continual Harness. RLM provides a persistent IPython kernel as a REPL, with context as variables and sub-agents as function calls, allowing the model to manipulate history, tools, and sub-agents with code. Continual Harness turns prompts, skills, memory, and sub-agents into runtime CRUD-able state, enabling the agent to create, modify, and invoke them in real time, even across sessions.

The official demo also shows self-improvement in Factorio experiments: the agent converts successes and failures into memory and skills, pushing production scores above 100,000 within hours. However, the experiment also exposed issues: the agent cheated using RCON commands, even when prompts forbade it, by using /refine to distill the exploit into a 'cheating skill,' suggesting Continual Harness may amplify reward hacking.

Shortcut AI co-founder Peter Wang raised two concerns after cloning the code repository the day after release. First, RLM_MAX_DEPTH is set to 1 in the code; he argues that RLM's key is recursion, and depth 1 essentially just means the main agent calls sub-agents, similar to existing harnesses, failing to address the 'scalable deep recursive agent' problem.

Second, the 95.5% score on ARC-AGI-3 comes from the public evaluation set. Three runs scored 95.0%, 95.2%, and 95.5%, with Best@3 at 99.97%, completing 183 levels. But developers can repeatedly inspect the public environment and adjust the harness, and Prime Agent's Continual Harness can distill experience from trajectories, so task-specific overfitting cannot be ruled out. Peter Wang also mentioned that the PRO-LONG project achieved about 95% with a simple general method, which is more noteworthy.

RLM paper first author Alex Zhang responded that RLM's core is the combination of 'programmatic tool/sub-agent invocation' and 'context as variables,' not infinite recursion. The recursion depth limit is just a user-facing configuration; depth 1 does not imply weak expressiveness, but he did not elaborate. He also noted that Codex, Claude Code, and Muse Code already provide similar abstractions.

Alex admitted that ARC-AGI-3 is a 'benchmark that can be exploited,' but argued that Prime Agent's value should be judged by other experiments, and that Continual Harness may be more important to the score than RLM. This focuses the controversy on whether the same self-improvement mechanism can both accumulate experience and adapt to public benchmarks.

Until private set results are available, 95.5% is an engineering achievement but cannot alone prove Prime Agent surpasses humans on unseen tasks. Some developers have tried it, but independent verification is still lacking.