DeepSeek + Pi Combo Wins Benchmark, Costs One-Seventh of Claude Code

Composio's public test used the DeepSeek V4 Flash model across 8 harnesses on 30 high-difficulty tasks. Pi Agent passed 20 (66.7%), Oh My Pi passed 17, Claude Code, Codex, and Deep Agents each passed 16, Prime Agent and Hermes Agent each passed 15, and OpenCode passed 14. With the same model, simply changing the harness improved the success rate from 46.7% to 66.7%, a 20-percentage-point difference.

The cost gap is even more pronounced: Pi's average cost per successful task was $0.028, while Claude Code's was $0.195, nearly 7 times higher. Median time for Pi was 132.2 seconds, slightly slower than Claude Code's 122.7 seconds and OpenCode's 129.7 seconds, but considering success rate, speed, and cost, Pi stands out.

Pi used a fresh default installation with no custom settings or tuning, only the MCP plugins required for the test. In contrast, Prime Agent had massive sessions, some consuming 3.5 million tokens and making 33 tool calls, causing scorer timeouts and 6 runs unscored. This challenges the notion that more configuration is better; lightweight harnesses provide a shorter path, reducing the chance of the model getting lost.

Developer 0xEvan used Pi with DeepSeek V4 Flash to process nearly 1 billion tokens, achieving a 99.93% cache hit rate, costing $2.65 instead of an estimated $132 without caching. Another developer, Shantanu Goel, reported that cache hit rates in other harnesses typically range from 94% to 97%, while Pi consistently exceeds 99%.

DeepSeek's API uses prefix caching, requiring matches to start from the first token; any prefix change causes subsequent tokens to miss. Pi's programmability allows developers to modify system prompts, filter history, customize compression, and even rewrite the final payload, leaving room for cache optimization.

The open-source project Reasonix is designed around DeepSeek's prefix caching, with core principles of keeping the context front stable and appending rather than modifying. Developers ported its methods to Pi's DeepPi, claiming cache hit rates stable at 99.7%-99.9%. The pi-deepseek-cache extension achieves similar optimizations by freezing dates and working directories, deterministic summaries, and SHA-256 hash diagnostics, reducing input costs by 98%-99%.

Composio emphasizes that models should not be evaluated in isolation; leaderboards that only list model names without specifying the harness are incomplete. The harness multiplier effect means tools can amplify or diminish model performance; choosing the right harness can improve reliability and efficiency, while the wrong choice can have the opposite effect.

DeepSeek has not yet released an official harness, but the 'DeepSeek Harness Team' official account has been registered, internal testing has started, and a public release is imminent. An official harness could offer native adaptation, coordinated optimization with the model training team, and access to non-public information, which third parties cannot achieve.

Third-party harnesses can only reverse-engineer optimizations through public APIs, which has limitations. The release of an official harness could change the current ecosystem, but its specific capabilities remain to be verified.