Announcing New Mobile AI Inference Benchmarking with Liquid AI

Artificial Analysis, in collaboration with Liquid AI, benchmarks small models at 4-bit or lower precision on the iPhone 17 Pro and Galaxy S26 Ultra. Inference tests are conducted in a controlled environment using Liquid AI's open-source software, measuring end-to-end generation time, output speed, peak memory, and other metrics, with a free app called Pipette available for user self-testing.

The intelligence ranking is based on the average of five evaluations: BFCL, IFBench, AA-Omniscience, GPQA Diamond, and MATH-500, with a default 16K context limit to reflect phone memory constraints on KV cache. The portable device category is defined as models that, after quantization (including KV cache), fit within 8GB memory at 8K context.

The evaluation will evolve with new models, devices, frameworks, and quantization techniques, and the page will be continuously updated.

Under the 16K limit, Nanbeige4.2-3B and LFM2.5-2.6B tie with an average score of 63, leading Ornith-1.0-9B (62) and Qwen3.5 9B (Reasoning) (61). However, efficiency differences are significant: LFM2.5-2.6B processes a 1,024-token prompt in just 8.0 seconds and 2.3GB memory on the iPhone 17 Pro, while Nanbeige4.2-3B takes 21.4 seconds and 4.0GB, and 9B models require over 25 seconds and 6.9GB.

The 16K limit reshapes rankings: Qwen3.5 9B (Reasoning) consumed 74.5M output tokens in one benchmark run, with 29% of generations hitting the 16K cap, ultimately placing fourth. If expanded to 64K, Ling 3.0 Tiny tops with 66 points, but a 64K window is impractical on phone memory, and generating 64K tokens at 55 tokens/s would take over 20 minutes, draining battery.

Therefore, the primary results use a 16K cap, with additional results for 64K and a one-minute generation time limit.

On the iPhone 17 Pro, the speed-intelligence Pareto frontier includes only six models: LFM2.5-230M (27 points, 0.9s), MiniCPM5-1B (45 points, 2.9s), LFM2.5-8B-A1B (58 points, 5.7s), Ling 3.0 Tiny (59 points, 5.7s), LFM2.5-2.6B (63 points, 8.0s), and Nanbeige4.2-3B (63 points, 21.4s).

LFM2.5-8B-A1B and Ling 3.0 Tiny are mixture-of-experts models, activating about 1B parameters per token, thus achieving 8B-level weights and answering within 6 seconds.

End-to-end generation times (generating 256 tokens after a 1,024-token input) span 30-fold, from 0.9 to 26.7 seconds; peak memory at 4K context spans about 19-fold, from 0.4GB to 6.9GB, with high-memory models potentially squeezing system space on 12GB phones.

Leading models have complementary strengths: Nanbeige4.2-3B is the most balanced (BFCL 76%, MATH-500 96%, GPQA Diamond 67%); Qwen3.5 9B (Non-reasoning) excels in tool calling (BFCL 77%) and scientific reasoning (GPQA Diamond 79%); LFM2.5-2.6B is best in instruction following (IFBench 59%), with MATH-500 above 90%, and significantly better hallucination resistance (AA-Omniscience non-hallucination rate 79%, compared to Nanbeige4.2-3B's 33% and Qwen3.5 9B (Reasoning)'s 24%).

Additionally, Falcon-H1R-7B achieves 97% on MATH-500 but does not rank high overall.

These differences indicate that model selection should weigh specific task requirements.

The evaluation sets a 16K context limit, causing some verbose models not designed for phone memory constraints to have relatively lower scores, such as Qwen3.5 9B (Reasoning) whose ranking is affected by exceeding the limit.

Results will be updated with new models, devices, inference frameworks, and quantization techniques; current results are only an initial version.

Users can test models on their own devices via the free Pipette app, but note that the evaluation environment is controlled, and actual performance may vary by device and usage scenario.