Tsinghua, Infinigence AI, and SJTU Open-Source APXInf Edge Inference Engine, Boosting Robot On-Device LLM Speed by 10.7x

On September 15, Tsinghua University together with Infinigence AI and Shanghai Jiao Tong University open-sourced APXInf, an on-device inference engine for embodied models. The core numbers given by the project team are: under the Thor chip FP8 configuration, inference latency dropped from 278ms to 26ms, an end-to-end speedup of about 10.7x, and this speedup was achieved through end-to-end full-stack optimization without changing the π0.5 model itself. The post-optimization control frequency of 38.46Hz is described as bringing robot control into the real-time range.

If these numbers hold, they answer the question of "can it run on-device." But they simultaneously expose a more thorny problem: this kind of optimization is highly specialized, and once the model is updated or the chip is upgraded, the path must be redone. What APXInf really wants to answer is actually the second question—when model iteration speed far exceeds the pace of human team expansion, can the on-device optimization capability itself be accumulated and reused.

The evidence boundary must first be made clear: the above latency, speedup, and frequency numbers all come from the project team's and the reporting party's own accounts, belonging to source_claim, with no third-party independent reproduction under the same hardware and configuration yet.

The inference engine is embedded in the middle of the robot control loop: the main controller collects observations, the compute box infers an action chunk, pushes it to the arm or chassis for execution, and then proceeds to the next frame. This position imposes quite specific requirements on the engine—small batch, real-time, low latency jitter, and stable invocation by the main controller via websocket or ROS.

The approach of general inference frameworks is to use a unified intermediate representation to lower the model layer by layer, then hand it to the backend to generate executable code, with one compilation stack covering as many models and hardware as possible; solutions like vLLM and SGLang are designed around cloud throughput. Each has succeeded on its own terms, but their gains are built on the premise of many model types, large batches, and ample scheduling space—and on the embodied edge, none of these three conditions holds.

The edge constraints are hard: compute, bandwidth, power, and heat dissipation are simultaneously limited, yet a single inference must complete multi-view perception, model forward pass, and action generation. The comparison given in the report is that between edge modules and discrete graphics cards, memory bandwidth differs by 4 to 8 times and power consumption differs by 5 to 10 times. This is also why models that run smoothly in the cloud may perform far worse when moved to Thor or Orin.