BUPT, PKU, Tsinghua and Others Release PhyAI: First Unified Edge-Cloud Inference Runtime for Physical AI

Deployment of embodied intelligence goes far beyond prototype validation. The paper notes that typical scenarios include benchmark evaluation, cloud RL post-training rollout, edge real-time control, and factory MaaS (where a factory's shared GPUs serve multiple robots). These scenarios have varying requirements for latency, throughput, and multi-GPU communication. Existing work often maintains a separate model codebase for each scenario, leading to high migration costs, redundant development, and inconsistent inference efficiency.

However, although the four scenarios vary in batch size, model precision, and execution device, image preprocessing, model inference logic, caching, and action output can be reused and must remain consistent. This is the pain point PhyAI aims to solve: covering all scenarios with a single codebase.

PhyAI's core idea is to place model semantics in the Model Adapter, leaving scheduling, caching, operators, graph execution, and parallelism to the runtime. Specific components include: Model Runner, which holds vision-language conditions, action experts, video action generation, solvers, etc.; Scheduler, responsible for selecting DP, TP, CFG, and device groups; Runner, managing KV cache, buffers, CUDA Graph, and request states; and Layers, which choose fused or distributed operators based on shape, dtype, and hardware.

This design allows the same model path to run on single-GPU, edge, and cloud multi-GPU environments without rewriting code for different scenarios.

The paper reveals model-side bottlenecks through module-level profiling: for PI0.5 at batch=1, the action expert accounts for only 8.8% of estimated FLOPs but 57.2% of latency; at batch=32, it drops to 13.5% with throughput around 100 samples/s. For Cosmos3, increasing batch from 1 to 16 only improves throughput by 14.3%, approaching compute-bound limits.

However, model-side Roofline alone cannot directly determine control rate, due to RTC, network latency, and image pre/post-processing delays. To address this, the paper proposes Control-Time Roofline, which measures whether the bottleneck stems from model inference or the physical environment. Tests show that on AGX Orin, the main bottleneck is inference, while on RTX Pro 6000, it is the physical environment.

This conclusion implies that on devices with ample compute, further reducing latency will not proportionally increase the ideal control frequency. Algorithm, hardware, and infrastructure need co-design; time saved by framework optimization should support larger models, slower devices, or more concurrency.

In 11 same-model, same-device single-request comparisons, PhyAI achieves 1.40x to 4.65x speedup over official implementations. Specific cases: MiniCPM-Robot on H100 drops from 105.38ms to 22.64ms; Cosmos3-Nano-Policy-DROID on 8 H20s with CFG=2 and TP=4 drops from 2.46s to 1.18s.

Inference acceleration also benefits cloud RL rollout. For RLinf's PI0.5 GRPO with 4 A800s and 32 environments, inference time is 955.8s, accounting for 15.9% of total RL time. PhyAI boosts inference speed by 2.55x; keeping other work unchanged, estimated inference time drops to 863.2s, reducing total RL time by 9.7%.

Additionally, official demos show that with three robots running inference in parallel, PhyAI reduces latency by up to 2.3x, completing tasks without stuttering, while the official framework stutters.

PhyAI integrates inference needs across four scenarios into a single runtime, eliminating redundant implementation of model adaptation, operator optimization, and multi-GPU support, building a unified inference stack for embodied intelligence. However, Control-Time Roofline also bounds the practical gains: robot operation is constrained by both inference and environment, with the bottleneck shifting based on device compute and model size.

Therefore, the paper emphasizes the need to focus on Control-Time Roofline and pursue algorithm-hardware co-design. This reminds us that framework optimization is not a panacea; it must be combined with specific hardware and physical environments.