METASTONE Launches Meta-Infer Engine, Boosting DeepSeek Inference Throughput on PCIe GPUs Nearly 7x

METASTONE independently developed and launched the Meta-Infer efficient deployment engine, positioned as an enterprise-grade high-performance large model inference engine for heterogeneous accelerator chips, adopting a pure software optimization route. Its target is a specific class of hardware: long-tail GPUs that lack high-speed inter-card interconnect and are not in the official validation matrices of mainstream open-source frameworks, such as PCIe-Only GPUs and some domestic GPUs.

According to the source, this class of hardware is not "unable to run." Open-source frameworks can complete model loading, weight downloading, and service startup, but operators automatically fall back to inefficient generic implementations, collective communication follows tuning parameters designed for NVLink hardware, and memory and parallelism configurations follow defaults for other hardware. The hardware itself has no defects; the performance loss comes from the adaptation gap between the model framework and the hardware.

Meta-Infer's optimization logic is divided into two major stages: compute-model synergy and communication-compute reconstruction, ultimately crystallizing into four core capabilities: kernel completion, operator optimization and communication reconstruction, parallelism and capacity tuning, and cache reuse. The source emphasizes that all optimizations converge inside the engine, requiring no changes to model weights or model structure throughout, and without changing the semantics of original business tasks—meaning the improvement comes from software adaptation, not changes at the algorithm or model level.

The most complete set of data comes from DeepSeek-V4.1-Flash. In the community Day0 baseline version on an 8 PCIe-Only GPU environment, input throughput was only 1932 tok/s. After repair and adaptation in the compute-model synergy stage—completing the sparse-MLA Prefill fast path, replacing the FP8 dense GEMM slow kernel, and expanding the coverage of the PCIe-IPC communication fast path—throughput rose to 5850 tok/s.

Then came the communication-compute reconstruction stage, with methods including attention operator fusion, trimming speculative decoding draft length, compute-communication overlap, differentiating Prefill/Decode parallelism strategies, and re-tuning memory capacity parameters. Input throughput further rose from 5850 tok/s to 13274 tok/s, achieving an overall 6.87x throughput improvement, and supporting 1M ultra-long context.

It is worth noting that the source characterizes the first-stage improvement as "not creating a brand-new algorithm, but releasing the performance that the hardware and framework already possess but that is locked by adaptation problems." This statement itself is the vendor's explanation of the source of the gains, and is a source claim.

The same approach is replicated on other models: with only the compute-model synergy stage completed, multiple models including DeepSeek-V4-Flash and GLM5.3 obtained throughput gains ranging from 20% to 33%. This shows that a considerable portion of the loss indeed occurs at the adaptation layer, not in the compute itself.