Back to feed
News Story
meng shao (X)
1 sources

SGLang × Qwen × NVIDIA Collaboration: NVFP4 KV Cache Breaks Long-Context and Agentic Inference Bottlenecks

SGLang, Qwen, and NVIDIA have collaborated to integrate NVIDIA's NVFP4 format into SGLang's paged KV Cache, maintaining accuracy and performance at 4-bit precision. The implementation uses three attention paths for prefill, extend, and decode, with the decode phase reading NVFP4 directly and dequantizing on the fly within the kernel. Tests show peak decode throughput improvements of about 26-30%, but a 1.60 percentage point regression on SWE-bench Verified for a smaller model.

SynthePulse Insight · AI deep readingMembers

The Ledger of 4-bit KV Cache: What NVFP4 Saves in SGLang, and What It Costs

Version 1 · 1 source

SGLang, Qwen, and NVIDIA brought Blackwell's NVFP4 format into SGLang's paged KV Cache. The official blog's ledger is honest: about 1.78× capacity and up to 78% faster long-context decode, but small models regress 1.60 percentage points on SWE-bench Verified, and global scaling is still uncalibrated.

  • NVFP4 uses two-level scaling: 4-bit E2M1 values + one FP8 block scale per 16 values + one FP32 global scale; 16 values occupy 9 bytes total, about 56.25% of FP8, for an ideal capacity ratio of about 1.78×.
  • The implementation splits into three attention paths by phase: initial prefill uses BF16 Q × FP8 K/V and writes quantized KV into the cache; chunked prefill/extend dequantizes into a cross-layer shared FP8 workspace; the decode kernel reads NVFP4 directly and dequantizes on the fly internally.
  • At equal concurrency, peak decode throughput improves about 26–30%; at equal capacity, resident concurrency goes from 44→70, 10→15, 1→2, with throughput gains of 37.37%, 57.75%, and 78.46% respectively—the longer the context, the larger the gain.
Open section navigationThe Core of the Ledger: 4-bit Does Not Equal 2×

The Core of the Ledger: 4-bit Does Not Equal 2×

The pain point of naive 4-bit quantization is too narrow a dynamic range. NVFP4's solution is two-level scaling: the values themselves use 4-bit E2M1 (1 sign + 2 exponent + 1 mantissa), every 16 values get one FP8 (E4M3) block-level scale, and the whole tensor gets one FP32 global scale.

The storage ledger is therefore not the theoretical 2×. 16 values packed occupy 8 bytes, plus 1 byte of block scale for 9 bytes total, while FP8 needs 16 bytes for the same 16 values—about 56.25% of FP8's footprint, for an ideal capacity ratio of about 1.78×. That 1 byte of block scale is the tax paid for accuracy.

This is also the baseline for understanding all the performance numbers that follow: the ceiling on capacity gains is set by the format itself, not by kernel optimization.

Three Paths: Putting the Quantization Cost on the Compute-Bound Side

The implementation plugs into SGLang's paged KV Cache and splits into three paths by phase. In the initial prefill phase, attention computes directly with BF16 Q × FP8 K/V while quantizing KV from FP8 into NVFP4 and writing it to the persistent cache; this phase does not read the cache, so there is no dequantization overhead.

In the chunked prefill/extend phase, the cached prefix is gathered out and dequantized into a cross-layer shared FP8 workspace, concatenated with the current chunk's KV for FP8 attention, and after computation the current chunk is quantized into the NVFP4 cache; the shared workspace exists to control extra VRAM overhead.

Decode is the most critical part. Decode is memory-bound, and dequantizing before computing would add a full extra VRAM round trip, so the kernel reads NVFP4 directly and dequantizes on the fly internally: packed K/V tiles and E4M3 block scales go into shared memory, the cvt.rn.bf16x2.e2m1x2 instruction converts pairs in registers, the global scale is passed in through the BMM scaling parameter, and computation is BF16 throughout.

The trade-off of this design can be summarized as: prefill is compute-bound, so it can tolerate one extra quantized write; decode is memory-bound, so it must read exactly as much as it computes.

Free for now

Read the full analysis

4 more sections of analysis, plus the full takeaway

Loading

Credibility boundary

All facts, numbers, and conclusions in this article are taken from the SGLang/Qwen/NVIDIA official blog and its retelling materials, with no external information introduced and no third-party independent reproduction seen. The accuracy and performance numbers are official self-reported figures; among them, the near-zero difference for large models, the degradation magnitude for small models, the equal-concurrency and equal-capacity throughput gains, the TTFT changes, and the AgentX watershed are all statements from the source side, treated here as source_claim and not elevated to confirmed fact.

Sources for this version

  1. SGLang × Qwen × NVIDIA 三方合作：用 NVFP4 KV Cache 突破长上下文与 Agentic 推理瓶颈 长上下文和 Agentic …meng shao (X)

Primary report

meng shao (X)

Primary source