NVIDIA shares insights from its Exemplar Cloud initiative, revealing that identical AI hardware can yield 8-12% performance gaps due to configuration choices. The company provides lessons to help partners optimize training throughput on H100, GB200 NVL72, and GB300 NVL72 systems.
SynthePulse Insight · AI deep reading
Version 1 · 1 source
NVIDIA reveals through four real-world cases that training throughput differences of 8%–12% can occur in AI clusters built with identical hardware, with the root cause being compound deviations in kernel, hypervisor, BIOS, and NCCL configurations rather than a single hardware fault.
In Exemplar Cloud validation, NVIDIA found that training throughput differences in AI clusters built with identical hardware are typically 8%–12%, rather than being caused by a single fault. These gaps stem from compound deviations in kernel, hypervisor, BIOS, and NCCL configurations, each contributing a few percent, which accumulate to push performance below the 95% threshold of the reference architecture.
Common patterns include: missing SMMU and virtualization configurations on Grace CPUs, incorrect CPU power management and NUMA binding, failure to pass runtime topology files into containers, mismatched NCCL settings with the network, and training process binding that does not account for topology affinity.
A GB200 NVL72 cluster running DeepSeek-V3 MoE FP8 pre-training inside a virtual machine showed iteration times 12%–14% longer than the bare-metal reference architecture. Nsight Systems revealed abnormally high CPU overhead, and perf record showed 24% of CPU cycles spent in the arm_smmu_cmdq_issue_cmdlist function.
This function submits invalidation instructions to the SMMU command queue; in virtualized environments, each map/unmap operation triggers a host trap and serialization. After enabling CMDQV/VCMDQ (Command Queue Virtualization extensions), arm_smmu_cmdq_issue_cmdlist no longer appeared in hotspots, and the MoE iteration time gap narrowed to within the reference architecture tolerance.
An H100 cluster suffered a 12% performance loss due to incorrect CPU C-state and NUMA configurations. Core frequencies ran lower than expected, and NUMA/process binding did not match the platform topology, increasing memory access latency.
By optimizing CPU power management policies and ensuring process binding aligned with NUMA topology, performance was restored to reference architecture levels.
On high-bandwidth fabrics such as ConnectX-8 SuperNICs, insufficient NCCL queue-pair concurrency settings can limit communication performance. Default configurations may fail to fully utilize 1.6 Tbps bandwidth, increasing latency for AllGather and ReduceScatter operations.
After adjusting NCCL queue-pair concurrency parameters to match network scale and message size, communication performance improved significantly.
NCCL topology files and environment variables were correctly set on the host but not propagated into containers, causing training processes to be unaware of network topology and resulting in severe slowdowns in AllGather/ReduceScatter operations.
After ensuring all topology and environment variables were accessible within the containerized training environment, performance returned to normal.
This article is based on NVIDIA's official technical blog, summarizing experiences from NVIDIA engineers during Exemplar Cloud validation. Cases are from real partner clusters. All data and conclusions originate from that blog, with no external information introduced.
Optimizing AI cluster performance requires a systematic check of kernel, virtualization, power management, NCCL configuration, and container environments; adjustments at a single level are insufficient to address compound deviations.
Primary report
Primary source