Xiaohongshu's dots infra team has open-sourced BigMac, a new paradigm for multimodal large model training. It uses dependency-safe nested pipelines to efficiently train multimodal models without increasing LLM pipeline bubbles or activation memory, achieving 1.08–1.9x speedup. BigMac is already a core component in dots multimodal model training.
SynthePulse Insight · AI deep reading
Version 1 · 1 source
The dots infra team at Xiaohongshu open-sources BigMac, which efficiently implements multimodal pipeline training by leveraging dependency-safe nested pipelines without increasing LLM pipeline bubbles or activation memory, achieving 1.08x–1.9x training speedup over baselines.
Multimodal large language models (MLLMs) typically consist of a modality encoder, an LLM backbone, and a modality generator, all running on the same GPU cluster but with different computation rhythms. When integrating these three modules into a single training pipeline, the system faces a Pareto frontier: throughput and memory often cannot both be prioritized.
Compute-efficient designs separate encoder and generator computations from the LLM pipeline to avoid cross-module bubbles, but they require long-term activation retention, with memory growing with the number of microbatches. Memory-efficient designs integrate all modules into the same pipeline, shortening activation lifetimes, but dependency coupling between modules introduces bubbles. For large-scale multimodal training, especially with generators, this trade-off becomes a bottleneck.
BigMac's starting point is to retain the LLM pipeline as the scheduling backbone, since large-scale LLMs already rely on mature schedules like 1F1B or interleaved 1F1B. BigMac inserts encoder and generator computations at dependency-safe positions—where inputs are ready and the LLM execution order is not disrupted. This design is called dependency-safe nested pipelines.
This design brings two important properties: first, it preserves computational efficiency, as timing fluctuations in encoders and generators do not propagate along the LLM pipeline; second, it limits modality activation memory—BigMac runs encoder/generator backward as soon as gradients are ready, reducing encoder and generator activation memory to O(1) while keeping LLM activation behavior unchanged.
BigMac's runtime includes a Scheduler and an Executor: the Scheduler generates a global operator table covering all pipeline ranks, microbatches, and module types; the Executor interprets the local operator sequence and dispatches it to corresponding backends. This separation makes scheduling strategies visible and inspectable, while execution is backend-friendly.
BigMac provides pipeline-parallel interfaces that are transparent to algorithm engineers: engineers only need to specify module production and consumption relationships, and BigMac handles the underlying schedule, handoff, and communication. Additionally, BigMac is equipped with schedule-aware profilers, simulators, and visualization toolchains to help locate bubble sources and support rapid parallel strategy iteration.
The team evaluated BigMac on two workloads: MLLM-Understanding and MLLM-Generation. MLLM-Understanding uses Qwen3-30B-A3B as the LLM backbone and a 1.3B ViT encoder. Compared to the compute-efficient baseline Optimus, BigMac achieves 1.08x–1.1x speedup; compared to the memory-efficient baseline Megatron-DistTrain, it achieves 1.6x–1.9x speedup.
In terms of memory, as per-GPU batch size increases, BigMac's peak memory remains stable, similar to the memory-efficient baseline; while Optimus's memory grows rapidly and OOMs at larger batch sizes. On the MLLM-Generation workload (adding a 20B MMDiT generator), Optimus OOMs at all tested batch sizes, while BigMac avoids this issue by running generator backward in time, achieving 1.5x–1.9x speedup.
This article is based on a technical post by the dots infra team at Xiaohongshu published on Machine Heart, including details from the paper and open-source project. Experimental data are self-reported by the team and have not been independently verified by third parties.
BigMac breaks the Pareto frontier between computational efficiency and memory efficiency in multimodal training through nested pipeline design, offering a viable solution that balances speed and memory for large-scale multimodal training.
Primary report
Primary source