Tencent Hunyuan Study: When Do Larger Batches Help Scale LLM Reinforcement Learning?

When pretraining brings a large model to a new capability starting point, reinforcement learning determines how much further the model can go through continuous exploration and feedback. The larger the share of RL in model development, the more directly training efficiency affects the compute budget. According to the illustrative account given in the paper: 10,000 GPUs running continuously for 30 days at $3 per card per hour costs $21.6 million in total; if the same number of cards achieves the same result but training time is shortened by 10%, 720,000 GPU hours can be saved, saving about $2.16 million.

One fundamental source of efficiency gains is Batch Size. The Tencent Hunyuan team observed in experiments that moderately increasing the Batch while adjusting the learning rate in tandem can improve throughput while maintaining learning efficiency; but when the Batch is too large, the extra sample cost in turn outweighs the throughput gain. The paper depicts this contrast as mountain climbing: B, 2B, and 4B maintain approximately the same learning progress along the cumulative-sample dimension, and the larger Batch reaches the summit faster because of higher throughput; once expanded to 16B, the sample-efficiency loss exceeds the throughput gain, and reaching the summit is actually slower than B.

The paper's core measurement target is time-to-target: how much real time elapses from the start of training until the model first reaches a preset validation target. It is jointly determined by two kinds of efficiency—sample efficiency (how much learning progress each training response brings) and system efficiency (how many training responses the entire pipeline generates and processes per second). The relationship given in the paper is: relative to a reference configuration, the comparison of the samples-to-target ratio and the end-to-end throughput ratio determines whether a larger Batch is faster; only when the throughput improvement exceeds the increase in samples-to-target will a larger Batch shorten training time.

The classic Batch Scaling discussion is about how a larger Batch changes gradient estimation, the number of updates, and training parallelism. In 2017, Priya Goyal, Yangqing Jia, Kaiming He, and others at Facebook, in "Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour," used linear learning-rate scaling and gradual warmup to extend ResNet-50 training on ImageNet from about 29 hours on 8 GPUs with Batch 256 to 1 hour on 256 GPUs with Batch 8192, while maintaining small-Batch accuracy. In 2018, Sam McCandlish, Jared Kaplan, and Dario Amodei at OpenAI, in "An Empirical Model of Large-Batch Training," studied when the returns from enlarging the Batch begin to diminish, and linked the turning point to the gradient noise scale.

LLM reinforcement learning adds a key closed loop: the Transformer acting as an Agent must first autoregressively generate using the current policy, or continuously interact with the environment, to obtain responses, trajectories, and rewards; these online-generated data then enter the learning stage, used for forward propagation, backpropagation, and parameter updates; the updated Transformer then continues to generate the next batch of data. The model is both an inference system that produces training data and a learning system that consumes that data.

From a system architecture perspective, generation is token-by-token autoregressive inference, requiring repeated reads of model weights, maintenance of the KV Cache, and scheduling of concurrent sequences; learning performs forward and backward propagation over the entire batch of tokens, and must also save activations, compute gradients, and update optimizer states. Therefore enlarging the Batch is not just stuffing more data into the same kind of computation: it may improve generation efficiency through higher concurrency, and it also increases computation and memory overhead on the training side. The learning curve alone cannot answer how much faster training will be, and generation throughput alone cannot answer how many more samples the model still needs.