USTC and HiDream.ai Propose Hi-DiT Hybrid Diffusion Paradigm, Accepted to ECCV 2026

High-resolution image generation has long had two routes: Latent-space diffusion compresses images into a low-dimensional Latent manifold via a VAE, giving low compute cost and stable training, but compression and decoding lose some high-frequency information, capping the recovery of texture, boundaries, and local detail; Pixel-space diffusion bypasses the VAE and models directly on pixels or patchified Pixel tokens, theoretically better at preserving detail, but the raw pixel distribution is high-dimensional and high-entropy, so the model must learn both global semantics and local high-frequency texture in the same denoising pass, which tends to cause capacity competition, slow optimization, and unstable training.

Hi-DiT's starting point is to restate this contradiction as a time-allocation problem: diffusion denoising is not homogeneous at every timestep—high-noise early stages lean more toward recovering low-frequency structure and global semantics, while only low-noise late stages need more synthesis of high-frequency details such as edges and textures. Rather than having the same model carry both objectives throughout, it is better to let Latent and Pixel each do their job at different stages. This judgment is the paper's core argumentative premise; it is the research team's reasoning claim, not an independently verified conclusion.

Hi-DiT is a unified Hybrid Latent-Pixel Diffusion Transformer, not simply a concatenation of Latent tokens and Pixel tokens from start to finish. The model as a whole runs in a parameter-shared Transformer backbone, allowing Latent and Pixel representations to interact, while different input projections and output prediction heads carry different objectives.

The early high-noise stage mainly relies on the Latent Stream, taking VAE Latent as input to learn global structure, object layout, and low-frequency semantics, providing a stable semantic scaffold; entering the later low-noise stage, the Pixel Stream is activated, directly receiving Pixel tokens consistent with the current noise level and feeding them into the shared Transformer together with Latent tokens. At this point the Latent predictor continues to ensure structural consistency, while the Pixel predictor is responsible for restoring sharp high-frequency details.

Two key designs serve the same goal. Time-Gated Injection uses a gate G(t) to control when the Pixel stream participates; the High-Frequency Pixel Predictor addresses the problem that traditional Pixel diffusion commonly uses a single-layer linear projection to regress large patches directly from tokens, which is difficult, by instead using hierarchical sub-Pixel decoding: first gradually upsampling via convolution and PixelShuffle, then predicting local sub-patches, to reduce the difficulty of high-dimensional pixel regression.

The paper claims competitive results on ImageNet class-conditional generation, ImageNet 512 high-resolution generation, and MS-COCO text-to-image generation, among which ImageNet 256×256 reaches 1.06 FID under the CFG setting, and ImageNet 512×512 reaches 1.26 gFID. Qualitative results show that compared with the pure Pixel-space baseline JiT, Hi-DiT generates more stable images with clearer local texture and object shapes, and tighter semantic alignment between text-to-image output and the input prompt. These are all self-reported results from the paper, sourced from a QuantumBit report on a submitted paper, with no third-party reproduction or independent evaluation yet.

The ablation was conducted under an 80-epoch rapid-validation setting: using only a vanilla Pixel stream with no Latent input gives FID 3.38, indicating that optimizing the raw image distribution directly from patchified Pixel tokens is very difficult; adding Latent input lowers FID to 1.74, indicating that the Latent stream provides an effective semantic scaffold; further using the High-Frequency Pixel Predictor brings FID to 1.65, verifying the help of the hierarchical pixel prediction head for detail modeling.

The time-gating threshold τ is equally critical. If the Pixel stream opens too early (e.g., τ=0.7), the model predicts both low-frequency semantics and high-frequency details in the high-noise stage, reintroducing capacity competition; if it opens too late (e.g., τ=0.1), the Pixel stream participates too little and detail recovery is insufficient. With the same 80 epochs of training, τ=0.3 achieves the best result, FID 1.65 and IS 267.7.

The paper emphasizes that Hi-DiT does not trade a large increase in compute for its effect. Compared with SiT and VA-VAE, training and inference overhead increases only slightly: per-epoch training time rises from 17.1/17.7 minutes to 18.1 minutes, single-image inference time from 1.52/1.58 seconds to 1.67 seconds, with peak VRAM 35.52G. On this basis the paper argues that, considering FID improves further from 2.06/1.35 to 1.06, the quality gain from this extra overhead is relatively worthwhile.

Facts and judgments must be distinguished: the above FID, gFID, IS, time, and VRAM numbers all come from the paper's self-report, relayed through a QuantumBit report, belonging to the source_claim level; the ECCV 2026 acceptance status likewise comes from that report. The paper's directional conclusion—that one need not choose between the Latent and Pixel routes, but can let different representations play different roles at different stages of the denoising trajectory—is an inference based on the above experiments, and its generality awaits testing on more tasks and independent reproduction.

The verifiable entry point is the code: the paper gives the GitHub repository HiDream-ai/Hi-DiT. In the absence of third-party reproduction, readers should treat the performance conclusions as strong claims awaiting verification, not as established industry conclusions.