NVIDIA announces integration of NeMo Automodel with Hugging Face Diffusers, enabling scalable fine-tuning of video and image models. This collaboration aims to simplify and accelerate generative AI model customization for enterprises.
SynthePulse Insight · AI deep reading
Version 1 · 1 source
Deep integration of NVIDIA NeMo Automodel with Hugging Face Diffusers enables distributed fine-tuning of any Diffusers-format model without conversion, scaling from a single GPU to hundreds.
NVIDIA NeMo Automodel is an open-source PyTorch DTensor-native training library deeply integrated with Hugging Face Diffusers. Users simply point pretrained_model_name_or_path to any Diffusers model ID on the Hub to start training. Checkpoints can be directly loaded into DiffusionPipeline for inference or uploaded back to the Hub, remaining compatible with downstream tools (quantization, compilation, LoRA adapters, custom samplers).
The integration supports flow-matching models, training in latent space, and boosts throughput via pre-encoded VAE outputs and multi-resolution bucketed data loading. Currently supported models include Wan 2.1 T2V (1.3B/14B), Wan 2.2 T2V A14B (27B total, 14B active), FLUX.1-dev (12B), FLUX.2-dev (32B), HunyuanVideo 1.5 (13B), and Qwen-Image (20B), all with ready-to-use fine-tuning recipes.
NeMo Automodel's design principle is 'one program, any scale.' Parallelism strategies (FSDP2, tensor parallelism, expert parallelism, context parallelism, pipeline parallelism) are switchable via configuration declarations without rewriting model code. This enables training large models like FLUX.1-dev (12B) and HunyuanVideo (13B).
The training workflow includes dataset pre-encoding, launching training with existing YAML configurations, and generating from fine-tuned checkpoints. For example, fine-tuning FLUX.1-dev on a 78-image Rider-Waite tarot dataset with full Transformer fine-tuning, users can directly reuse a checked-in YAML config, overriding paths and settings via command line. The pre-encoding step caches image and text embeddings as .pt files, avoiding repeated encoding per step.
When a new diffusion model lands in Diffusers, enabling it in NeMo Automodel only requires adding a data processing handler and a model adapter, rather than writing a full custom training script. The rest of the recipe stack (FSDP2, bucketed data loading, checkpointing, generation) remains unchanged, and the same YAML-driven workflow applies.
Future plans include a Pythonic recipe API to further simplify usage. Current multi-node orchestration supports SLURM, with Kubernetes support coming soon. All code and configurations are open-sourced under Apache 2.0 license. NVIDIA Docker containers (nvcr.io/nvidia/nemo-automodel:26.06) are recommended for deployment, but pip3 install nemo-automodel or installation from source are also available.
This article's information comes from an official blog jointly published by NVIDIA and Hugging Face, a first-party source. All technical details and performance claims are from that blog, with no external knowledge introduced.
The NVIDIA and Hugging Face integration significantly lowers the barrier for diffusion model fine-tuning, enabling researchers and developers to efficiently train the latest models at any scale through standardized interfaces and scalable parallelism strategies.
Primary report
Primary source
Also covered by 1 sources