Researchers from Peking University, Donghua University, and South China University of Technology propose UniMotion, which incorporates continuous motion as a standalone modality into a unified multimodal model, connecting motion, text, and RGB. The paper has been accepted at ECCV 2026, with code and project page released.
SynthePulse Insight · AI deep reading
Version 1 · 1 source
While multimodal models are still switching between text and images, a team from Peking University, Donghua University, and South China University of Technology integrates 'continuous motion' as an independent modality into a unified framework, using continuous latents rather than discrete tokens to represent actions, aiming to complete understanding, generation, prediction, and editing within the same model.
Today's multimodal large models can understand and generate text and images, but rarely treat 'how humans move continuously' as an independent modality. Single-frame Pose only describes the body configuration at one frame, while multi-frame Motion records how the body structure evolves over time; per-frame plausible poses do not guarantee natural sequences—foot sliding occurs when the foot continues to translate after landing, joint angle jumps cause jitter, and incorrect ordering of action phases breaks the complete semantics.
Existing Motion-Text methods often quantize trajectories into discrete tokens for easy integration with language models, but this may lose fine-grained variations and temporal continuity; visual large models' human modeling mainly revolves around static poses. UniMotion's starting point is to let continuous motion enter the unified understanding and generation process as a complete modality.
UniMotion is built on Show-o2 1.5B. Text, Motion, and RGB enter the shared LLM backbone; text continues to use discrete tokens and autoregressive generation, while Motion and RGB are encoded as continuous latents and generated by their respective flow heads. Unification happens in the semantic and reasoning backbone, while modality-specific interfaces are retained.
Motion latents go through semantic and generative dual-path embedders: the former summarizes action meaning and phases, the latter preserves local joint and temporal positions; Hybrid Attention and modality-routed LoRA coordinate the different modeling needs of continuous signals and text. Cross-Modal Aligned Motion VAE (CMA-VAE) encodes motion into continuous latents, using DPA during training to align the posterior with image semantics to the Motion Encoder that only reads motion, and removes the visual fusion encoder at inference, so pure motion tasks need no extra images.
After continuous representation is established, LRA uses motion latents as dense conditions, letting the flow head reconstruct target latents from noise, jointly calibrating the embedders, backbone adaptation branches, and generation heads. Motion-to-Motion provides only a pre-training signal, but simply and clearly improves downstream task capabilities.
The same interface supports generating continuous motion from text, summarizing action descriptions from continuous trajectories, editing source motion according to text instructions (e.g., changing 'waving' to 'drinking coffee'), continuing future trajectories based on observed prefixes (the example provides 1.05 seconds of action, predicting the next 4.35 seconds), as well as recovering human pose from images, generating descriptions for images or videos, and letting reference motion participate in human image editing. Motion thus can serve as input, output, and cross-modal generation condition.
Experiments cover datasets like HumanML3D, MotionFix, Human3.6M, and MoVid. Radar charts show UniMotion is the only model among comparisons covering all task directions; the paper reports competitive results on Text-to-Motion, Motion-to-Text, motion prediction, motion editing, and visual human pose recovery.
However, the paper also admits that Text-to-Motion distribution metrics are not all optimal, and specialized human pose recovery models still maintain lower error. Under complex prompts like 'a person stomps up some stairs', UniMotion executes continuous leg lifting, weight transfer, and stepping force more completely than MoMask and MotionGPT.
All information in this article comes from a secondary report by Jiqizhixin on the paper; paper details (such as specific metrics and dataset results) are not first-hand data, and related conclusions should be regarded as source claims rather than confirmed facts.
UniMotion demonstrates a way to integrate continuous, structured non-native modalities into a unified multimodal large model: sharing semantic and reasoning capabilities while retaining modality-specific representations and generation interfaces. This approach can serve general human behavior understanding, digital humans, and content creation, and also provides a design reference for other continuous signals.
Primary report
Primary source