UniWorld-View, developed by RabbitPre Intelligence in collaboration with Peking University and Peng Cheng Laboratory, has topped the WorldScore leaderboard from Fei-Fei Li's team. The model generates novel views from single images or videos with precise camera control, supports domestic Ascend computing power, and is fully open-sourced.
SynthePulse Insight · AI deep reading
Version 1 · 1 source
Developed by Tuzhan AI in collaboration with Peking University and Pengcheng Laboratory, UniWorld-View has topped the WorldScore leaderboard from Li Feifei's team. By employing occlusion-aware point cloud rendering and dual-stream conditional injection, it resolves foreground-background tearing and backside light leakage in large-baseline novel view synthesis. The code and weights are fully open-sourced.
Novel view synthesis essentially requires AI to 'imagine' unphotographed angles. The difficulty lies in 'large baselines'—showing AI only a frontal view and asking it to generate a profile or even the back of the head. Traditional NeRF and 3DGS methods rely on multi-view reconstruction; when monocular video coverage is insufficient, they produce holes and artifacts. Generative methods can 'paint,' but camera poses are only input as abstract condition vectors, lacking explicit 3D modeling, leading to instability during large rotations.
In the past two years, a third approach has emerged: first use a geometry model to convert the image into a 3D point cloud, then render the target view from the point cloud as a condition for a video diffusion model, as seen in ViewCrafter and NVIDIA's GEN3C. However, the UniWorld-View team discovered two unresolved issues in point cloud rendering: foreground-background tearing and backside light leakage.
Point clouds consist of isolated points lacking connectivity and surface information. When the viewpoint changes significantly, foreground-background tearing manifests as texture stretching at depth boundaries; backside light leakage occurs because point clouds do not automatically occlude, allowing front points to show through to the back. UniWorld-View's first key technique is 'occlusion-aware point cloud rendering,' comprising two methods:
First: dual projection. The source image is projected to the target view and then back-projected. Pixels that cannot 'return' are occlusion regions, accumulated frame by frame into an occlusion mask and removed, preventing erroneous textures from misleading the generative model. Second: normal filtering. Normals are estimated for each point, and the angle with the viewing direction is computed; points facing away from the camera are directly culled. Combining these two methods clears erroneous signals in the condition image, retaining only reliable geometry and clearly defined areas to be filled.
The point cloud rendered image is 'positionally correct but lacks content,' while the source video is 'content-rich but positionally incorrect.' UniWorld-View employs a 'dual-stream conditional injection' architecture: the geometry stream encodes the point cloud rendered image plus mask and adds it to the latent variable, ensuring generated content aligns with 3D structure; the appearance stream uses a newly designed Ref-DiT module, taking the new view features as Query and source video features as Key/Value for cross-attention, allowing the model to 'borrow materials' from the source video to complete textures, even repairing blur caused by point cloud artifacts.
The geometry stream handles composition, while the appearance stream handles coloring—clear division of labor.
UniWorld-View further converts monocular video into multi-view video, achieving 4D reconstruction (dynamic 3DGS). Conventional methods have the camera 'walk and shoot,' coupling spatial transformation with temporal progression, leading to uneven view distribution. UniWorld-View decouples the two: first, freeze time and generate a static surround view at the first frame as appearance anchors; then, generate synchronized multi-view video at fixed camera positions, aligning each position's first frame with the 'reference image,' and completing foreground self-occlusion through iterative generation. After generating multi-view video, it is fed into 4DGS optimization to obtain the final 4D scene.
Tuzhan AI was co-founded by Peking University alumni and young leading talents in the visual field from Peking University, focusing on visual AI large model development. The company has independently developed a series of models including Open-Sora Plan and UniWorld. Open-Sora Plan is the industry's first open-source text-to-video model, ranking first globally in code citations in 2024.
UniWorld-View is compatible with domestic Ascend computing power, with code and weights fully open-sourced. Training data comes from YuanKong Intelligence, a Peking University-affiliated startup.
This article's information primarily comes from a QuantumBit report, where technical details and model performance are based on public descriptions from the Tuzhan AI team and have not been independently verified by third parties. The leaderboard ranking is based on the WorldScore leaderboard from Li Feifei's team, but specific scoring criteria and comparison methods are not detailed in the source.
UniWorld-View effectively addresses key challenges in large-baseline novel view synthesis through occlusion-aware point cloud rendering and dual-stream conditional injection, topping an authoritative leaderboard and being open-sourced, providing a new path for 4D scene generation.
Primary report
Primary source