Daxiao Open-Sources L5 Embodied Dataset ACE-Data-0 with 17M Frames of Real Home Interactions

When embodied intelligence enters the physical world, the fundamental issue is what data to learn human action capabilities from. Ordinary videos can record what happens but struggle to capture contact timing, force variations, object motion, and the sequential dependencies between tasks. Existing first-person videos often lack precise human, hand, and object states; motion capture data often comes from idealized labs, lacking natural occlusion, first-person perspective, audio, and tactile feedback. Much data is limited to a few seconds of single-step actions, failing to describe long-term dependencies in real home scenarios.

ACE-Data-0 is proposed to address this gap: instead of cutting activities into isolated clips, it continuously records perception, action, contact, and state changes, allowing different modalities to jointly describe the same physical process. Its core features can be summarized as complete perception, long-horizon tasks, and multimodal synchronization.

Fine-grained hand manipulation and room-scale activities impose different requirements on collection systems. Fingertip contact and grasp changes require close-range dense sensors; movement across kitchen, living room, and bedroom requires wide coverage and unified spatial coordinates. To this end, the environment-based collection engine ACE designs two complementary configurations: tabletop-scale and room-scale.

The tabletop-scale system targets fine-grained hand-object interactions, deploying multi-view cameras, optical motion capture, and tactile devices, focusing on recording hand poses, object trajectories, and contact changes in tasks like grasping, pouring, wiping, cutting, and folding. The room-scale system covers the entire home space, using wide-baseline cameras and full-space motion capture to continuously record participants' full-body motion, cross-region movement, and continuous interactions.

Both configurations share unified data collection, synchronization, calibration, quality control, and annotation pipelines. Participants wear first-person devices, and the system synchronously records four-channel fisheye video, IMU, head-mounted device poses, full-body motion, and hand joint states; multi-view external cameras supplement occlusion information. Additionally, object motion, full-palm tactile pressure, and multi-channel audio are recorded.

The real difficulty of multimodal data is not the number of devices but the accurate correspondence between different devices' data. Cameras, motion capture, tactile gloves, and audio devices typically have independent clocks and different sampling rates; a few milliseconds of deviation can cause the hand not to touch the cup in the image while the tactile signal has already been generated; without a unified spatial reference, geometric relationships cannot be established.

ACE-Data-0 uses time synchronization and spatial calibration to unify video frames, human motion, object states, tactile readings, and audio onto the same timeline, and registers external cameras, first-person devices, human body, hands, and objects into a shared world coordinate system. Thus, each collection becomes a unified record of the same physical process, allowing researchers to directly find the corresponding human pose, object position, contact pressure, and sound changes for a given frame.

The dataset also provides camera parameters, synchronized timelines, human and hand states, object meshes with 6-DoF poses, bounding boxes, motion trajectories, hand-object contact information, and natural language descriptions aligned with the physical timeline. Much of the annotation comes directly from calibrated collection systems rather than relying entirely on model post-estimation, thereby maintaining consistency in occlusion, fast motion, and long-horizon tasks.

ACE-Data-0 covers three types of tasks: atomic-level human-object interactions (each about 3 minutes, such as pouring water, cleaning, wiping, cutting, folding); long-horizon household activity chains (lasting 20-30 minutes, such as the complete process from opening the fridge to cleaning up); and human-scene interactions (about 5 minutes, including moving, sitting, exercising, opening/closing furniture).

Unlike scripted collection, ACE-Data-0 uses goal-level instructions: participants are only told the final task and can freely choose items, order, and paths, possibly hesitating, changing plans, backtracking, or handling unexpected events. These differences might be considered noise in standardized collection, but they are exactly the reality that robots entering home environments must understand.

The difficulty of long-horizon tasks is not just 'longer videos': earlier decisions change subsequent executable actions, objects may leave and re-enter the field of view, and subtasks may be interrupted, reordered, or continued elsewhere. Models must continuously remember task goals, object states, and completed steps, rather than treating activities as a sequence of independent actions.

Based on ACE-Data-0, DaXiao has built a three-level embodied perception evaluation benchmark: the first level is low-level signal inference, studying whether models can predict contact and tactile information from visual observations; the second level is scene composition recovery, evaluating models' ability to recover human and hand motion states under occlusion, complex poses, and continuous motion; the third level is embodied interaction understanding, requiring models to recover complete hand-object interaction processes such as approach, grasp, adjustment, and release from first-person and third-person videos.

The research team evaluated over 30 representative methods, and the results show that existing models still have significant capability gaps in contact, severe occlusion, first-person ego-motion, extreme viewpoints, and long-horizon tasks. This benchmark not only judges task success but also attempts to answer where exactly the model fails—whether it is failure to perceive contact, incorrect object state estimation, loss of task context, misunderstanding of action order, or inaccurate hand motion recovery.

The value of ACE-Data-0 extends beyond evaluation. Since the multimodal data is in a unified spatiotemporal framework, it can serve imitation learning, robot policy learning, world models, VLA, and transfer of human demonstrations to different robot embodiments.