Einsia AI Releases World Models' Last Exam in Physics, Top Score Only 57.76

Navers Lab, under Einsia AI, released a benchmark named "World Models' Last Exam in Physics," used to test whether video generation models obey physical laws. The benchmark contains 40 video generation tasks, covering 9 categories of physical phenomena including mechanics, optics, fluids, phase change, and electromagnetism, and tests 8 video generation models in total.

The task format is straightforward: each task provides an initial image and a generation prompt, requiring the model to output a video showing the subsequent physical process. The model needs to preserve the object and scene relationships in the image, keep the specified process continuously occurring, and make the motion and changes within it conform to the corresponding physical laws.

Looking at the results, among the 8 participating models, Seedance 2.5 achieves the highest average composite score at 57.76, with MiniMax H3 close behind at 54.89. The top score being under 60 constitutes the most intuitive conclusion of this evaluation.

The evaluation divides scoring into a consistency score C and a physics score P, both on a 0-100 scale, with consistency accounting for 15% and physics for 85% of the total score S. The consistency score C is given by the vision-language model Qwen3.6-27B based on a dedicated prompt for each task, measuring whether the subjects, events, and scenes in the video remain coherent and providing the observational basis needed to complete the task.

The mechanism for the physics score P is completely different: a program extracts observable quantities such as trajectories, periods, angles, and liquid levels from the video through segmentation, tracking, geometric fitting, and event detection, then calculates errors and scores according to each task's metrics, without needing comparison with a reference video and without relying on a vision-language model to directly judge physical correctness.

The physics part also has an inclusion condition: the physics score record is only calculated when the consistency score C ≥ 80, otherwise the physics part is not included in the total score. According to the example given by the source, when C is 100 and P is 60, the total score is 66; if C is only 60, even if a relatively high physics score is recorded, the total score is only 9. This design is intended to prevent problems such as missing subjects or key events not occurring from being masked by high scores on local physics metrics.

Taking the simple pendulum task as an example, under the small-angle approximation and the same gravity conditions, the square of the ratio of periods should equal the ratio of pendulum lengths. The error scale is preset at 0.20 and is consistent for all models: an error of 0 scores 100, a relative error of 20% scores 50, and a relative error of 40% scores about 33.3.

The free-fall task best explains why the two sets of scores must be separated. In this task, the average consistency score across models reaches 98.75, while the average physics score is only 13.88. The ball can fall completely and coherently, but the measured motion may still deviate from the physical relationship required by the task.

Specifically for the models, Seedance scores 29.50 and Wan scores 22.72, both showing approximately uniform falling motion that does not conform to physics. The source attributes this phenomenon to the current models' insufficient capability in modeling temporal dynamics.

This is precisely the core concern of the evaluation design: a ball can always fall along a continuous trajectory, but the relationship of its position changing over time may still fail to conform to the law of free fall. Visual coherence does not equal physical correctness.

The phase change (ice melting) task shows extreme divergence: only Seedance 2.5 scores 95.61, while the rest of the models score at most 15. Among them, the ice block generated by MiniMax does not melt and does not follow the Prompt, scoring 0.00. Based on this, the source judges that most models lack understanding of basic physical laws.

Optics is likewise a common difficulty for the models. In the "refraction and total internal reflection of light" task, the average composite scores of all 8 models do not exceed 40. The models can generate the visual appearance of light beams and medium interfaces, but find it difficult to continuously maintain the corresponding physical constraints during the process of light path changes.

In terms of specific scores, MiniMax H3 can generate partial refracted light rays but the result is wrong, scoring 31.67; Cosmos cannot generate it, scoring 0.00.

To evaluate the agreement between physics quantity scoring and human judgment, the research team invited 10 annotators to conduct manual evaluation of 320 videos covering the 40 tasks, and compared the measurement method with direct scoring by a vision-language model (Direct VLM).

The comparison is divided into two types: one first has annotators score the videos, then checks whether the automatic scoring and human scoring agree on the ranking of video quality; the other directly shows two videos and lets annotators judge which one better conforms to physical laws. In the two comparisons, this method improves by 8.18 and 7.50 percentage points respectively over Direct VLM, and there is still a gap of about 5 percentage points in agreement rate with humans.

Based on this, the source believes that extracting physical quantities from videos and scoring according to specific relationships is closer to human judgment than letting a vision-language model directly judge physical correctness. But the gap of about 5 percentage points also shows that automatic scoring and human judgment are not yet fully consistent.

It should be made clear that this benchmark was released by Navers Lab, under Einsia AI, itself, and its evaluation method and results have not yet been reproduced or verified by an independent third party. The specific scores of each model on tasks such as optics, phase change, and free fall come from a single source, and complete per-task data is not provided for cross-checking; the source also does not explain details that could affect comparability, such as the specific versions, inference configurations, or evaluation times of the 8 participating models.

The source article states that these results show that current generation models still do not understand the laws of the physical world in most scenarios, and that understanding physical laws remains a weak point of current models. This is the author's judgment based on the benchmark results, rather than independent confirmation.

The article ultimately reduces this research to a more fundamental question: what a world model generates is merely a "realistic-looking" image, or a future that truly conforms to physical laws. This is an open question, and the source does not give a definitive answer. The paper for this work is titled World Models' Last Exam in Physics, and a project homepage, GitHub repository, and arXiv link have been made public.