CRONOS: Benchmarking Counterfactual
Physical Consistency in Video Models
1University of Freiburg, Germany
2Max Planck Institute for Informatics, Germany
We introduce CRONOS, an intervention-based benchmark to evaluate counterfactual physical consistency: whether a model's predictions of physical events respond appropriately to controlled changes in the visual input, such as variations of camera viewpoint, scene, object category, and object appearance. Built in a photorealistic Unreal Engine environment, CRONOS enables controlled, high-fidelity generation of videos across diverse scenes and dynamics, including sampling multiple physically plausible outcomes of the same physical event. In contrast to previous benchmarks, CRONOS systematically intervenes on four factors — camera viewpoint, scene, object category, and object appearance — while keeping the underlying physical event type, such as a fall, collision, or occlusion, fixed. Our evaluation of recent open-source video generators reveals substantial failures in counterfactual physical consistency: prediction quality for the same physical event type is affected by appearance, environment, and particularly viewpoint changes.
A model's ability to produce predictions of physical events that remain coherent across counterfactual variants of the visual input, such as changes in viewpoint, scene context, object appearance, or object category.
CRONOS frames video model evaluation as a controlled counterfactual experiment. Each physical event is rendered into multiple counterfactual observations by intervening on one scene factor at a time while holding all others fixed.
We span three fundamental rigid-body interactions, chosen to isolate distinct aspects of physical reasoning.
An object rolls across a surface and falls from an edge — testing prediction across changing contact conditions and free-fall motion.
One object impacts another — testing whether generated videos preserve plausible interaction dynamics, temporal/spatial coherence, and object permanence.
An object rolling across a smooth surface becomes fully occluded behind another object and later reappears — testing whether models capture long-range temporal coherence.
Physical parameters such as mass, elasticity, or friction cannot be inferred uniquely from one or a few conditioning frames, so a single "correct" future would unfairly penalize models. CRONOS therefore renders three rollouts of every sequence from the same conditioning signal, using a high, medium, and low value of a key physical parameter. During evaluation, each generated video is compared against all rollouts and scored against the best match. In total, the benchmark contains 2,025 videos.
For each event, CRONOS applies one intervention at a time while holding the remaining variables fixed.
The rendering viewpoint is changed. Probes whether models disentangle scene geometry from observed motion.
Visual attributes (e.g., color) of the primary object are changed without altering physical parameters — isolating whether models disentangle appearance from dynamics.
The full scene is replaced — background, lighting, and event-relevant layout details change. Alters how the event unfolds across scenes, testing whether models adapt the event dynamics coherently to the new context.
The object of interest is replaced with another, changing both visual properties and physical parameters (mass, friction). Probes generalization across instances.
CRONOS decomposes generation quality into complementary per-video metrics, validated against human ratings.
To measure counterfactual consistency, we compute each model's sensitivity to interventions: how much these metrics change under an intervention, relative to resampling the seed.
We evaluate Cosmos2.5 (2B, 14B), Cosmos3-Super, CogVideoX1.5 (5B), MAGI-1 (4.5B), Wan2.2 (14B), HunyuanVideo-1.5, and MiniMax-H3 in the I2V setting, and Cosmos and MAGI-1 additionally in the V2V setting.
All evaluated open-source video models struggle to reliably generate short clips of basic rigid-body physics. Even the strongest model (MiniMax-H3) achieves only a 36% success rate, with most models below 20%.
Models that generate stable backgrounds and objects can still fail to produce the expected physical event. On falling objects, Wan2.2 has the best background stability and motion similarity, yet its physical plausibility falls below the calibrated threshold.
Generation quality changes substantially across interventions, especially for scene and viewpoint changes. The models with the highest success rates are generally the least sensitive (Spearman ρ = −0.87).
V2V improves not only motion fidelity (as expected) but also background and object stability and the consistency of generation quality across interventions.
Scaling Cosmos2.5 from 2B to 14B parameters improves neither generation quality nor consistency under interventions (V2V success drops from 23% to 14%). Model size alone does not guarantee counterfactual physical consistency.
Performance averaged across all videos. The table reports the headline Success rate; per-metric scores (background stability, motion similarity, appearance stability, 3D shape, physical plausibility) are plotted in the radar on the right — all axes on a 0-1 scale. Solid lines are V2V, dashed lines I2V; by default the radar shows I2V variants. Click a row to toggle that model on or off in the radar.
| Rank | Model | Mode | Success |
|---|---|---|---|
| 1 | MiniMax-H3 | I2V | 0.36 |
| 2 | Cosmos3-Super | V2V | 0.30 |
| 3 | Cosmos2.5-2B | V2V | 0.23 |
| 4 | Wan2.2-14B | I2V | 0.20 |
| 5 | Cosmos3-Super | I2V | 0.19 |
| 6 | Cosmos2.5-14B | V2V | 0.14 |
| 7 | Cosmos2.5-2B | I2V | 0.12 |
| 8 | Cosmos2.5-14B | I2V | 0.08 |
| 9 | HunyuanVideo-1.5 | I2V | 0.06 |
| 10 | MAGI-1-4.5B | I2V | 0.02 |
| 10 | CogVideoX1.5-5B | I2V | 0.02 |
| 12 | MAGI-1-4.5B | V2V | 0.01 |
Side-by-side comparisons of generated futures for the same physical event. While most models preserve coarse scene structure, they fail to generate plausible motion, static camera viewpoints, or consistent objects.
Generated video and four intervention variants for a selected model. All videos are generated with the same random seed. A counterfactually consistent model would produce stable, comparable predictions across all variants.
AK acknowledges support via his Emmy Noether Research Group funded by the German Research Foundation (DFG) under grant number 468670075.
This research was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under grant number 539134284, through EFRE (FEIH_2698644) and the state of Baden-Württemberg.
@misc{begiristain2026cronos,
title={CRONOS: Benchmarking Counterfactual Physical Consistency in Video Models},
author={Le{\'o}n Begiristain and Olaf D{\"u}nkel and Adam Kortylewski},
year={2026},
eprint={2605.23699},
archivePrefix={arXiv},
primaryClass={cs.CV},
url={https://arxiv.org/abs/2605.23699},
}