A paper titled Masked Visual Actions for Unified World Modeling appeared on arXiv with a roster that reads like a who’s who of spatial and embodied AI: Fei-Fei Li, creator of ImageNet and now focused on spatial intelligence; Wu Jiajun, a Stanford assistant professor in physical reasoning who just won the IJCAI 2026 Computers and Thought Award; Zhang Lvmin, author of ControlNet; Gordon Wetzstein; robot foundation-model advocate Huang Wenlong; and Harvard assistant professor Yilun Du. What surprised observers is that Li and Du represent two opposed embodied-AI camps, one arguing AI should think in the image it can see, the other that everything should first be compressed to abstract numbers, and their last joint work was five years ago at MIT.
The missing translation interface
The paper’s core point is that today’s video models understand physics but cannot issue commands. A model fed the start of a kettle tilting will continue the scene plausibly, water pours into the cup, yet it has never done anything, and faced with a robotic arm told to move a cup left, it freezes, because it does not know how ‘left’ becomes a motor signal. The gap is translation. A robot speaks a bare language of joint angles, centimetres moved and gripper closure, which does not match the video model’s language of how pixels flow. Past attempts, UniPi’s text-guided video planning, CMU and Meta’s Track2Act point tracks, and Ctrl-World’s raw joint angles, all forced the video model to learn a foreign tongue.
Speak the model’s native language
The method instead speaks to the video model in its own terms. Step one: use SAM, Segment Anything Model, to mask the robot and the object in every frame, producing two moving trajectories, one for how the robot moves, one for how the object changes. That, the paper says, is the entire definition of an action: no angles, no coordinates, just coloured blocks moving through the frame. The model then runs both forward, as a world simulator, and backward, as an action generator. With only 15 hours of fine-tuning, robot manipulation generalised from a single arm to an unseen two-arm setup.
The industrial implication is large. If a video model can drive a robot through a pixel-mask interface, the cross-embodiment problem, where swapping one arm crashes the system, eases, and a factory or sorting station would no longer need a bespoke model per machine. That directly challenges the premise behind the many startups valued in the billions on a single robot foundation model.
Read the original report (LeiPhone)
Translated and adapted from LeiPhone (leiphone.com).