Picture a robot watering flowers in order, holding the spout a few centimetres above each bloom. Getting that right is easy for people and hard for general AI, and a Chinese team behind RoboTracer argues the problem is how vision models represent space.
Why general models get lost in 3D
RoboTracer drops XYZ coordinates in favour of pixel positions plus absolute depth, a shortcut the team says lets a general vision-language model learn 3D far faster. The model is built to fix missing scale sense, correcting even centimetre-level errors, and it scores every reasoning step rather than only the final answer. A structured format forces output such as [Measuring] [first flower height]: 0.35m, and a per-attribute reward tolerates physical error within set bounds, so a derivation that respects real-world constraints earns a positive signal.
Where 4.5 million trajectories came from
The training data comes from three pipelines. One turns 2D photos into low-cost pseudo-3D scenes. A second plans an avoidance path first, then reverse-infers the manipulation command. A third pulls gripper trajectories straight out of messy real-world video. Together they produced 4.5 million trajectories.
One 3D route book drives both arm and humanoid
On the same 3D tracking benchmark, off-the-shelf vision-language models scored just 3 per cent. In real tests the RoboTracer setup ran stably on a UR5 arm, and cross-form reuse on a Unitree G1 humanoid was difficult but worked. The team’s closing point is that embodied intelligence does not need a universal black box, and the next break in spatial understanding will come from models that reason in physical units.
More images from the source report:
Editor’s note: This is an adapted translation of the original Leiphone report. It has been trimmed and restructured for readability for an international business audience.
Translated and adapted from Leiphone (https://www.leiphone.com/category/private/sgaqS952UXuoJUVJ.html).