Researchers led by Tian Lyu have built JointGV, an add-on that corrects the drift of visual world models, which predict what a robot will see after its actions, according to a paper on arXiv. Over long predictions, small errors in where parts are and how they relate add up and spoil planning and control.
Alongside the visual model, JointGV predicts a scene graph for the task, links the changes the two predict, and uses the positions of the graph's parts to compute a limited correction. Its training leaves the original visual model unchanged.
Attached to DINO-WM and DreamerV3 and tested on simulated cheetah, walker and finger tasks, it cut joint-position errors by 11 to 28% over 32 steps. On recordings of a real inverted pendulum, it lowered DINO-WM's error in the pole angle after 32 steps from 12.56° to 6.69°.
In a simulated pendulum control task, a controller trained with the corrected predictions succeeded in 46 of 50 episodes. Trained with the original predictions, it succeeded in none.



