MDF++ combines touch and vision in a world model for contact-heavy robot tasks

Zixuan Huang, Jinhao Yu, Zachary Charlick and Dmitry Berenson have described MDF++ in an arXiv paper, a world-action model that predicts both a robot's next actions and what it will see and feel. Instead of relying on frozen pretrained encoders, it learns representations of visual and tactile data end to end with a diffusion backbone, using multi-task learning across the two senses to avoid feature collapse.

On contact-rich manipulation tasks, including screwing in a bolt and fitting a nut on a real robot, MDF++ outperformed world-action models built on frozen pretrained representations by up to 50%, the authors report.