Counterfactual world models help robots work out which action would have worked

Randall Balestriero of Brown University and AMI Labs, Kerda Varaku and Turing Award winner Yann LeCun of New York University and AMI Labs describe counterfactual world models in a preprint, The Register reported. World models, used to train robots and other physical AI, predict what an action will lead to; the new technique lets them estimate what would have happened had a different action been taken.

The method takes the gap between what the model predicted and what actually happened, and adds it to the prediction for the other action, Balestriero told The Register. It works on any world model that predicts the average next state, and the team ran it on off-the-shelf DINOv2 and MAE encoders without fine-tuning. In simulated tasks, choosing actions this way reached the goal in 35 to 69% of surprise events, against 14 to 23% for a planner that knew the true simulator but ignored the observed outcome.

The paper was given to The Register before its release on arXiv, expected this week. Balestriero said the models could help a robot retry an action after a surprise, adapt to lasting faults such as a miscalibrated motor, and help people understand failures.