Researchers led by Jinhao Zhang have built CoT-WAM, which trains a vision-language model and a world-action model together so that a robot gains both an understanding of the scene and foresight about the physical world, according to a paper on arXiv. World-action models predict how a scene will change and which actions to take, but they are weak at understanding meaning and planning long tasks.
A learned embedding carries information between the two models. During training it is supervised with text and goal images so that it encodes what the task is meant to achieve, and a two-stage training scheme optimises the models together.
On six benchmarks, CoT-WAM achieved the best published results on LIBERO, CALVIN and RoboCasa-Kitchen, generalised better without fine-tuning on LIBERO-Plus and LIBERO-Pro, and planned long tasks better on RoboCasa365. The authors say experiments on real robots also showed that it works.



