DexForesight raises dexterous robot hand success from 49.4% to 60.5% in simulation

Researchers led by Zhongxi Chen have presented DexForesight, a framework that gives dexterous robot hand policies a sense of how the hand and scene will move without explicitly predicting the future, in a paper posted on arXiv.

During training, a spatial world model learns from scene depth, the hand's geometry and demonstrations to predict 3D motion of points on the hand, including fingers hidden from the cameras, and changes in visual features. The world model is then frozen and the policy's queries are aligned with its predictions. Only the policy is kept for use, working from colour images, language and the robot's state without depth.

On 11 DexJoCo simulation tasks, it raised the base policy's overall success rate from 49.4% to 60.5%. On three real-robot dexterous tasks it succeeded 58.3% of the time, against 50.0% for the strongest baseline, the authors report.