Researchers led by Yanghong Liu have built PriDex, a system that turns videos of people into demonstrations for robots with two dexterous hands, according to a paper on arXiv. Motion estimates from video are uncertain, and human and robot bodies differ, so copying the motion directly is hard.

PriDex uses how people handle objects in the video to pick grasps suited to the robot. Reinforcement learning in simulation then refines the motion within the robot's limits, guided by the object's path and by the relative path of the two wrists. When the object's motion is hard to estimate, for example when it is hidden, PriDex relies more on the wrists.

The refined policies generate robot demonstrations that train a vision-based diffusion policy. The team tested PriDex on four two-handed tasks: pouring, sweeping into a dustpan, unscrewing a bottle cap and cleaning a plate.

Source: arXiv Robotics (cs.RO)

Background

Yanghong Liu

Robotics researcher

Robotics researcher.