RoboMoCap pairs 200 hours of motion capture with head-mounted video to teach humanoids

Researchers led by Mayur Deshmukh have released RoboMoCap, a dataset that pairs first-person video with marker-based optical motion capture to help humanoid robots learn to move and handle objects from human demonstrations, in a paper posted on arXiv.

Learning such skills needs accurate whole-body motion matched with first-person views, especially for the legs and feet, which a head camera often cannot see, the authors write. RoboMoCap holds more than 200 hours of recordings from 14 participants, captured with six head-mounted cameras, visual and inertial sensors on the wrists and optical motion capture of the full body at 120 Hz.

It covers walking at different speeds, turning, moving through tight spaces and over terrain, carrying objects and handling them while moving. The authors tested existing methods on three tasks: rebuilding full-body motion from first-person video, estimating foot contact and placement, and retargeting human motion to humanoids.