Pipeline turns monocular video into a whole-body policy that controls 700 muscles

Ruiyang Kong and Takeshi Oishi have presented a pipeline that learns whole-body musculoskeletal control from human motion, which they say matters for human-like robots, assistive devices and rehabilitation, in a paper submitted to IEEE ICRA 2027.

It works in two stages. Muscle-aware retargeting adjusts motion references within hard semantic, anatomical and contact limits so that muscle-driven dynamics can follow them. Null-space exploration estimates how muscle excitations map to forces on the joints and adds bounded, anatomically structured changes along the null space of that mapping to the actions of a flow policy, which has to search a 700-dimensional action space.

On a benchmark of 24 motion-capture clips, the two stages together solved every reference, needed half as many samples as the baseline to reach the full horizon and lowered joint-centre error by 44%. The authors believe it is the first pipeline to go from monocular video to a trained policy that imitates whole-body motion with 700 muscles.