RAGu retrieves mismatched demonstrations to guide a robot policy in new settings

Researchers led by Subin Kim have presented RAGu, a method that improves vision-language-action models in a new setting by retrieving demonstrations from datasets recorded elsewhere, such as DROID, in a paper posted on arXiv.

Such models still struggle with small changes such as a new camera angle, background, object or robot body, the authors write. RAGu calibrates source demonstrations geometrically to fit the target setup, picks the trajectory that best suits the base model from a large pool of pre-warped candidates in a coarse-to-fine search, and uses it to guide the model's flow-matching inference.

Tested in the real world, in simulation, from simulation to reality and across embodiments including humanoid manipulation, it raised real-world success rates from 20% to 80% and beat the closest baselines in simulation, the authors report.