Q-OIL method lets robots learn faster from people stepping in to correct them

A paper posted on arXiv by Pranav Teegavarapu and seven co-authors, including Natasha Jaques and Abhishek Gupta, describes Q-OIL, a reinforcement learning method for robots that learn from human interventions. Studying intervention data from human participants, the team found that people tend to step in when a robot stops making progress and that their corrections are often less than ideal, and built a progress-based model of intervention that ranks algorithms more like real users do.

Q-OIL adds optimistic bonuses at the moments when a human intervened, nudging the robot's exploration towards those corrections, and uses separate critics to keep training stable. The authors report 10 to 50% higher success than competing methods in simulation and on real robots, and better results for pretrained vision-language-action models on two two-armed manipulation tasks.