HomeLoop benchmark tests home robots that must work with smart devices and alarms

Researchers led by Ramy ElMallah and Chi-Guhn Lee have presented HomeLoop in an arXiv paper, a simulation framework and benchmark built on Isaac Sim for home robots that must work with cameras, smart devices and residents' wearables. Its 20 tasks in five families are based on real household hazards, and the same alert can have different causes: a smoke alarm may mean a pan about to catch fire, one already burning, someone cooking or a faulty sensor.

The authors tested six open and proprietary vision-language models. The best reached a family-balanced score of 61.9%, and the models tended to overreact: failures from forbidden actions outnumbered missed required actions by more than two to one, often because a model turned off the stove before checking why the alarm went off. HomeLoop will be released with 2,100 home layouts and five robot bodies.