Researchers led by Junseong Pyo have built PAIR, a simulated smart home where robots can operate appliances by their control panels or through device APIs, according to a paper on arXiv. It is based on Matter, the industry standard for smart-home devices, and has 84 smart devices whose panels and APIs act on a shared state.

From a survey of 1,443 requests, the team built 40 tasks that all need handling objects but can be solved without the API, with 340 human demonstrations for training.

Two fine-tuned vision-language-action policies completed none of the 120 test episodes, alone or with a planner, with or without API access. Even with perfect plans, over 90% of failed runs stalled at moving around, grasping or opening a door. With a perfect body in place of the robot, the best planner still got through only 57.3% of its plan.

Source: arXiv Robotics (cs.RO)

Background

Junseong Pyo

Robotics researcher

Robotics researcher.