Researchers led by Mariana Sosa Guzmán have built a system that lets a robot arm reason about hazards between objects in a scene and avoid them while it works, they report in a paper on arXiv. Safety depends not only on geometry but on the task, the surroundings and how objects interact, they write.
A vision-language model receives a colour and depth image and a task instruction in plain language. It first decides whether the task should be refused because it could end in harm that cannot be undone. Otherwise it names the object pairs that pose a preventable hazard and the spatial relations to avoid, and segmentation and depth data then place those objects in 3D and build continuous risk fields around the dangerous regions.
An obstacle avoidance module uses the fields to correct the robot's planned motions while it moves. The team tested the framework on preventable hazards, tasks that should be refused and safe versions of them, in simulation and in the real world.
Background
Mariana Sosa Guzmán
Robotics researcher
Researcher in the Department of Computer Science at McGill University.



