Researchers led by Muhammad Farooq have tested whether vision-language models can predict robot failures before they happen, according to a paper on arXiv. They tried Qwen, InternVL and LLaVA-OneVision on video and odometry data from three robot case studies, HuRoN, PAL Robotics and CrowdBot.

The models classified situations as safe, potentially unsafe or unsafe, and the team measured how uncertain they were and how that uncertainty related to their accuracy. Qwen gave the best overall classification and the most reliable predictions, while InternVL warned earlier in several setups. Whether information about the robot's state and recent history helped depended on the model and the goal.

The authors conclude that general-purpose vision-language models can support early safety monitoring of robots, but the best choice depends on whether accurate safety judgments, reliable confidence or early warning matters most.

Source: arXiv Robotics (cs.RO)

Background

Muhammad Farooq

Robotics researcher

Robotics researcher.