Benchmark tests whether robots pick the camera views a manipulation command needs

Yina Jian and Mingkun Xu have introduced Seeing What Matters in an arXiv paper, a benchmark for whether a robot gathers the specific views that make a manipulation command executable, for example when to look at the cup and when at the portafilter. It covers a coffee workstation, drawers and cabinets, and a microwave, with 860 fixed evaluation cases in simulated tabletop manipulation.

A reference pipeline that tracks what the robot knows about the scene and ranks views by missing evidence reached 98.8% closed-loop success and 98.9% on functional out-of-distribution cases. Seven vision-language models tested on the same cases reached only 26 to 43%.