Researchers led by Mingcheng Zhu have built VGE, a model that finds the objects a sentence refers to in an image, whether there are none, one or several, according to a paper on arXiv. For robot manipulation it has to stay accurate when the same request is worded differently or wrapped in more text.
Grounding models that predict boxes directly are thrown by such changes in wording, while vision-language models often miss some targets or fail to say that a target is absent. VGE joins a frozen vision-language model to a trainable grounding model, so the language model's understanding guides which objects are picked while the boxes are still predicted directly.
Across three datasets, VGE had the highest average F1 score of the methods compared on all five test splits, and its scores moved by no more than 1.2 points between wordings. It was also best on the dataset of robot scenes, and it ran 5.3 to 13.0 times faster than the baseline built on the same vision-language model.



