MIT and NYU build FLEX, a robot AI that infers the intent behind instructions

MIT said on 9 November that a team from MIT and New York University has built FLEX, a learning platform that uses a large language model to plan a robot's movements and to work out the general rule behind a person's instruction, including principles the person did not spell out.

In the team's example, a robot carrying a cup of coffee is told not to spill it on a laptop. With FLEX it avoids the laptop and also keeps the liquid away from a smartphone and other electronic devices nearby. The system first picks out the new object to be careful with, then uses a model based on GPT-4o to infer what the instruction means in general, and finally uses computer vision to leave out objects that are not in the scene.

The researchers tested FLEX in three simulated tasks: carrying a coffee cup, moving a candle and carrying a bowling ball. The robot told similar hazards apart on its own, such as electronics, flammable materials and fragile objects. The results come from simulation only, and how well it works in real homes or factories still has to be tested.

The work was led by Andreea Bobu, a professor in MIT's department of aeronautics and astronautics, and is to be presented at the Conference on Robot Learning (CoRL) this month. The team expects it could help home service robots and care robots follow everyday speech without complicated commands.