UC Berkeley and Nvidia show AI agents that write and test robot code in simulation

Researchers at UC Berkeley and Nvidia have presented an approach they call agentic robotics, in which teams of AI agents write robot-control programs, try them out in simulation and fix the code when something fails. UC Berkeley professor Ken Goldberg is presenting the results at the Conference on Robot Learning, and the work is published on arXiv.

Their Graph-as-Policy framework splits a robot task into separate skills, such as locating an object, planning how to grasp it and moving the gripper clear of obstacles. AI agents join these skills into a program and rehearse it across variations of the task with Nvidia's Isaac simulation libraries before it is used. In one test, OpenAI's Astra model built a physics simulation in under an hour from a short video of a robot wiping ink off a metal bar with a sponge, and the program the AI then produced worked on a real robot.

Berkeley-based Ambi Robotics applied the methods this year to its parcel-sorting data. It says each robot can now sort more than 15,000 extra packages a year, and the update runs on 30% of its US fleet.

Goldberg said the approach does not need any extra training data. The team is releasing open-source code built on Nvidia's Isaac framework and a benchmark for robots doing industrial tasks. Spencer Huang, Nvidia's director of robotics engineering, said building robots for real-world settings is still slow and hard to scale.