Researchers led by Jia Hu have built a world model for testing self-driving systems in which the other cars react realistically and still look right from new angles, according to a paper on arXiv. Existing world models struggle to get both the behaviour and the images right, the authors say.

The other cars first learn from recorded driving, then are fine-tuned with multi-agent reinforcement learning in closed loop against an end-to-end planner that drives from the rendered images. The car models, made of 3D Gaussians, are repaired where they were seen too little, and an editor renders each car's signal lights to match what it intends to do.

In rare closed-loop scenarios, the model cut the gap to real interaction behaviour by 85% and improved image quality from new viewpoints by 2.0 dB of PSNR over the previous best world model. The rendered signal lights changed how the ego car drove in a quarter of the scenarios.

Source: arXiv Robotics (cs.RO)

Background

Jia Hu

Robotics researcher

Robotics researcher.