Researchers led by Yuze Wu, with Fei Gao among the authors, have built VLN-AR, a framework that lets a drone navigate by language instructions with all the computing on board, according to a paper on arXiv. It separates reasoning about the scene and the instruction from low-level control, so its predictions of 3D poses and intentions become movements the drone can actually fly.

Because 3D training data for drones is scarce, the team built a fully automatic pipeline based on 3D Gaussian splatting that generates photorealistic navigation data. A three-stage training scheme and a small agent with memory that breaks long tasks into steps improve reasoning over long flights.

Techniques such as dropping less important tokens and shrinking the model made it run 7.1 to 9.4 times faster on drones with limited computers while still navigating well, and the authors report clear gains over existing methods both in simulation and on real drones.

Source: arXiv Robotics (cs.RO)

Background

Yuze Wu

Robotics researcher

Robotics researcher.