Researchers led by Hao Chen have built LaST1.0, a world-action model for robot manipulation that predicts the future in a compact latent space rather than as full video, according to a paper on arXiv. Predicting whole videos spends much of a model's capacity on details of appearance that do not matter for control.

LaST1.0 instead predicts geometry-aware targets taken from future views of the wrist camera, focusing on how the robot and objects interact. It drops the separate vision encoder for a light patch embedder, giving the backbone direct access to fine visual details, and adds an attention mechanism called LaST-Attention for structured links across space, time and modalities.

Across two robots, five demanding manipulation tasks and three simulation benchmarks, RLBench, LIBERO and LIBERO-Plus, it achieved the best success rates published so far and generalised to new scenes, the authors say.

Source: arXiv Robotics (cs.RO)

Background

Hao Chen

Robotics researcher

Robotics researcher.