Researchers led by R. Gnana Praveen have built ReLAP, a way to pretrain robot policies on videos that have no action labels, according to a paper on arXiv. It learns compact latent actions from neighbouring video frames and predicts several steps ahead in the representation space of a frozen JEPA model, so the actions stay consistent over time.

The learned representation is passed to a vision-language model, which is then adapted to robot demonstrations. When deployed, the extra branches are removed and the model predicts physical actions directly, which makes it simpler to run.

With 11 million pretraining samples and 1.06 billion trainable parameters, ReLAP reached 97.9% on the standard LIBERO simulation benchmark and 70.0% on LIBERO-Plus perturbations it was not trained on. It ranked second among the compared methods on the LARY latent-action benchmark.

Source: arXiv Robotics (cs.RO)

Background

R. Gnana Praveen

Robotics researcher

Robotics researcher.