Researchers led by Jiachen Tao have built CoachSI, a way for a robot policy to improve on long tasks with help from a vision-language model, according to a paper on arXiv. The frozen, off-the-shelf model watches the robot and suggests the next step, phrased like the policy's own simple instructions, but never gives an action itself.
Successful coached runs are then labelled with the task's original instruction, and the policy is fine-tuned on them before it works alone. Its unassisted success on long tasks in unseen scenes rose from 47.4% to 59.3%. That beat fine-tuning on human demonstrations by 9.8 points, and fine-tuning on its own successes with the same amount of data by 7.3 points.



