Robot planners that search hard against learned human-preference rewards can backfire

Soovadeep Bakshi and Apurba Bose have measured in an arXiv paper what happens when a robot picks its actions by searching for the highest score from a reward model trained on people's preferences. In simulated tasks, harder MPPI search cut the true return by up to 76%: the drawer-opening task succeeded in none of 20 runs with human labels, compared with all 10 runs of the starting policy, while the reward model's accuracy on held-out labels stayed between 0.75 and 0.97.

They prove that a regularised form of MPPI, with its temperature set by how much the reward model's errors vary, is optimal in the worst case. Calibrated on 100 proposal pairs, it was the safest fix they tested, keeping on average 86 to 94% of the starting policy's return on two Meta-World tasks, though it is cautious where search would help.