Researchers led by Yige Liu have built Emo-H2A, which drives the expressions of a robot face directly from the hidden states of a large language model while it writes its reply, according to a paper on arXiv. Language models have given humanlike robots fluent speech, but their faces still lag behind the conversation.

Without labelling an emotion, a training-free method called EmoManifold reads a point for valence, arousal and dominance from each hidden state and moves the face token by token, so the expression changes as the reply unfolds. The reading was checked on six language models, and the same projection drives the team's facial robot, which has 29 degrees of freedom.

Driven this way, the face changed its expression in the matching direction for 87.5% of shifts in tone within a reply, while holding one label for the whole reply did about as well as chance. It began moving after 0.96 seconds instead of waiting 10 to 26 seconds for the reply to finish.

Source: arXiv Robotics (cs.RO)

Background

Yige Liu

Robotics researcher

Robotics researcher.