Embodied vision-language models guide robot planning and manipulation, but predicting coordinates for grounding is slow or brittle in existing designs, according to a paper on arXiv. Discrete tokenization breaks spatial continuity, while plain point regression fails on multi-contact and irregular targets.
Ruimeng Liu and colleagues propose PhysMoE, which shares lower layers for multimodal context and routes upper layers to specialized pixel and text experts by physical modality, removing gradient interference between them.
Each coordinate is encoded as a single token reusing the visual encoder's positional embeddings, which the authors say injects 2D spatial priors and speeds up decoding.
