UniTracker: Learning Universal Whole-Body Motion Tracker for Humanoid Robots

Kangning Yin1,4*, Weishuai Zeng2,4*, Ke Fan1,4, Zirui Wang3,4, Qiang Zhang5,
Zheng Tian6, Jingbo Wang4, Jiangmiao Pang4, Weinan Zhang1,4,
Shanghai Jiao Tong University1, Peking University2, Zhejiang University3,
Shanghai Artificial Intelligence Laboratory4,
The Hong Kong University of Science and Technology (Guangzhou)5, ShanghaiTech University6,
arXiv

Abstract

Humanoid robots require expressive and generalizable whole-body motion control to operate effectively in human-centric environments. Prior work often relies on teacher-student frameworks to distill policies under partial observations, but struggles to retain motion diversity during deployment. In this work, we propose a simplified yet effective pipeline that leverages a Conditional Variational Autoencoder (CVAE) within a teacher-student framework to preserve motion expressiveness and enable strong generalization. Unlike previous methods that rely purely on MLP-based DAgger to transfer information from the teacher, we leverage a CVAE prior to model motion diversity, allowing the student to preserve and generalize expressive behaviors, resulting in a single policy capable of tracking a wide range of whole-body motions. Extensive experiments in both simulation and real-world settings demonstrate that our approach significantly improves generalization across diverse motion types and unseen references, making it a practical solution for expressive whole-body humanoid control.


Kung Fu

Dance

Others motions like walk, run, jump, stretch

Downstream Applications


1. Text-to-Motion Generation


"A person is punching forward. "

"A person is squating down. "

"A person is dancing the waltz. "

2. Video-based Estimation

Approach Overview


BibTeX

@misc{yin2025unitrackerlearninguniversalwholebody,
      title={UniTracker: Learning Universal Whole-Body Motion Tracker for Humanoid Robots}, 
      author={Kangning Yin and Weishuai Zeng and Ke Fan and Zirui Wang and Qiang Zhang and Zheng Tian and Jingbo Wang and Jiangmiao Pang and Weinan Zhang},
      year={2025},
      eprint={2507.07356},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2507.07356},}