UniTracker: Learning Universal Whole-Body Motion Tracker for Humanoid Robots

Kangning Yin1,2*, Weishuai Zeng2,3*, Ke Fan1,2, Zirui Wang2,4, Qiang Zhang5,
Zheng Tian6, Jingbo Wang2, Jiangmiao Pang2, Weinan Zhang1,2,
Shanghai Jiao Tong University1, Shanghai Artificial Intelligence Laboratory2,
Peking University3, Zhejiang University4,
The Hong Kong University of Science and Technology (Guangzhou)5, ShanghaiTech University6
arXiv

Abstract

Achieving expressive and generalizable whole-body motion control is essential for deploying humanoid robots in real-world environments. In this work, we propose UniTracker, a three-stage training framework that enables robust and scalable motion tracking across a wide range of human behaviors. In the first stage, we train a teacher policy with privileged observations to generate high-quality actions. In the second stage, we introduce a Conditional Variational Autoencoder (CVAE) to model a universal student policy that can be deployed directly on real hardware. The CVAE structure allows the policy to learn a global latent representation of motion, enhancing generalization to unseen behaviors and addressing the limitations of standard MLP-based policies under partial observations. Unlike pure MLPs that suffer from drift in global attributes like orientation, our CVAE-student policy incorporates global intent during training by aligning a partial-observation prior to the full-observation encoder. In the third stage, we introduce a fast adaptation module that fine-tunes the universal policy on harder motion sequences that are difficult to track directly. This adaptation can be performed both for single sequences and in batch mode, further showcasing the flexibility and scalability of our approach. We evaluate UniTracker in both simulation and real-world settings using a Unitree G1 humanoid, demonstrating strong performance in motion diversity, tracking accuracy, and deployment robustness.


Kung Fu

Dance

Others motions like walk, run, jump, stretch

Challenging motions by fast adaption

Downstream Applications


1. Text-to-Motion Generation


"A person is punching forward. "

"A person is squating down. "

"A person is dancing the waltz. "

2. Video-based Estimation

Approach Overview


BibTeX

@misc{yin2025unitrackerlearninguniversalwholebody,
      title={UniTracker: Learning Universal Whole-Body Motion Tracker for Humanoid Robots}, 
      author={Kangning Yin and Weishuai Zeng and Ke Fan and Zirui Wang and Qiang Zhang and Zheng Tian and Jingbo Wang and Jiangmiao Pang and Weinan Zhang},
      year={2025},
      eprint={2507.07356},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2507.07356},}