Dyn-HaMR

Abstract

We propose Dyn-HaMR, to the best of our knowledge, the first approach to reconstruct 4D global hand motion from monocular videos recorded by dynamic cameras in the wild. Reconstructing accurate 3D hand meshes from monocular videos is a crucial task for understanding human behavior, with significant applications in augmented and virtual reality (AR/VR). However, existing methods for monocular hand reconstruction typically rely on a weak perspective camera model, which simulates hand motion within a limited camera frustum. As a result, these approaches struggle to recover the full 3D global trajectory and often produce noisy or incorrect depth estimations, particularly when the video is captured by dynamic or moving cameras, which is common in egocentric scenarios. Our Dyn-HaMR consists of a multi-stage, multi-objective optimization pipeline, that factors in (i) simultaneous localization and mapping (SLAM) to robustly estimate relative camera motion, (ii) an interacting-hand prior for generative infilling and to refine the interaction dynamics, ensuring plausible recovery under (self-)occlusions, and (iii) hierarchical initialization through a combination of state-of-the-art hand tracking methods. Through extensive evaluations on both in-the-wild and indoor datasets, we show that our approach significantly outperforms state-of-the-art methods in terms of 4D global mesh recovery. This establishes a new benchmark for hand motion reconstruction from monocular video with moving cameras.

AI-generated Podcast (Paper Introduction)

Gallery

In-the-wild 4D global hand motion reconstructions on HOI4D dataset.

In-the-wild 4D global hand motion reconstructions on FPHA dataset.

In-the-wild 4D global hand motion reconstructions on EgoDexter dataset.

Qualitative evaluation on InterHand2.6M dataset.

Qualitative results of hand pose estimation under complex hand interactions and hand-object interaction. In the figure, (a)-(b) are from InterHand2.6M, while (f)-(i) are from H2O, FPHA, HOI4D, and EgoDexter. Finally, (e) and (j) are from in-the-wild web videos.

Comparison with state-of-the-art hand reconstruction approach on InterHand2.6M. We compare with HaMeR under challenging interactions.

Qualitative comparison with state-of-the-art method HaMeR. The first row is from the H2O dataset, while the second & third rows are from the HOI4D dataset and web videos respectively.

BibTeX

@inproceedings{yu2024dynhamr,
  title = {Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera},
  author = {Yu, Zhengdi and Zafeiriou, Stefanos and Birdal, Tolga},
  booktitle = {arXiv preprint.},
  month     = {December},
  year      = {2024}
  }

Dyn-HaMR: Recovering 4D Interacting Hand Motion from a Dynamic Camera