ScoreHMR: 3D Pose Estimation using Diffusion Guided Models

​

Example results from the new model that uses an iterative refinement approach to achieve better image-model alignment to solve inverse problems for 3D human pose and shape reconstruction.

Invention Summary:

Recovering a 3D human pose and shape from 2D images is a challenging problem to solve computationally due to the complex nature of the articulated 3D joint locations in a human body. Due to its widespread applications in a great variety of areas, such as human motion analysis, human–computer interaction (VR video games), robotics, 3D human pose estimation has recently attracted increasing attention in the computer vision community. The global animation market and VR market are estimated at US$ 260 million and US$ 16 billion in 2024 and are both estimated to grow in the next 5 years. Although several approaches to human pose estimation have been developed, optimization remains challenging. 

Rutgers researchers have developed a new method named Score-Guided Human Mesh Recovery (ScoreHMR) for 3D human pose estimation based on diffusion guided models. ScoreHMR mimics current fitting approaches, but alignment with the image observation is achieved through score guidance in the latent space of a diffusion model. ScoreHMR consistently outperforms other techniques in several settings, including single-frame model fitting, reconstruction from multiple uncalibrated views, and reconstructing humans in video sequences. 

Market Applications:

• Human motion analysis, human analytics or crowd analytics 

• Human–computer interaction, including VR gaming and other virtual or augmented environments 

• Special effects, animated movies and other computer animations 

• Security applications such as for identifying individuals and digital surveillance 

Advantages:

• Surpasses existing optimization approaches across all datasets and evaluation settings  

• Solves inverse problems for various applications without the need for model retraining and without relying on task-specific designs 

• The only approach enhancing the 3D pose performance of the state-of-the-art monocular feed-forward system 

Intellectual Property & Development Status: Patent pending. Software available for licensing and/or research collaboration. For any business development and other collaborative partnerships, contact:  marketingbd@research.rutgers.edu