3D Menagerie: Modeling the 3D Shape and Pose of Animals

By Silvia Zuffi, Angjoo Kanazawa, David W. Jacobs, Michael J. Black. 2016

504 citations recorded by the source corpus.

There has been significant work on learning realistic, articulated, 3D models of the human body. In contrast, there are few such models of animals, despite many applications. The main challenge is that animals are much less cooperative than humans. The best human body models are learned from thousands of 3D scans of people in specific poses, which is infeasible with live animals. Consequently, we learn our model from a small set of 3D scans of toy figurines in arbitrary poses. We employ a novel part-based shape model to compute an initial registration to the scans. We then normalize their pose, learn a statistical shape model, and refine the registrations and the model together. In this way, we accurately align animal scans from different quadruped families with very different shapes and poses. With the registration to a common template we learn a shape space representing animals including lions, cats, dogs, horses, cows and hippos. Animal shapes can be sampled from the model, posed, animated, and fit to data. We demonstrate generalization by fitting it to images of real animals including species not seen in training.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Learning Category-Specific Mesh Reconstruction from Image Collections (2018, methodology): “These techniques have since been applied to model human bodies [1,19], hands [27,17], and more recently on quadruped animals [40].”
One-2-3-45: Any Single Image to 3D Mesh in 45 Seconds without Per-Shape Optimization (2023, methodology): “By combing 2D image encoder and 3D generators, they generates 3D data in various representations, including 3D voxels [18, 79, 10, 81, 80, 85], point clouds [15, 88, 19, 1, 44, 90], polygon meshes [30, 73, 77, 51], and parametric models [54, 94, 95].”
From Synthetic to Real: Unsupervised Domain Adaptation for Animal Pose Estimation (2021, background): “Existing works overcome this problem by transferring knowledge from other more accessible domains such as synthetic animal data [23, 5, 46, 47, 48] or human”
Perspectives in machine learning for wildlife conservation (2021, background): “Recent work defines the SMAL (Skinned Multi Animal Linear) model, a 3D articulated shape model for a set of quadruped families (111).”
Zero-1-to-3: Zero-shot One Image to 3D Object (2023, unknown): “CAD models) or category-specific priors [37, 21, 36, 67, 68, 66, 25, 24].”
Nerfies: Deformable Neural Radiance Fields (2020, background): “Domain-Specific Modeling: Many reconstruction methods use domain-specific knowledge to model the shape and appearance of categories with limited topological variation, such as faces [5, 7, 9], human bodies [30, 58], and animals [12, 64].”
Who Let the Dogs Out? Modeling Dog Behavior from Visual Data (2018, methodology): “To visualize the dog movements we use a 3D model of a dog from [57].”
3D-CODED: 3D Correspondences by Deep Deformation (2018, methodology): “In SMALR [50], Zuﬃ et al. showed that the SMAL model could be generalized to other animals using only an image dataset as input, demonstrating it on 17 additional categories.”