DeepLabCut: markerless pose estimation of user-defined body parts with deep learning
By Alexander Mathis, Pranav Mamidanna, Kevin M. Cury, Taiga Abe, Venkatesh N. Murthy, Mackenzie W. Mathis, Matthias Bethge. Nature Neuroscience, 2018
4,439 citations recorded by the source corpus.
Quantifying behavior is crucial for many applications in neuroscience. Videography provides easy methods for the observation and recording of animal behavior in diverse settings, yet extracting particular aspects of a behavior for further analysis can be highly time consuming. In motor control studies, humans or other animals are often marked with reflective markers to assist with computer-based tracking, but markers are intrusive, and the number and location of the markers must be determined a priori. Here we present an efficient method for markerless pose estimation based on transfer learning with deep neural networks that achieves excellent results with minimal training data. We demonstrate the versatility of this framework by tracking various body parts in multiple species across a broad collection of behaviors. Remarkably, even when only a small number of frames are labeled (~200), the algorithm achieves excellent tracking performance on test frames that is comparable to human accuracy. Using a deep learning approach to track user-defined body parts during various behaviors across multiple species, the authors show that their toolbox, called DeepLabCut, can achieve human accuracy with only a few hundred frames of training data.
Subfields: Cell Biology, Genetics, Social Psychology
View Paper: https://www.semanticscholar.org/paper/6001895c2fcf69528d01306e9d293d9d2a4cc67b
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Cell Biology
- E. Davis
- Robert M. Kanko
- E. Laende
- K. Deluzio
- W. Selbie
- Alexander Mathis
- Mackenzie W. Mathis
- Pietro Perona
In Genetics
- David J. Anderson
- Pietro Perona
- Kristin Branson
- Joshua W. Shaevitz
- Diego E. Aldarondo
- Mala Murthy
- S. H. Wang
- Mikhail Kislin
In Social Psychology
Citations in the corpus
- PifPaf: Composite Fields for Human Pose Estimation (2019, background): “Later works accelerate the prediction time [5] and broaden the applications to track animal behavior [30].”
- Concurrent assessment of gait kinematics using marker-based and markerless motion capture (2020, positive): “…capture has the potential to alleviate some of the technical and practical issues of marker-based motion analysis without sacrificing data quality by replacing physical palpation of bony landmarks with probabilistic estimation of segment pose by highly trained neural networks (Mathis et al., 2018).”
- Machine learning and deep learning—A review for ecologists (2022, background): “…(Aodha et al., 2018 ; e.g. Fritzler et al., 2017; Gray et al., 2019; Guirado et al., 2018 ; Lasseck, 2018; Tabak et al., 2019 ) or for extracting trait or behavioural information (Dunker et al., 2020; Graving et al., 2019; Mathis et al., 2018 ; Ott & Lautenschlager, 2021; Pereira et al., 2019 ).”
- Computational neuroethology: a call to action (2019, background): “…for tracking e.g., paws during reaching (see below) has revealed the behavioral richness and variability that underlies even simple behavioral reports like pellet grabs or lever presses (Graving et al., 2019; Guo et al., 2015; Mathis et al., 2018; Nath et al., 2019; Pereira et al., 2019).”
- Perspectives in machine learning for wildlife conservation (2021, background): “ing (29, 30, 79), semi-automatic detection of large mammals (48, 80), birds (81) and, in low altitude flight, even identification of individuals (82).”
- Deep learning for cellular image analysis (2019, methodology): “The object-tracking package DeepLabCut, originally based on deep learning approaches to object detection and pose estimation, tracked appendages of flies and mice to quantify behavior during neuroscience experiments 128 .”
- Deep learning tools for the measurement of animal behavior in neuroscience (2019, methodology): “Currently, two paths are possible: one is to apply pose estimation algorithms after tracking individuals (for which any package could be used); or, two, extract multiple detections for each part on each animal (Figure 2h; [48 ,72]) and link them using part affinity fields [19 ], pairwise predictions”