Fast animal pose estimation using deep neural networks

By Talmo D. Pereira, Diego E. Aldarondo, Lindsay Willmore, Mikhail Kislin, S. H. Wang, Mala Murthy, Joshua W. Shaevitz. Nature Methods, 2018

621 citations recorded by the source corpus.

The need for automated and efficient systems for tracking full animal pose has increased with the complexity of behavioral data and analyses. Here we introduce LEAP (LEAP estimates animal pose), a deep-learning-based method for predicting the positions of animal body parts. This framework consists of a graphical interface for labeling of body parts and training the network. LEAP offers fast prediction on new data, and training with as few as 100 frames results in 95% of peak performance. We validated LEAP using videos of freely behaving fruit flies and tracked 32 distinct points to describe the pose of the head, body, wings and legs, with an error rate of <3% of body length. We recapitulated reported findings on insect gait dynamics and demonstrated LEAP’s applicability for unsupervised behavioral classification. Finally, we extended the method to more challenging imaging situations and videos of freely moving mice. LEAP is a deep-learning-based approach for the analysis of animal pose. LEAP’s graphical user interface facilitates training of the deep network. The authors illustrate the method by analyzing Drosophila and mouse behavior.

Subfields: Ecology

View Paper: https://www.semanticscholar.org/paper/422e8ed1ee90b8e788bf1756afdeb3a98f50912a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Ecology

1. Florian Hartig
2. Maximilian Pichler

Citations in the corpus

• Deep learning for cellular image analysis (2019, methodology): “The recent software package LEAP 129 builds on this work by incorporating a graphical user interface to assist in data annotation 129 , while the idtracker.ai 130 package tracks multiple organisms by using deep-learning-based image classifiers to identify animals that cross paths.”
• Machine learning and deep learning—A review for ecologists (2022, background): “…(Aodha et al., 2018 ; e.g. Fritzler et al., 2017; Gray et al., 2019; Guirado et al., 2018 ; Lasseck, 2018; Tabak et al., 2019 ) or for extracting trait or behavioural information (Dunker et al., 2020; Graving et al., 2019; Mathis et al., 2018 ; Ott & Lautenschlager, 2021; Pereira et al., 2019 ).”
• Using DeepLabCut for 3D markerless pose estimation across species and behaviors (2018, methodology): “In contrast, LEAP may be faster, due to its 15 layer convolutional network, as the authors report 185 FPS with a batch size of 32 images for 192 x 192 sized images [21].”
• SLEAP: A deep learning system for multi-animal pose tracking (2022, methodology): “For this dataset, we used the existing manual labels for a 32-node skeleton split into 1,200 training, 150 validation and 150 test frames 6 .”
• Deep learning tools for the measurement of animal behavior in neuroscience (2019, methodology): “LEAP (LEAP estimates animal pose), a DNN method for predicting the positions of animal body parts.”