BABEL: Bodies, Action and Behavior with English Labels
By Abhinanda R. Punnakkal, Arjun Chandrasekaran, Nikos Athanasiou, Alejandra Quiros-Ramirez, Michael J. Black Max Planck Institute for Intelligent Systems, Universität Konstanz. Computer Vision and Pattern Recognition, 2021
348 citations recorded by the source corpus.
Understanding the semantics of human movement – the what, how and why of the movement – is an important problem that requires datasets of human actions with semantic labels. Existing datasets take one of two approaches. Large-scale video datasets contain many action labels but do not contain ground-truth 3D human motion. Alternatively, motion-capture (mocap) datasets have precise body motions but are limited to a small number of actions. To address this, we present BABEL, a large dataset with language labels describing the actions being performed in mocap sequences. BABEL consists of language labels for over 43 hours of mocap sequences from AMASS, containing over 250 unique actions. Each action label in BABEL is precisely aligned with the duration of the corresponding action in the mocap sequence. BABELalso allows overlap of multiple actions, that may each span different durations. This results in a total of over 66000 action segments. The dense annotations can be leveraged for tasks like action recognition, temporal localization, motion synthesis, etc. To demonstrate the value of BABEL as a benchmark, we evaluate the performance of models on 3D action recognition. We demonstrate that BABEL poses interesting learning challenges that are applicable to real-world scenarios, and can serve as a useful benchmark for progress in 3D action recognition. The dataset, baseline methods, and evaluation code are available and supported for academic research purposes at https://babel.is.tue.mpg.de/.
Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering
View Paper: https://www.semanticscholar.org/paper/fe9bc34b3e3b181de6caee3ff79e9c5bf1bbcf98
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
In Control and Systems Engineering
Citations in the corpus
- Generating Diverse and Natural 3D Human Motions from Text (2022, background): “6M [15], MoVi [9] and BABEL [33], in the form of everyday actions and sports movements.”
- Human Action Recognition From Various Data Modalities: A Review (2020, neutral): “[407] 2019 WiFi CSI 6 1 1,394 NTU RGB+D 120 [151] 2019 RGB,S,D,IR 120 106 114,480 155 IKEA ASM [408] 2020 RGB,S,D 33 48 16,764 3 EPIC-KITCHENS-100 [409] 2020 RGB,Au,Ac - 45 89,979 Egocentric ETRI-Activity3D [410] 2020 RGB,S,D 55 100 112,620 RareAct [411] 2020 RGB 122 - 905 BABEL [412] 2021 Mocap 252”
- Human Motion Diffusion as a Generative Prior (2023, methodology): “Transitions were either learned according to a smoothness prior [Athanasiou et al. 2022; Mao et al. 2022] or from data [Athanasiou et al. 2022; Wang et al. 2022], using the BABEL dataset [Punnakkal et al. 2021], which explicitly annotates transitions between actions.”
- MotionCLIP: Exposing Human Motion Generation to CLIP Space (2022, methodology): “Morerecently, BABEL dataset [Punnakkal et al. 2021] provided per-frame textual labels ordered in 260 classes to the larger AMASS dataset [Mahmood et al. 2019], including about 40 hours of motion capture.”
- MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model (2022, methodology): “Implementation Details: We train our proposed MotionDif-fuse on the BABEL dataset [72] with 50 K iterations.”