The KIT Motion-Language Dataset

By Matthias Plappert, Christian Mandery, Tamim Asfour. Big Data, 2016

494 citations recorded by the source corpus.

Abstract Linking human motion and natural language is of great interest for the generation of semantic representations of human activities as well as for the generation of robot activities based on natural language input. However, although there have been years of research in this area, no standardized and openly available data set exists to support the development and evaluation of such systems. We, therefore, propose the Karlsruhe Institute of Technology (KIT) Motion-Language Dataset, which is large, open, and extensible. We aggregate data from multiple motion capture databases and include them in our data set using a unified representation that is independent of the capture system or marker set, making it easy to work with the data regardless of its origin. To obtain motion annotations in natural language, we apply a crowd-sourcing approach and a web-based tool that was specifically build for this purpose, the Motion Annotation Tool. We thoroughly document the annotation process itself and discuss gamification methods that we used to keep annotators motivated. We further propose a novel method, perplexity-based selection, which systematically selects motions for further annotation that are either under-represented in our data set or that have erroneous annotations. We show that our method mitigates the two aforementioned problems and ensures a systematic annotation process. We provide an in-depth analysis of the structure and contents of our resulting data set, which, as of October 10, 2016, contains 3911 motions with a total duration of 11.23 hours and 6278 annotations in natural language that contain 52,903 words. We believe this makes our data set an excellent choice that enables more transparent and comparable research in this important area.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Social Psychology

View Paper: https://www.semanticscholar.org/paper/e738d166c72513feaede2e19b4e713fd410cfce8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Social Psychology

Citations in the corpus

Generating Diverse and Natural 3D Human Motions from Text (2022, methodology): “Empirical evaluations are carried on both the in-house HumanML3D and KIT-ML [31] datasets.”
Executing your Commands via Motion Diffusion in Latent Space (2022, methodology): “We evaluate state-of-the-art methods on HumanML3D and KIT with suggested metrics [17] under the 95% conﬁdence interval from 20 times running.”
Human Motion Diffusion Model (2022, methodology): “We evaluate our model using two leading benchmarks - KIT (Plappert et al., 2016) and HumanML3D (Guo et al., 2022a), over the set of metrics suggested by Guo et al. (2022a): R-precision and Multimodal-Dist measure the relevancy of the generated motions to the input prompts, FID measures the dissimila”
TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts (2022, background): “– KIT Motion-Language [36] contains 3,911 3D human motion clips and 6,278 text descriptions.”
MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model (2022, methodology): “Recently, TEMOS [12] utilizes KIT Motion-Language MoCap dataset [13] to demonstrate accurate human motion synthesis.”
MoMask: Generative Masked Modeling of 3D Human Motions (2023, methodology): “Empirical evaluations are conducted on two widely used motion-language benchmarks, HumanML3D [15] and KIT-ML [37].”
MotionCLIP: Exposing Human Motion Generation to CLIP Space (2022, background): “The KIT dataset[Plappert et al. 2016] provides about 11 hours of motion capture sequences, each sequence paired with a sentence explicitly describing the action performed.”
Language2Pose: Natural Language Grounded Pose Forecasting (2019, methodology): “Our models are trained an evaluated on KIT Motion-Language Dataset [25] which combines human motion with natural language descriptions.”
MotionGPT: Human Motion as a Foreign Language (2023, unknown): “2) We then learn the motion-language relation by the supervision of paired text-motion datasets [10, 31].”
TEMOS: Generating diverse human motions from textual descriptions (2022, methodology): “Table 1: State-of-the-art comparison: We compare our method with recent works [2, 13, 29], on the KIT Motion-Language dataset [40] and obtain significant improvements on most metrics (values in meters) even if we are sampling a random motion per text conditioning for our model.”
Generating Human Motion from Textual Descriptions with Discrete Representations (2023, methodology): “We conduct experiments on two standard datasets for text-driven motion generations: HumanML3D [22] and KIT Motion-Language (KIT-ML) [54].”
ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model (2023, methodology): “TEMOS uses a Variational-Auto-Encoder (VAE) to synthesize detailed motions, utilizing the KIT Motion-Language dataset [17].”