LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities

By Baoxiong Jia, Yixin Chen, Siyuan Huang, Yixin Zhu, Song-Chun Zhu. 2020

77 citations recorded by the source corpus.

Understanding and interpreting human actions is a long-standing challenge and a critical indicator of perception in artificial intelligence. However, a few imperative components of daily human activities are largely missed in prior literature, including the goal-directed actions, concurrent multi-tasks, and collaborations among multi-agents. We introduce the LEMMA dataset to provide a single home to address these missing dimensions with meticulously designed settings, wherein the number of tasks and agents varies to highlight different learning objectives. We densely annotate the atomic-actions with human-object interactions to provide ground-truths of the compositionality, scheduling, and assignment of daily activities. We further devise challenging compositional action recognition and action/task anticipation benchmarks with baseline models to measure the capability of compositional action understanding and temporal reasoning. We hope this effort would drive the machine vision community to examine goal-directed human activities and further study the task scheduling and assignment in the real world.

Subfields: Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Aside from daily life activity, prior ego datasets focus on conversation [170], inter-person interactions [66,168,194,231], place localization [183,208], multimodal sensor data [124,166,204], human hands [16, 134] human-object interaction [106, 184], and object tracking [56].”
SQA3D: Situated Question Answering in 3D Scenes (2022, methodology): “Recent research has also introduced QA tasks on video (Lei et al., 2018; Jia et al., 2020; 2022; Grunde-McLaughlin et al., 2021; Wu et al., 2021; Datta et al., 2022), and 3D data (Ye et al., 2021; Azuma et al., 2022; Yan et al., 2021).”