A database for fine grained activity detection of cooking activities

By Marcus Rohrbach, Sikandar Amin, Mykhaylo Andriluka, Bernt Schiele. 2012

634 citations recorded by the source corpus.

While activity recognition is a current focus of research the challenging problem of fine-grained activity recognition is largely overlooked. We thus propose a novel database of 65 cooking activities, continuously recorded in a realistic setting. Activities are distinguished by fine-grained body motions that have low inter-class variability and high intra-class variability due to diverse subjects and ingredients. We benchmark two approaches on our dataset, one based on articulated pose tracks and the second using holistic video features. While the holistic approach outperforms the pose-based approach, our evaluation suggests that fine-grained activities are more difficult to detect and the body model can help in those cases. Providing high-resolution videos as well as an intermediate pose representation we hope to foster research in fine-grained activity recognition.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval (2021, neutral): “MPII Cook [47] cooking 44 600 6K 8 TACos [44] cooking 7K 360 18K 15.”
Towards Automatic Learning of Procedures From Web Instructional Videos (2017, methodology): “2013), MPII (Rohrbach et al. 2012), 50Salads (Stein and McKenna 2013), Coffee (Alayrac et al.”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “Table 4 also compares EPIC-KITCHENS-100 with other datasets used for action anticipation [30,54,55,56,61,62,40].”
CNN Features Off-the-Shelf: An Astounding Baseline for Recognition (2014, background): “The advent of many new datasets with ﬁne-grained annotations such as Oxford ﬂowers [25], Caltech bird species [40], dog breeds [1], cooking activities [35], cats and dogs [30] has helped the ﬁeld develop quickly.”
Dense-Captioning Events in Videos (2017, neutral): “Similar works have used hand-crafted features [40] or object-centric features [30] to recognize actions in ﬁxed camera settings.”
Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding (2016, background): “There are threefold advantages of using the Hollywood in Homes approach for dataset collection: (a) Unlike datasets shot in controlled environments ( e.g ., MPII [14]), crowdsourcing brings in diversity which is essential for generalization.”
Grounding Action Descriptions in Videos (2013, methodology): “For the experiments reported below we use the same setup as Rohrbach et al. (2012b) and use all videos in MPII Composites and MPII Cooking (Rohrbach et al., 2012a), excluding the 127 videos used during evaluation.”
Learning human activities and object affordances from RGB-D videos (2012, methodology): “The outputs from the skeleton and object tracking along with the segmentation information and RGBD videos are then used to generate the features described in Section VII.”
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset (2018, methodology): “6M 50 2967 52 0 0 25 1 MPII Cooking 2 [26] × × × 2012 2.”
Coherent Multi-sentence Video Description with Variable Level of Detail (2014, methodology): “Our hand detector alone significantly improves over the state-of-the-art FPS approach of [18].”
Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos (2015, methodology): “The MPII Cooking dataset [21] and Breakfast dataset [11] refocus efforts toward human action within restricted action domains (Table 1).”
The “Something Something” Video Database for Learning and Evaluating Visual Common Sense (2017, neutral): “Many shortcomings of existing video datasets may be re-Table [26] action (cooking) 44 600s - TACoS [23] action (cooking) 127 360s - Charades [29] action (human) 10, 000 30s - KITTI [6] action (driving) 21 30s - Something-Something (ours) human-object interaction 108,499 4.03s 174 ﬁne-grained…”
End-to-End Learning of Action Detection from Frame Glimpses in Videos (2015, background): “The agent receives a sequence of video frames V = {v1, ..., vT } as input, and can observe a fixed proportion of the frames.”
COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis (2019, background): “MPII [35] 9h,48m 44 5,609 cooking activities self-collected - 2012 YouCook [14] 2h,20m 88 cooking activities YouTube - 2013 50Salads [40] 5h,20m 50 966 cooking activities self-collected - 2013 Breakfast [28] 77h 1,989 8,456 cooking activities self-collected 10 2014 "5 tasks" [10] 5h 150 comprehensiv”
TALL: Temporal Activity Localization via Language Query (2017, background): “[30] extended two-stream [28] framework with person detection and bi-directional LSTMs and achieved state-of-the-art performance on MPIICooking dataset [24].”
Temporal Generative Adversarial Nets with Singular Value Clipping (2016, methodology): “In supervised learning of videos, while a common approach is to use dense trajectories [45, 30, 29], recent methods have employed CNN and achieved state-of-the-art results [14, 16, 33, 43, 24, 46, 47].”
Connectionist Temporal Modeling for Weakly Supervised Action Labeling (2016, background): “The second is dense action labeling, where the goal is to label each frame with the occurring actions [19,24,26,32,48], and the fully annotated temporal boundaries of actions are given during training.”