Visual Semantic Role Labeling
By Saurabh Gupta, Jitendra Malik. 2015
488 citations recorded by the source corpus.
In this paper we introduce the problem of Visual Semantic Role Labeling: given an image we want to detect people doing actions and localize the objects of interaction. Classical approaches to action recognition either study the task of action classification at the image or video clip level or at best produce a bounding box around the person doing the action. We believe such an output is inadequate and a complete understanding can only come when we are able to associate objects in the scene to the different semantic roles of the action. To enable progress towards this goal, we annotate a dataset of 16K people instances in 10K images with actions they are doing and associate objects in the scene with different semantic roles for each action. Finally, we provide a set of baseline algorithms for this task and analyze error modes providing directions for future work.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Extraction
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Cordelia Schmid
- Trevor Darrell
- Abhinav Kumar Gupta
- Jitendra Malik
- Quoc V. Le
- Yoshua Bengio
- Jianfeng Gao
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Jitendra Malik
- Trevor Darrell
- Dacheng Tao
- Ming-Hsuan Yang
- Cordelia Schmid
- Xiaogang Wang
- Andrew Zisserman
In Information Extraction
- Giovanni Maria Farinella
- Antonino Furnari
- Francesco Ragusa
- Spyridon Gidaris
- Nikos Komodakis
- Hiroki Ohashi
- Masato Tamura
- Chen Gao
Citations in the corpus
- Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, neutral): “ted and annotated to learn a generalisable model for that task. Increasingly, datasets have been used for novel tasks, through pre-training [10,11], self-supervision [12,13] or additional annotations [14,15]. However, task adaptation demonstrates that models overt to the data and annotations [16]. A”
- The Open Images Dataset V4 (2018, methodology): “, 2016), or using mean average precision (mAP) as for object detection (Gupta and Malik, 2015).”
- AVA: A Video Dataset of Spatio-Temporally Localized Atomic Visual Actions (2017, background): “The AVA dataset is also related to still image action recognition datasets [7, 9, 12].”
- Simitate: A Hybrid Imitation Learning Benchmark (2019, neutral): “presented a dataset [35] based on a subset of the COCO [12] dataset targeting semantic role labeling by verbs describing people interacting with objects.”
- Attend Refine Repeat: Active Box Proposal Generation via In-Out Localization (2016, methodology): “Object proposals have also been used in various other tasks, such as weakly-supervised object detection [7], exemplar 2D-3D detection [30], visual semantic role labelling [17], caption generation [24] or visual question answering [37].”
- Detecting and Recognizing Human-Object Interactions (2017, methodology): “0 AP (evaluated by Average Precision on a triplet, called ‘role AP’ [14]), with the gain mainly due to inferring the target object’s relative position from the human appearance.”
- From Lifestyle Vlogs to Everyday Interactions (2017, methodology): “We examined 15 VLOG categories that overlap cleanly with any of [35, 23, 43, 5, 15].”
- Understanding Human Hands in Contact at Internet Scale (2020, background): “Recent work has largely taken the approach of recognizing verb-noun pairs [17, 9, 10, 19, 6].”
- Asynchronous Temporal Fields for Action Recognition (2016, background): “Recently, there has been significant progress in modelling the complete human-centric aspect, where image recognition is phrased in terms of objects and their roles [62, 10].”
- Semantic Image Retrieval via Active Grounding of Visual Situations (2017, background): “in [18], requires a system to, given an image, predict the most salient verb, along with its subject and object (“semantic roles” [19]).”
- Situation Recognition: Visual Semantic Role Labeling for Image Understanding (2016, background): “Both provide instance-level information with bounding regions for objects [23, 40].”
- MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain (2022, methodology): “Different third person vision datasets have been proposed to study human behavior understanding exploiting the tasks of Human-Object Interaction (HOI) detection [43, 7, 58] and action recognition [22, 47, 5, 40].”
- Semantic Image Interpretation - Integration of Numerical Data and Logical Knowledge for Cognitive Vision (2018, unknown): “Closely related to the visual relationship detection is the visual semantic role labelling [42, 85, 108].”
- Learning Human-Object Interactions by Graph Parsing Neural Networks (2018, methodology): “We use HICO-DET [1] and V-COCO [17] datasets for benchmarking our GPNN model.”
- Learning to Detect Human-Object Interactions (2017, background): “Gupta and Malik [11] augmented MS-COCO [22] by connecting interacting people and objects and labeling their semantic roles.”
- HOTR: End-to-End Human-Object Interaction Detection with Transformers (2021, methodology): “Human-Object Interaction (HOI) detection has been formally defined in [8] as the task to predict a set of (cid:104) human, object, interaction (cid:105) triplets within an image.”
- PPDM: Parallel Point Detection and Matching for Real-Time Human-Object Interaction Detection (2019, methodology): “There are mainly two commonly used HOI detection benchmarks: VCOCO [11] and HICODet [2], and a human-centric relationship detection dataset: HCVRD [36].”
- HAKE: Human Activity Knowledge Engine (2019, methodology): “HAKE is based on the existing well-designed datasets, for example, HICO-DET [12], V-COCO [16], OpenImage [17], HCVRD [22], HICO [13], MPII [1], AVA [10], which are structured around a rich semantic ontology.”