Describing Common Human Visual Actions in Images

By Matteo Ruggero Ronchi, Pietro Perona. 2015

69 citations recorded by the source corpus.

Which common human actions and interactions are recognizable in monocular still images? Which involve objects and/or other people? How many is a person performing at a time? We address these questions by exploring the actions and interactions that are detectable in the images of the MS COCO dataset. We make two main contributions. First, a list of 140 common ‘visual actions’, obtained by analyzing the largest online verb lexicon currently available for English (VerbNet) and human sentences used to describe images in MS COCO. Second, a complete set of annotations for those ‘visual actions’, composed of subject-object and associated verb, which we call COCO-a (a for ‘actions’). COCO-a is larger than existing action datasets in terms of number instances of actions, and is unique because it is data-driven, rather than experimenter-biased. Other unique features are that it is exhaustive, and that all subjects and objects are localized. A statistical analysis of the accuracy of our annotations and of each action, interaction and subject-object combination is provided.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Scene Graph Generation by Iterative Message Passing (2017, background): “The recent success of deep learning-based recognition models [15, 21, 36] has resurged interest in examining the detailed structures of a visual scene, especially in the form of object relationships [5, 20, 26, 33].”
Detecting and Recognizing Human-Object Interactions (2017, background): “The older TUHOI [19] and HICO [4] datasets only have imagelevel labels and thus do not allow for grounding interactions in a detection setting, while COCO-a [28] is promising but only a small beta-version is currently available.”
Semantic Image Interpretation - Integration of Numerical Data and Logical Knowledge for Cognitive Vision (2018, unknown): “Closely related to the visual relationship detection is the visual semantic role labelling [42, 85, 108].”
Situation Recognition: Visual Semantic Role Labeling for Image Understanding (2016, background): “Both provide instance-level information with bounding regions for objects [23, 40].”
Annotating Objects and Relations in User-Generated Videos (2019, background): “Compare with image datasets for similar purpose [13, 18], the constructed dataset will serve as a benchmark dataset that can be used to develop and evaluate techniques at video level directly.”
Learning Models for Actions and Person-Object Interactions with Transfer to Question Answering (2016, methodology): “Various datasets have been released for single image based action recognition, including the older PASCAL VOC [16] and Stanford 40 Actions [17], and newer MPII Human Pose Dataset [15], COCO-A [18] and Humans Interacting with Common Objects (HICO) dataset [14].”
Commonly Uncommon: Semantic Sparsity in Situation Recognition (2016, background): “Situation Recognition Situation recognition has been recently proposed to model events within images [19, 36, 43, 44], in order to answer questions beyond just “What activity is happening?” such as “Who is doing it?”, “What are they doing it to?”, “What are they doing it with?”.”
Attentional Pooling for Action Recognition (2017, background): “Human action recognition is a well studied problem with various standard benchmarks spanning across still images [7, 13, 34, 36, 58] and videos [24, 27, 41, 45].”