Situation Recognition: Visual Semantic Role Labeling for Image Understanding

By Mark Yatskar, Luke S. Zettlemoyer, Ali Farhadi. 2016

286 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Experience Grounds Language (2020, methodology): “We posit that models must be able to watch and recognize objects, people, and activities to understand the language describing them (Li et al., 2019b; Krishna et al., 2017; Yatskar et al., 2016; Perlis, 2016).”
PIQA: Reasoning about Physical Commonsense in Natural Language (2019, background): “Research has studied predicting visual relationships in images (Krishna et al. 2016) and as well as actions and their dependent objects (Yatskar, Zettlemoyer, and Farhadi 2016).”
Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints (2017, methodology): “Dataset We evaluate on imSitu (Yatskar et al., 2016) where activity classes are drawn from verbs and roles in FrameNet (Baker et al.”
Mitigating Gender Bias in Natural Language Processing: Literature Review (2019, methodology): “For example, when RBA was applied to the visual semantic role labelling (Yatskar et al., 2016), it restricted the ratio of males to females predicted to be doing particular activities to prevent the model from amplifying bias through predictions.”
The “Something Something” Video Database for Learning and Evaluating Visual Common Sense (2017, background): “Unlike labels that are encoded using a fixed datatype, as described, for example, in [39], natural language labels allow us to represent a spectrum of complexity, from simple objects and actions encoded as one-hot labels, to full-fledged captions.”
Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview (2019, background): “The predicted distribution is therefore also dissimilar to the target ideal distribution : For example, in the imSitu image captioning data set (Yatskar et al., 2016), 58% of captions involving a person in a kitchen were found to mention women, but when standard models were trained on such data,…”
Neural Motifs: Scene Graph Parsing with Global Context (2017, background): “Actions and relations have been a particularly fruitful source of context [27, 45], especially when combined with pose to create human-object interactions [43, 3].”
Modeling Naive Psychology of Characters in Simple Commonsense Stories (2018, methodology): “…combination of automatic methods (SGD over interan-notator agreement using the quality control objectives described in the supplementary material of Yatskar et al., 2016) and manual methods (e.g. searching for workers who always selected the same answers, who had strange answering patterns, who…”
Do Neural Language Representations Learn Physical Commonsense? (2019, positive): “We allow annotators to choose from the 504 verbs from the imSitu dataset (Yatskar, Zettlemoyer, & Farhadi, 2016).”
A Domain Based Approach to Social Relation Recognition (2017, methodology): “Then, for each attribute category, we either leverage existing datasets to train attribute classifiers using our double-stream CaffeNet (default) or use pre-trained models published in previous works [10, 47].”
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks (2022, methodology): “We additionally convert data from several other datasets in a VQA format, including imSitu (Yatskar et al., 2016), where we treat predicting the verb and then the related slots as separate VQA questions, VisualCOMMET (Park et al.”
Mitigating Gender Bias Amplification in Distribution by Posterior Regularization (2020, methodology): “We follow the settings in Zhao et al. (2017) to focus on the imSitu vSRL dataset (Yatskar et al., 2016), in which we are supposed to predict the activities and roles in given images and this can be regraded as a structure prediction task (see Fig.”
What's "up" with vision-language models? Investigating their struggle with spatial reasoning (2023, background): “However, recent work has re-surfaced a concern that has long plagued vision-language models (Yatskar et al., 2016; Johnson et al., 2017): new multimodal models still exhibit poor behavior on simple tasks like attribute attachment, counting, etc. (Yamada et al., 2022; Thrush et al., 2022;…”
Learning to generalize to new compositions in image understanding (2016, unknown): “The dataset of (Yatskar et al., 2016) has combinations of entities”
Webly Supervised Concept Expansion for General Purpose Vision Models (2022, background): “[86] Mark Yatskar, Luke Zettlemoyer, and Ali Farhadi.”
DOCK: Detecting Objects by Transferring Common-Sense Knowledge (2018, background): “Our field has witnessed the rise of several interesting knowledge bases, including ConceptNet [23], BabelNet [29], WordNet [27], WebChild [38], Visual Genome [19], ImSitu [44], etc.”
Data Feedback Loops: Model-driven Amplification of Dataset Biases (2022, methodology): “We train the default ResNet18-backed conditional random fields model (Yatskar et al., 2016), proposed as a baseline alongside the dataset.”