Commonly Uncommon: Semantic Sparsity in Situation Recognition

By Mark Yatskar, Vicente Ordonez, Luke S. Zettlemoyer, Ali Farhadi. 2016

46 citations recorded by the source corpus.

Semantic sparsity is a common challenge in structured visual classification problems, when the output space is complex, the vast majority of the possible predictions are rarely, if ever, seen in the training set. This paper studies semantic sparsity in situation recognition, the task of producing structured summaries of what is happening in images, including activities, objects and the roles objects play within the activity. For this problem, we find empirically that most substructures required for prediction are rare, and current state-of-the-art model performance dramatically decreases if even one such rare substructure exists in the target output. We avoid many such errors by (1) introducing a novel tensor composition function that learns to share examples across substructures more effectively and (2) semantically augmenting our training data with automatically gathered examples of rarely observed outputs using web data. When integrated within a complete CRF-based structured prediction model, the tensor-based approach outperforms existing state of the art by a relative improvement of 2.11% and 4.40% on top-5 verb and noun-role accuracy, respectively. Adding 5 million images with our semantic augmentation techniques gives further relative improvements of 6.23% and 9.57% on top-5 verb and noun-role accuracy.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Extraction

Citations in the corpus

Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints (2017, methodology): “In vSRL, we use the imSitu formalism (Yatskar et al., 2016, 2017), where the goal is to predict activities, objects and the roles those objects play within an activity.”
Cross-Task Weakly Supervised Learning From Instructional Videos (2019, methodology): “It is also related to methods that compose new classifiers from others, including [25, 34, 15] among many others.”
General Multi-label Image Classification with Transformers (2020, background): “(3) Structured Output: The third kind describes label dependencies using structured output inference formulation [29, 47, 2, 20, 32, 33, 56, 38].”
Text2Scene: Generating Compositional Scenes From Textual Descriptions (2018, background): “Since our model learns about objects and relations separately, we also observed that it is often able to generalize to uncommon situations (as defined in [35]).”