Learning to generalize to new compositions in image understanding

By Yuval Atzmon, Jonathan Berant, Vahid Kezami, Amir Globerson, Gal Chechik. 2016

71 citations recorded by the source corpus.

Recurrent neural networks have recently been used for learning to describe images using natural language. However, it has been observed that these models generalize poorly to scenes that were not observed during training, possibly depending too strongly on the statistics of the text in the training data. Here we propose to describe images using short structured representations, aiming to capture the crux of a description. These structured representations allow us to tease-out and evaluate separately two types of generalization: standard generalization to new images with similar scenes, and generalization to new combinations of known entities. We compare two learning approaches on the MS-COCO dataset: a state-of-the-art recurrent network based on an LSTM (Show, Attend and Tell), and a simple structured prediction model on top of a deep network. We find that the structured model generalizes to new compositions substantially better than the LSTM, ~7 times the accuracy of predicting structured representations. By providing a concrete method to quantify generalization for unseen combinations, we argue that structured representations and compositional splits are a useful benchmark for image captioning, and advocate compositional models that capture linguistic and visual structure.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, background): “Similarly, in vision, neural approaches often learn overly specialized features that do not easily transfer to different datasets or held-out combinations of attributes (Yosinski et al., 2014; Atzmon et al., 2016; Santoro et al., 2018b).”
A Comprehensive Survey of Scene Graphs: Generation and Application (2021, background): “pensate for the difference between model complexity and dataset complexity, its effectiveness will also be affected when the semantic word embedding falls short [118].”
Visual Translation Embedding Network for Visual Relation Detection (2017, background): “Note that there are works [3, 27, 33, 24] that exploit language priors to boost relation detection, but we are only interested in visual models.”
Benchmark for Compositional Text-to-Image Synthesis (2021, background): “Compositionality has also been explored in visual relationship detection [34, 1] and action recognition [27].”
Learning to Reason with Third-Order Tensor Products (2018, background): “They are catastrophically affected by systematic [10, 11] differences between training and test data [12, 13, 14, 15].”
Commonly Uncommon: Semantic Sparsity in Situation Recognition (2016, background): “Recent efforts to gain insight on such issues extract subject-verb-object (SVO) triplets from captions and count prediction failures on rare tuples [3].”
Pixels to Graphs by Associative Embedding (2017, background): “Visual relationship detection has drawn much recent attention [18, 28, 27, 2, 17, 19, 22, 23].”
Scene Graph Generation from Objects, Phrases and Region Captions (2017, background): “Recently, researchers pay more attention to general visual relationship detection [25, 36, 28, 39, 1, 44, 45] .”