Learning Visually Grounded Sentence Representations

By Douwe Kiela, Alexis Conneau, Allan Jabri, Maximilian Nickel. 2017

71 citations recorded by the source corpus.

We investigate grounded sentence representations, where we train a sentence encoder to predict the image features of a given caption—i.e., we try to “imagine” how a sentence would be depicted visually—and use the resultant features as sentence representations. We examine the quality of the learned representations on a variety of standard sentence representation quality benchmarks, showing improved performance for grounded models over non-grounded ones. In addition, we thoroughly analyze the extent to which grounding contributes to improved performance, and show that the system also learns improved word embeddings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Language and Linguistics, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Language and Linguistics

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, neutral): “uded supervised training in their comparison— namely, on machine translation data (using the WMT’14 English/French and English/German pairs), dictionary deﬁnitions and image captioning data (see also Kiela et al. (2017)) from the COCO dataset (Lin et al., 2014). These models obtained signiﬁcantly lo”
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “These include image-caption matching with MSCOCO (Lin et al., 2014), following Kiela et al. (2018); the small-to-medium-data text-understanding tasks collected in NLI format by Poliak et al. (2018); ordinal common sense inference (Zhang et al., 2017); POS tagging on the Penn Treebank (Marcus et…”
Fusion of Detected Objects in Text for Visual Question Answering (2019, background): “Modeling visual contexts can aid in learning useful sentence representations (Kiela et al., 2017) and even in training language models (Ororbia et al.”
Probing What Different NLP Tasks Teach Machines about Function Word Comprehension (2019, methodology): “We use image-caption pairs from the MSCOCO dataset (Lin et al., 2014) with an ob-jective that minimizes the cosine distance between sentence representations and corresponding image features, as described in Kiela et al. (2018).”
Looking for ELMo's friends: Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “These include image-caption matching with MSCOCO (Lin et al., 2014), following Kiela et al. (2018); the small-to-medium-data textunderstanding tasks collected in NLI format by Poliak et al. (2018); ordinal common sense inference (Zhang et al., 2017); POS tagging on the Penn Treebank (Marcus et al.,…”
Visually Grounded Compound PCFGs (2020, background): “Promising results from grounded learning have been emerging in areas such as representation learning (Bruni et al., 2014; Kiela et al., 2018; Bordes et al., 2019).”
No Training Required: Exploring Random Encoders for Sentence Classification (2019, background): “…Jernite et al., 2017; Pagliardini et al., 2017), a collection of books (Kiros et al., 2015), labelled entailment corpora (Conneau et al., 2017), image-caption data (Kiela et al., 2017), raw text labelled with discourse relations (Nie et al., 2017), or parallel corpora (Wieting & Gimpel, 2017).”
Emergent Translation in Multi-Agent Communication (2017, background): “ Andrew et al., 2013; Chandar et al., 2016). Other works minimize pairwise ranking loss to learn multimodal embeddings (Socher et al., 2014; Kiros et al., 2014; Ma et al., 2015; Vendrov et al., 2015; Kiela et al., 2017). Most recently, others extended this work to learn joint representations between”
Transformer is All You Need: Multimodal Multitask Learning with a Unified Transformer (2021, unknown): “[24, 25, 41, 7, 38]).”
Illustrative Language Understanding: Large-Scale Visual Grounding with Image Search (2018, background): “…representations with image-based representations (Bruni et al., 2014; Lazari-dou et al., 2015; Chrupala et al., 2015; Mao et al., 2016; Silberer et al., 2017; Kiela et al., 2017; Collell et al., 2017; Zablocki et al., 2018) and representations derived from a knowledge-graph (Thoma et al., 2017).”
Vokenization: Improving Language Understanding via Contextualized, Visually-Grounded Supervision (2020, methodology): “This tendency inspires the design of our vokenizer in merging previous sentence-level (Frome et al., 2013) and token-level (Kiela et al., 2018) approaches into a contextual token-level approach.”
Talk the Walk: Navigating New York City through Grounded Dialogue (2018, background): “Recently, grounding has also been applied to the learning of sentence representations (Kiela et al., 2017), image captioning (Lin et al., 2014; Xu et al., 2015), visual question answering (Antol et al., 2015; de Vries et al., 2017), visual reasoning (Johnson et al., 2017; Perez et al., 2018), and…”
IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages (2022, background): “A natural overlap of the two areas is language grounding in perception (typically vision; Deng et al., 2009; Kiela et al., 2018), where the perceptual input can be considered as an inherent cross-lingual signal (Kiela et al., 2015; Gella et al., 2017; Caglayan et al., 2021).”
UniT: Multimodal Multitask Learning with a Unified Transformer (2021, unknown): “[25, 26, 42, 7, 39]).”
Language learning using Speech to Image retrieval (2019, methodology): “We trained deep neural networks (DNNs) to create sentence embeddings without the use of prior knowledge of lexical semantics (see [7, 9, 10] for other studies on this task).”