Like a Baby: Visually Situated Neural Language Acquisition

By Alexander Ororbia, Ankur Arjun Mali, Matthew Alex Kelly, David Reitter. 2018

19 citations recorded by the source corpus.

We examine the benefits of visual context in training neural language models to perform next-word prediction. A multi-modal neural architecture is introduced that outperform its equivalent trained on language alone with a 2% decrease in perplexity, even when no visual context is available at test. Fine-tuning the embeddings of a pre-trained state-of-the-art bidirectional language model (BERT) in the language modeling framework yields a 3.5% improvement. The advantage for training with visual context when testing without is robust across different languages (English, German and Spanish) and different models (GRU, LSTM, Delta-RNN, as well as those that use BERT embeddings). Thus, language models perform better when they learn like a baby, i.e, in a multi-modal environment. This finding is compatible with the theory of situated cognition: language is inseparable from its physical context.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning, Cognitive Science & Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chris Alberti
2. Michael Collins
3. David Reitter
4. Tatiana Shavrina
5. Oleg Serikov
6. Jeffrey Ling
7. E. Voloshina
8. Luke S. Zettlemoyer

In Computer Vision and Pattern Recognition

1. Chris Alberti
2. Michael Collins
3. David Reitter
4. Oleg Serikov
5. Jeffrey Ling
6. Tatiana Shavrina
7. E. Voloshina
8. Zhe Gan

In Natural Language Processing

1. Chris Alberti
2. Michael Collins
3. David Reitter
4. Oleg Serikov
5. Jeffrey Ling
6. Tatiana Shavrina
7. E. Voloshina
8. Luke S. Zettlemoyer

In Multimodal Learning

1. Michael Collins
2. David Reitter
3. Zhe Gan
4. Jingjing Liu
5. Linjie Li
6. Yu Cheng
7. Yen-Chun Chen
8. Amanpreet Singh

In Cognitive Science & Psychology

1. Michael Collins

Citations in the corpus

• Experience Grounds Language (2020, neutral): “, 2019) and language modeling (Ororbia et al., 2019).”
• Fusion of Detected Objects in Text for Visual Question Answering (2019, methodology): “The ensemble of B2T2 models, pretrained on Conceptual Captions, obtained absolute accuracy improvements of 8 .”
• Is neural language acquisition similar to natural? A chronological probing study (2022, background): “In (Ororbia et al., 2019), authors examine whether neural language models acquire language better when trained in a multi-modal setting (namely, accompanied with visual contexts) compared to traditional purely textual pre-training.”