Experience Grounds Language

By Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Yue Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich and 2 more. 2020

440 citations recorded by the source corpus.

Successful linguistic communication relies on a shared experience of the world, and it is this shared experience that makes utterances meaningful. Despite the incredible effectiveness of language processing models trained on text alone, today's best systems still make mistakes that arise from a failure to relate language to the physical world it describes and to the social interactions it facilitates. Natural Language Processing is a diverse field, and progress throughout its development has come from new representational theories, modeling techniques, data collection paradigms, and tasks. We posit that the present success of representation learning approaches trained on large text corpora can be deeply enriched from the parallel tradition of research on the contextual and social nature of language. In this article, we consider work on the contextual foundations of language: grounding, embodiment, and social interaction. We describe a brief history and possible progression of how contextual information can factor into our representations, with an eye towards how this integration can move the field forward and where it is currently being pioneered. We believe this framing will serve as a roadmap for truly contextual language understanding.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Cognitive Science & Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Cognitive Science & Psychology

Citations in the corpus

Measuring Massive Multitask Language Understanding (2020, background): “While text is capable of conveying an enormous number of concepts about the world, many important concepts are conveyed mainly through other modalities, such as images, audio, and physical interaction (Bisk et al., 2020).”
AI and the Everything in the Whole Wide World Benchmark (2021, background): “Several researchers have raised the need to establish effective physical and social grounding as part of the process of moving towards robust and effective natural language understanding, warning against text-only learning as a limited approach [Bisk et al., 2020, Zellers et al., 2021].”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, background): “The longstanding goal of grounding language learning in embodied settings (Bisk et al., 2020) has lead to substantial work on interactive environments.”
Recipes for Building an Open-Domain Chatbot (2020, background): “Further, these models, which are disem-bodied, also have no way of grounding to entities, actions and experience in the world, which could also stunt their abilities (Bisk et al., 2020).”
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “Note here that, in contrast to other work (Bender and Koller, 2020; Bisk et al., 2020), we do not focus on whether LLMs “understand” language, instead we examine whether LLMs can answer questions about the emotions and mental states of participants of situations.”
Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data (2020, background): “The idea that as the learner gets access to more and more information in addition to the text itself, it can learn more and more facets of meaning is worked out in detail by Bisk et al. (2020).”
HONEST: Measuring Hurtful Sentence Completion in Language Models (2021, background): “BERT and GPT-2 have shown astonishing capabilities and pushed the envelope of natural language understanding - not without some doubts (Bisk et al., 2020; Bender and Koller, 2020).”
What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021, background): “Many task-speciﬁc papers contribute to this debate, as does a prominent recent thread advocating for an increased focus on grounding of various kinds by Bender and Koller (2020), Bisk et al. (2020), Zellers et al. (2020), and others.”
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, background): “By operating only in simulation, our model also misses the full range of experience that can ground language in the world (Bisk et al., 2020), such as haptic feedback during object manipulation (Thomason et al., 2020, 2016; Sinapov et al., 2014), and audio (Chen et al., 2020a) and speech (Harwath…”
Probing Contextual Language Models for Common Ground with Visual Representations (2020, background): “A widely investigated research direction aims to connect natural language to the physical world (Bisk et al., 2020; Tan and Bansal, 2019; Lu et al., 2019a,b; Chen et al., 2019; Li et al., 2020; Tan and Bansal, 2020).”
Underspecification in Scene Description-to-Depiction Tasks (2022, background): “Indeed, joint modeling of linguistic and visual signals has been argued to play a critical role in progress towards this ultimate goal, as precursors to modeling relationships between language and the social and physical worlds (Bisk et al., 2020).”
Rearrangement: A Challenge for Embodied AI (2020, background): “The rearrangement task allows studying grounding of language to perception and action, a topic relevant to natural language researchers interested in Em-bodied AI [7].”
Language Grounding with 3D Objects (2021, background): “Such world interaction is inextricable from language grounding [21], motivating language annotations for higher-ﬁdelity referents than static images.”
Do Neural Language Models Overcome Reporting Bias? (2020, background): “There has been little work on learning meaning from other modalities (Kiela and Clark, 2015; Zellers et al., 2019), but there is a shared understanding in the community that this is the imperative next step (Bisk et al., 2020; Bender and Koller, 2020).”
ConceptFusion: Open-set Multimodal 3D Mapping (2023, background): “Recent approaches exploring foundation models for robotics have demonstrated impressive open-set interaction abilities [36, 37, 38, 39, 40].”
TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment (2021, background): “Aligning or grounding language to videos is a challenging topic in the context of vision-language (VL) research as it requires the model to understand contents, dynamics, and causality presented in videos [3].”
PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World (2021, background): “…describing the initial world state, the action, and the action result. cent work arguing that no amount of language form could ever specify language meaning (McClelland et al., 2019; Bender and Koller, 2020; Bisk et al., 2020); connecting back to the Symbol Grounding Problem of Harnad (1990).”