PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World
By Rowan Zellers, Ari Holtzman, Matthew E. Peters, Roozbeh Mottaghi, Aniruddha Kembhavi, Ali Farhadi, Yejin Choi. 2021
83 citations recorded by the source corpus.
We propose PIGLeT: a model that learns physical commonsense knowledge through interaction, and then uses this knowledge to ground language. We factorize PIGLeT into a physical dynamics model, and a separate language model. Our dynamics model learns not just what objects are but also what they do: glass cups break when thrown, plastic ones don’t. We then use it as the interface to our language model, giving us a unified model of linguistic form and grounded meaning. PIGLeT can read a sentence, simulate neurally what might happen next, and then communicate that result through a literal symbolic representation, or natural language. Experimental results show that our model effectively learns world dynamics, along with how to communicate them. It is able to correctly forecast what happens next given an English sentence over 80% of the time, outperforming a 100x larger, text-to-text approach by over 10%. Likewise, its natural language summaries of physical interactions are also judged by humans as more accurate than LM alternatives. We present comprehensive analysis showing room for future work.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Yoshua Bengio
- Quoc V. Le
- Li Fei-Fei
- Mohit Bansal
- Trevor Darrell
In Computer Vision and Pattern Recognition
- Sergey Levine
- Luc Van Gool
- Trevor Darrell
- Li Fei-Fei
- Dacheng Tao
- Jitendra Malik
- Andrew Zisserman
- Xiaogang Wang
In Multimodal Learning
- Elias Stengel-Eskin
- Richard Johansson
- Tobias Norlund
- Alexander Toshev
- Sergey Levine
- Igor Mordatch
- Lovisa Hagström
- Kuang-Huei Lee
Citations in the corpus
- Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, methodology): “One approach to grounding language models in interaction is by learning downstream networks with pre-trained LLM representations [38, 22, 21, 39, 40, 41, 42, 43].”
- PaLM-E: An Embodied Multimodal Language Model (2023, background): “…incorporating affordance functions (Ahn et al., 2022), visual feedback (Huang et al., 2022c), generating world models (Nottingham et al., 2023; Zellers et al., 2021a), planning over graphs and maps (Shah et al., 2022; Huang et al., 2022a), visual explanations (Wang et al., 2023), program…”
- SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities (2024, unknown): “Though language model with interactive world experience show grounding improvements [64, 67], the introduction of large vision models, such as Flamingo [2], PaLI [13], or PaLM-E [18], has enabled a leap in performance.”
- Transferring Knowledge from Vision to Language: How to Achieve it and how to Measure it? (2021, background): “Recently, Zellers et al. (2021) also proposed a similar explicit decoupling but for augmenting a language model with knowledge about physical dynamics.”
- Visual Commonsense in Pretrained Unimodal and Multimodal Models (2022, background): “Zellers et al. (2021)107 learn physical commonsense through interaction,108 and uses this knowledge to ground language.”
- Behavior Cloned Transformers are Neurosymbolic Reasoners (2022, background): “, 2019), language grounding (Zellers et al., 2021), and semantic analysis (Cambria et al.”