Grounding language acquisition by training semantic parsers using captioned videos
By Candace Ross, Andrei Barbu, Yevgeni Berzak, Battushig Myanganbayar, Boris Katz. 2018
17 citations recorded by the source corpus.
We develop a semantic parser that is trained in a grounded setting using pairs of videos captioned with sentences. This setting is both data-efficient, requiring little annotation, and similar to the experience of children where they observe their environment and listen to speakers. The semantic parser recovers the meaning of English sentences despite not having access to any annotated sentences. It does so despite the ambiguity inherent in vision where a sentence may refer to any combination of objects, object properties, relations or actions taken by any agent in a video. For this task, we collected a new dataset for grounded language acquisition. Learning a grounded semantic parser — turning sentences into logical forms using captioned videos — can significantly expand the range of data that parsers can be trained on, lower the effort of training a semantic parser, and ultimately lead to a better understanding of child language acquisition.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Andrew Zisserman
- Abhinav Kumar Gupta
- Mohit Bansal
- Josh Tenenbaum
- Yoshua Bengio
- Christopher D. Manning
- Li Fei-Fei
- Cordelia Schmid
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Andrew Zisserman
- Jitendra Malik
- Cordelia Schmid
- Trevor Darrell
- Dacheng Tao
- Xiaogang Wang
- Andrea Vedaldi
In Natural Language Processing
- Mateusz Malinowski
- Andrew Zisserman
- Josh Tenenbaum
- Roger Levy
- Aida Nematzadeh
- Phil Blunsom
- Jiayuan Mao
- Jiajun Wu
In Multimodal Learning
- Josh Tenenbaum
- Andrew Zisserman
- Jiayuan Mao
- Jean-Baptiste Alayrac
- Lucas Smaira
- Jiajun Wu
- Roger Levy
- Mateusz Malinowski
Citations in the corpus
- Grammar-Based Grounded Lexicon Learning (2022, background): “There have also been approaches for learning grammatical structures from grounded texts [35, 50, 22, 7, 33, 43].”
- Visual Grounding in Video for Unsupervised Word Translation (2020, background): “Early work explored weak supervision and the correspondence problem between text annotations and image regions [7, 14], with more modern approaches exploring joint image-text word embeddings [17], or building a language conditioned attention map over the images in caption generation, visual question”
- A Survey on Semantic Parsing (2018, unknown): “Visually grounded semantic parsing : Recent work by Ross et al. [2018] involves learning a semantic parser using captioned videos.”