Grammar-Based Grounded Lexicon Learning

By Jiayuan Mao, Haoyue Shi, Jiajun Wu, Roger Levy, Josh Tenenbaum. 2022

18 citations recorded by the source corpus.

We present Grammar-Based Grounded Lexicon Learning (G2L2), a lexicalist approach toward learning a compositional and grounded meaning representation of language from grounded data, such as paired images and texts. At the core of G2L2 is a collection of lexicon entries, which map each word to a tuple of a syntactic type and a neuro-symbolic semantic program. For example, the word shiny has a syntactic type of adjective; its neuro-symbolic semantic program has the symbolic form {\lambda}x. filter(x, SHINY), where the concept SHINY is associated with a neural network embedding, which will be used to classify shiny objects. Given an input sentence, G2L2 first looks up the lexicon entries associated with each token. It then derives the meaning of the sentence as an executable neuro-symbolic program by composing lexical meanings based on syntax. The recovered meaning programs can be executed on grounded inputs. To facilitate learning in an exponentially-growing compositional space, we introduce a joint parsing and expected execution algorithm, which does local marginalization over derivations to reduce the training time. We evaluate G2L2 on two domains: visual reasoning and language-driven navigation. Results show that G2L2 can generalize from small amounts of data to novel compositions of words.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Denny Zhou
2. Quoc V. Le
3. Jason Wei
4. Xuezhi Wang
5. Ed H. Chi
6. Luke S. Zettlemoyer
7. Wenhu Chen
8. Graham Neubig

In Computer Vision and Pattern Recognition

1. Denny Zhou
2. Jason Wei
3. Quoc V. Le
4. Ed H. Chi
5. Xuezhi Wang
6. Dale Schuurmans
7. Fei Xia
8. Dacheng Tao

In Natural Language Processing

1. Denny Zhou
2. Jason Wei
3. Dale Schuurmans
4. Ed H. Chi
5. Xuezhi Wang
6. Maarten Bosma
7. Fei Xia
8. Quoc V. Le

Citations in the corpus

• Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, background): “First, in neuro-symbolic computing it has been recognized that generating intermediate results significantly improves accuracy when a desired input-output mapping involves multiple computational steps (Chen et al., 2019; Dong et al., 2019; Mao et al., 2021).”