Grounded PCFG Induction with Images

By Lifeng Jin, William Schuler. 2020

17 citations recorded by the source corpus.

Recent work in unsupervised parsing has tried to incorporate visual information into learning, but results suggest that these models need linguistic bias to compete against models that only rely on text. This work proposes grammar induction models which use visual information from images for labeled parsing, and achieve state-of-the-art results on grounded grammar induction on several languages. Results indicate that visual information is especially helpful in languages where high frequency words are more broadly distributed. Comparison between models with and without visual information shows that the grounded models are able to use visual information for proposing noun phrases, gathering useful information from images for unknown words, and achieving better performance at prepositional phrase attachment prediction.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Grammar-Based Grounded Lexicon Learning (2022, background): “There have also been approaches for learning grammatical structures from grounded texts [33, 48, 20, 7, 31, 41].”
Sequence-to-Sequence Learning with Latent Neural Grammars (2021, unknown): “…[101]) with contemporary parameterizations, (2) conditioning on other modalities such as images/audio for grounded grammar induction [100, 130, 60, 129], (3) adapting these methods to other structured domains such as programs and graphs, and (4) investigating how grammars and symbolic models…”