Visually Grounded Neural Syntax Acquisition

By Freda Shi, Jiayuan Mao, Kevin Gimpel, Karen Livescu. 2019

87 citations recorded by the source corpus.

We present the Visually Grounded Neural Syntax Learner (VG-NSL), an approach for learning syntactic representations and structures without any explicit supervision. The model learns by looking at natural images and reading paired captions. VG-NSL generates constituency parse trees of texts, recursively composes representations for constituents, and matches them with images. We define concreteness of constituents by their matching scores with images, and use it to guide the parsing of text. Experiments on the MSCOCO data set show that VG-NSL outperforms various unsupervised parsing approaches that do not use visual grounding, in terms of F1 scores against gold parse trees. We find that VGNSL is much more stable with respect to the choice of random initialization and the amount of training data. We also find that the concreteness acquired by VG-NSL correlates well with a similar measure defined by linguists. Finally, we also apply VG-NSL to multiple languages in the Multi30K data set, showing that our model consistently outperforms prior unsupervised approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Experience Grounds Language (2020, background): “Vision can even inform core tasks like syntax (Shi et al., 2019) and language modeling (Ororbia et al.”
Grammar-Based Grounded Lexicon Learning (2022, methodology): “There have also been approaches for learning grammatical structures from grounded texts [35, 50, 22, 7, 33, 43].”
Grounded PCFG Induction with Images (2020, methodology): “Models in Shi et al. (2019) also match embeddings of word spans to encoded images to induce unlabeled hierarchical structures with a concreteness measure (Hill et al., 2014; Hill and Ko-rhonen, 2014).”
Learning Syntax from Naturally-Occurring Bracketings (2021, unknown): “, 2019) and cross-modal (Shi et al., 2019) signals, structural constraints (Gillenwater et al.”
Vokenization: Improving Language Understanding via Contextualized, Visually-Grounded Supervision (2020, background): “…coreference resolution (Kong et al., 2014), machine translation (Elliott et al., 2016; Ive et al., 2019; Wu et al., 2019; Zhang et al., 2020), semantic parsing (Christie et al., 2016; Shi et al., 2019; Kojima et al., 2020), and bilingual lexicon learning (Kiela et al., 2015; Vulić et al., 2016).”
Universal Multimodal Representation for Language Understanding (2023, background): “Therefore, a trend of research has beenmotivated to apply non-linguistic modalities to language representations [7], [10], [11], [12], [13], [14].”
Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction (2020, methodology): “Second, as mentioned before, we have witnessed some previous work (Shen et al., 2018b; 2019; Htut et al., 2018; Li et al., 2019; Shi et al., 2019) where the right-skewness bias is implicitly exploited, although it could be regarded as not ideal.”
Sequence-to-Sequence Learning with Latent Neural Grammars (2021, unknown): “…grammars [101]) with contemporary parameterizations, (2) conditioning on other modalities such as images/audio for grounded grammar induction [100, 130, 60, 129], (3) adapting these methods to other structured domains such as programs and graphs, and (4) investigating how grammars and…”
Natural Language to Code Translation with Execution (2022, methodology): “This method has outperformed the maximum a posteriori (MAP) method on many tasks, including speech recognition, syntactic parsing (Titov and Henderson, 2006; Shi et al., 2019; Zhang et al., 2020), statistical machine translation (Kumar and Byrne, 2004; Zhang and Gildea, 2008), and neural machine tra”
Visually Grounded Compound PCFGs (2020, methodology): “We omit details of the parser and how the semantic representations of constituents are computed, as they are not relevant to our approach, and refer the reader to Shi et al. (2019).”