Resolving Language and Vision Ambiguities Together: Joint Segmentation & Prepositional Attachment Resolution in Captioned Scenes

By Gordon A. Christie, Ankit Laddha, Aishwarya Agrawal, Stanislaw Antol, Yash Goyal, Kevin Kochersberger, Dhruv Batra. 2016

31 citations recorded by the source corpus.

We present an approach to simultaneously perform semantic segmentation and prepositional phrase attachment resolution for captioned images. Some ambiguities in language cannot be resolved without simultaneously reasoning about an associated image. If we consider the sentence "I shot an elephant in my pajamas", looking at language alone (and not using common sense), it is unclear if it is the person or the elephant wearing the pajamas or both. Our approach produces a diverse set of plausible hypotheses for both semantic segmentation and prepositional phrase attachment resolution that are then jointly reranked to select the most consistent pair. We show that our semantic segmentation and prepositional phrase attachment resolution modules have complementary strengths, and that joint reasoning produces more accurate results than any module operating in isolation. Multiple hypotheses are also shown to be crucial to improved multiple-module reasoning. Our vision and language approach significantly outperforms the Stanford Parser (De Marneffe et al., 2006) by 17.91% (28.69% relative) and 12.83% (25.28% relative) in two different experiments. We also make small improvements over DeepLab-CRF (Chen et al., 2015).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Visual Dialog (2016, background): “A number of problems at the intersection of vision and language have recently gained prominence – image captioning [12, 13, 23, 56], video/movie description [45, 53, 54], text-to-image coreference/grounding [8, 19, 24, 39, 41, 44], visual storytelling [2, 20], and of course, visual question answerin”
Visually Grounded Neural Syntax Acquisition (2019, unknown): “Finally, it may be possible to extend our approach to other linguistic tasks such as dependency parsing (Christie et al., 2016b), coreference resolution (Kottur et al., 2018), and learning pragmatics beyond semantics (Andreas and Klein, 2016).”
Unified Visual-Semantic Embeddings: Bridging Vision and Language With Structured Meaning Representations (2019, methodology): “This is related to previous works on using image segmentation models to facilitate the sentence parsing [6].”
Vokenization: Improving Language Understanding via Contextualized, Visually-Grounded Supervision (2020, background): “…coreference resolution (Kong et al., 2014), machine translation (Elliott et al., 2016; Ive et al., 2019; Wu et al., 2019; Zhang et al., 2020), semantic parsing (Christie et al., 2016; Shi et al., 2019; Kojima et al., 2020), and bilingual lexicon learning (Kiela et al., 2015; Vulić et al., 2016).”