Generating Expressions that Refer to Visible Objects

By Margaret Mitchell, Kees van Deemter, Ehud Reiter. 2013

66 citations recorded by the source corpus.

Subfields: Multimodal Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Multimodal Learning

1. Devi Parikh
2. Dhruv Batra
3. Marcus Rohrbach
4. Mohit Bansal
5. Zhe Gan
6. Qi Wu
7. C. Lawrence Zitnick
8. Anton van den Hengel

In Natural Language Generation

1. Emiel Krahmer
2. Liang Wang
3. Albert Gatt
4. Devi Parikh
5. Alexander Toshev
6. Junhua Mao
7. Alan L. Yuille
8. Tamara L. Berg

Citations in the corpus

• VQA: Visual Question Answering (2015, background): “Several recent papers have explored tasks at the intersection of vision and language that are easier to evaluate than image captioning, such as coreference resolution [22, 36] or generating referring expressions [20, 35] for a particular object in an image that”
• ReferItGame: Referring to Objects in Photographs of Natural Scenes (2014, methodology): “…for studying REG have used relatively focused domains such as graphics generated objects (van Deemter et al., 2006; Viethen and Dale, 2008), crafts (Mitchell et al., 2010), or small everyday (home and office) objects arrayed on a simple background (Mitchell et al., 2013a; FitzGerald et al., 2013).”
• Generation and Comprehension of Unambiguous Object Descriptions (2015, background): “Such a description is known as a “referring expression” [50, 52, 41, 42, 14, 19, 27].”
• Modeling Context in Referring Expressions (2016, methodology): “The task of referring expression generation (REG) has been studied since the 1970s [40,22,30,7], with most work focused on studying particular aspects of the problem in some relatively constrained datasets.”
• Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “More recent work has started exploring the interface between computer vision and reg to produce descriptions of objects in complex, realistic visual scenes, including photographs (e.g., Mitchell et al., 2013; Kazemzadeh et al., 2014; Mao et al., 2016).”
• A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “edding space. Mao et al. [92] proposed a special type of text generation method for images. This method can generate a description for an specific object or region that is called referring expression [37, 46, 68, 102, 103, 136, 141]. Using this expression it can then infer the object or region which”
• See No Evil, Say No Evil: Description Generation from Densely Labeled Images (2014, background): “Visual re-ferring expression generation systems (Krahmer and Van Deemter, 2012; Mitchell et al., 2013; FitzGerald et al., 2013) aim to identify specific objects, a sub-problem we deal with when describing images more generally.”
• Collaborative Language Grounding Toward Situated Human-Robot Dialogue (2017, background): “Except for a few (Mitchell, Van Deemter, and Reiter 2013), most previous works have applied the assumption that agents and humans have access to the same domain information and the agent has a complete representation of the shared world.”
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods (2019, background): “Given these tasks, different approaches are proposed for referring expression generation (Golland et al., 2010; Mitchell et al., 2013), comprehension (Kazemzadeh et al.”
• Robots That Use Language (2020, background): “(179) generated expressions that refer to visible objects that a robot might observe with its camera.”
• ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes (2020, background): “2D ReferIt Game and Grounded Vision & Language: Several papers explore connecting referential language to image regions for co-reference resolution (e.g., [15,53,42] – in videos, e.g., [8]), for generating referring expressions [29,44,42], and more.”
• Is the Red Square Big? MALeViC: Modeling Adjectives Leveraging Visual Contexts (2019, methodology): “…of simple visual scenes and corresponding descriptions have been proposed (van Deemter et al., 2006; Viethen and Dale, 2008, 2011; Mitchell et al., 2010, 2013a), and forerun-ner GRE algorithms have been tested to refer to visible objects using attributes like size (Mitchell et al., 2011, 2013b).”
• Referring Expression Generation and Comprehension via Attributes (2017, methodology): “Early works use rule-based model [17, 3] to generate the expression, while recent works rely on the LSTM model [6].”