Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures (Extended Abstract)
By Raffaella Bernardi, Ruken Cakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, Barbara Plank. 2017
25 citations recorded by the source corpus.
Automatic image description generation is a challenging problem that has recently received a large amount of interest from the computer vision and natural language processing communities. In this survey, we classify the known approaches based on how they conceptualise this problem and provide a review of existing models, highlighting their advantages and disadvantages. Moreover, we give an overview of the benchmark image-text datasets and the evaluation measures that have been developed to assess the quality of machine-generated descriptions. Finally we explore future directions in the area of automatic image description.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Desmond Elliott
- Lucia Specia
- Mohit Bansal
- Yonatan Belinkov
- Loïc Barrault
- Fethi Bougares
- Tat-seng Chua
- Afra Alishahi
In Computer Vision and Pattern Recognition
- Mohit Bansal
- Desmond Elliott
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Jianfeng Gao
- Xiaogang Wang
- Bernt Schiele
In Natural Language Processing
- Lucia Specia
- Desmond Elliott
- Yonatan Belinkov
- Grzegorz Chrupała
- Loïc Barrault
- Afra Alishahi
- Stella Frank
- Lieke Gelderloos
Citations in the corpus
- Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning (2018, background): “This task has seen an explosion in proposed solutions based on deep learning architectures (Bengio, 2009), starting with the winners of the 2015 COCO challenge (Vinyals et al., 2015a; Fang et al., 2015), and continuing with a variety of improvements (see e.g. Bernardi et al. (2016) for a review).”
- A Shared Task on Multimodal Machine Translation and Crosslingual Image Description (2016, background): “…applications involving both Natural Language Processing (NLP) and Computer Vision (CV) include image description generation and video captioning (Bernardi et al., 2016), image retrieval based on textual and visual cues (Feng and Lapata, 2010), visual question answering (Yang et al., 2015),…”
- Visual Translation Embedding Network for Visual Relation Detection (2017, background): “We are witnessing the impressive development in connecting computer vision and natural language, from the arguably mature visual detection [18, 36] to the burgeoning visual captioning and question answering [3, 5].”
- Representations of language in a model of visually grounded speech signal (2017, background): “The field is large and growing, with most work dedicated to the grounding of written text, particularly in image captioning tasks (see Bernardi et al. (2016) for an overview). However, learning to ground language to visual information is also interesting from an automatic speech recognition point of”
- Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Description (2017, background): “The vast majority of image description research has focused on Englishlanguage description due to the abundance of crowdsourced resources (Bernardi et al., 2016).”