Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
By Raffaella Bernardi, Ruken Cakici, Desmond Elliott, Aykut Erdem, Erkut Erdem, Nazli Ikizler-Cinbis, Frank Keller, Adrian Muscat, Barbara Plank. 2016
393 citations recorded by the source corpus.
Automatic description generation from natural images is a challenging problem that has recently received a large amount of interest from the computer vision and natural language processing communities. In this survey, we classify the existing approaches based on how they conceptualize this problem, viz., models that cast description as either generation problem or as a retrieval problem over a visual or multimodal representational space. We provide a detailed review of existing models, highlighting their advantages and disadvantages. Moreover, we give an overview of the benchmark image datasets and the evaluation measures that have been developed to assess the quality of machine-generated image descriptions. Finally we extrapolate future directions in the area of automatic image description generation.
Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Yi Yang
- Andrew Zisserman
- Jitendra Malik
In Multimodal Learning
- Desmond Elliott
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Loïc Barrault
- Lucia Specia
- Dietrich Klakow
- Ozan Caglayan
In Natural Language Generation
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
- Giuseppe Fiameni
- Matteo Stefanini
- Marimuthu Kalimuthu
- Dietrich Klakow
- Aditya Mogadala
Citations in the corpus
- SPICE: Semantic Propositional Image Caption Evaluation (2016, background): “As such, there is an urgent need to develop new automated evaluation metrics for this task [8,9].”
- Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, methodology): “Our discussion will be brief, since image captioning has recently been the subject of an extensive review by Bernardi et al. (2016), and has also been discussed against the background of broader issues in research on the visionlanguage interface by Barnard (2016).”
- Supervised Multimodal Bitransformers for Classifying Images and Text (2019, background): “…research in NLP range from classification to cross-modal retrieval (Weston, Bengio, and Usunier 2011; Frome et al. 2013; Socher et al. 2013) to image captioning (Bernardi et al. 2016) to visual question answering (Antol et al. 2015) and multimodal machine translation (Elliott et al. 2017).”
- Improved Image Captioning via Policy Gradient optimization of SPIDEr (2016, methodology): “There are many ways to perform this task (see [5] for a recent review).”
- Visual question answering: Datasets, algorithms, and future challenges (2016, neutral): “While other evaluation metrics, especially CIDEr and METEOR, show more robustness in terms of agreement with human judges, they still often rank automatically generated captions higher than human captions [27].”
- A Shared Task on Multimodal Machine Translation and Crosslingual Image Description (2016, background): “…applications involving both Natural Language Processing (NLP) and Computer Vision (CV) include image description generation and video captioning (Bernardi et al., 2016), image retrieval based on textual and visual cues (Feng and Lapata, 2010), visual question answering (Yang et al., 2015),…”
- Evaluation of Text Generation: A Survey (2020, background): “Others survey specific NLG tasks or NLG models, such as image captioning (Bernardi et al., 2016; Kilickaya et al., 2017; Hossain et al., 2018; Li et al., 2019; Bai & An, 2018), machine translation (Dabre et al.”
- A Comprehensive Survey of Deep Learning for Image Captioning (2018, background): “So far, only three survey papers [8, 13, 75] have been published on this research topic.”
- Talking about other people: an endless range of possibilities (2018, methodology): “These datasets are typically used to train data-driven natural language generation systems to automatically learn to associate visual features with natural language descriptions (Bernardi et al., 2016).”
- Measuring the Diversity of Automatic Image Descriptions (2018, background): “Automatic image description is a challenging task because natural language and the visual world both have an unbounded range of variation (Bernardi et al., 2016).”
- Semantic Speech Retrieval With a Visually Grounded Model of Untranscribed Speech (2017, background): “has also been studied extensively, where the goal is to produce a natural language description for a visual scene [47]–[50].”
- Selective Replay Enhances Learning in Online Continual Analogical Reasoning (2021, background): “Some problems requiring these additional components include visual question answering [8, 44, 56], referring expression recognition [45, 68], visual query detection [3], and image captioning [11].”
- Fluency-Guided Cross-Lingual Image Captioning (2017, neutral): “ree leading approaches have been explored [2].”
- From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “Compared to previous surveys on image captioning [16], [17], [18], [21], [22], we provide a comprehensive and updated view on deep learning-based generative captioning models.”