Domain-Specific Image Captioning
By Rebecca Mason, Eugene Charniak. 2014
21 citations recorded by the source corpus.
We present a data-driven framework for image caption generation which incorporates visual and textual features with varying degrees of spatial structure. We propose the task of domain-specific image captioning, where many relevant visual details cannot be captured by off-the-shelf general-domain entity detectors. We extract previously-written descriptions from a database and adapt them to new query images, using a joint visual and textual bag-of-words model to determine the correctness of individual words. We implement our model using a large, unlabeled dataset of women’s shoes images and natural language descriptions (Berg et al., 2010). Using both automatic and human evaluations, we show that our captioning method effectively deletes inaccurate words from extracted captions while maintaining a high level of detail in the generated output.
Subfields: Computer Vision and Pattern Recognition, Multimodal Learning, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Trevor Darrell
- Dacheng Tao
- Jitendra Malik
- Xiaogang Wang
- Ming-Hsuan Yang
- Bernt Schiele
- Yun Raymond Fu
In Multimodal Learning
- Albert Gatt
- Emiel Krahmer
- C. Lawrence Zitnick
- Margaret Mitchell
- Ross B. Girshick
- Jacob Devlin
- Saurabh Gupta
- Percy Liang
In Natural Language Generation
- Emiel Krahmer
- Margaret Mitchell
- Albert Gatt
- Saurabh Gupta
- Ross B. Girshick
- Jacob Devlin
- Percy Liang
- Yonatan Oren
Citations in the corpus
- A Retrieve-and-Edit Framework for Predicting Structured Outputs (2018, methodology): “However, the desired output might be a variation of another, previously-observed example [14, 13, 30, 18, 24].”
- Exploring Nearest Neighbor Approaches for Image Captioning (2015, methodology): “Nearest neighbor approaches have a rich history in work on predicting words given images, used in face recognition [31] as well as recent work in retrieval-based caption generation [9], [27].”
- Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, methodology): “Mason and Charniak (2014) use a domain-specific dataset to extract descriptions and adapt them to a query image using a joint visual and textual bag-of-words model.”
- Visual to Text: Survey of Image and Video Captioning (2019, background): “image caption generation [70], which incorporates visual and textual features with varying degrees of spatial structures.”