TreeTalk: Composition and Compression of Trees for Image Descriptions
By Polina Kuznetsova, Vicente Ordonez, Tamara L. Berg, Yejin Choi. 2014
288 citations recorded by the source corpus.
We present a new tree based approach to composing expressive image descriptions that makes use of naturally occuring web images with captions. We investigate two related tasks: image caption generalization and generation, where the former is an optional subtask of the latter. The high-level idea of our approach is to harvest expressive phrases (as tree fragments) from existing image descriptions, then to compose a new description by selectively combining the extracted (and optionally pruned) tree fragments. Key algorithmic components are tree composition and compression, both integrating tree structure with sequence structure. Our proposed system attains significantly better performance than previous approaches for both image caption generalization and generation. In addition, our work is the first to show the empirical benefit of automatically generalized captions for composing natural image descriptions.
Subfields: Computer Vision and Pattern Recognition, Natural Language Generation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Trevor Darrell
- Dacheng Tao
- Xiaogang Wang
- Yi Yang
- Chunhua Shen
- Jitendra Malik
- Ming-Hsuan Yang
In Natural Language Generation
- Marcus Rohrbach
- Trevor Darrell
- Kate Saenko
- Lisa Anne Hendricks
- Subhashini Venugopalan
- Rita Cucchiara
- Lorenzo Baraldi
- Marcella Cornia
Citations in the corpus
- Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “The automatic generation of captions for images is a long standing and challenging problem in artificial intelligence [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19].”
- Long-term recurrent convolutional networks for visual recognition and description (2014, methodology): “As seen in Table 4, our finetuned (FT) LRCN model performs on par with the Nearest Neighbour (NN) on correctness and relevance, and better on Correctness Grammar Relevance TreeTalk [23] 4.”
- Show and tell: A neural image caption generator (2014, methodology): “Descriptions were preprocessed with basic tokenization, keeping all words that appeared at least 5 times in the training set.”
- Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “We then treat these correspondences as training data for our multimodal Recurrent Neural Network model which learns to generate the descriptions.”
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention (2015, methodology): “The second approach was based on first retrieving similar captioned images from a large database then modifying these retrieved captions to fit the query (Kuznetsova et al., 2012; 2014).”
- Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN) (2014, background): “The second category retrieves similar captioned images, and generates new descriptions by generalizing and re-composing the retrieved captions (Kuznetsova et al. (2014)).”
- Movie Description (2016, background): “While more and more movies are audio transcribed, it may take up to 60 person-hours to describe a 2-hour movie (Lakritz and Salway 2006), resulting in the fact that today only a small subset of movies and TV programs are available for the blind.”
- A dataset for Movie Description (2015, background): “In recent years there has been an increased interest in automatically describing images [23, 39, 40, 50, 45, 40, 41, 34, 61, 22] and videos [37, 27, 8, 28, 32, 62, 16, 26, 64, 55] with natural language.”
- Sequence to Sequence -- Video to Text (2015, background): “Describing visual content with natural language text has recently received increased interest, especially describing images with a single sentence [8, 5, 16, 18, 20, 23, 29, 40].”
- Translating Videos to Natural Language Using Deep Recurrent Neural Networks (2014, background): “Propelled by successes of deep learning, several groups released record breaking results in just the past year (Don-ahue et al., 2014; Mao et al., 2014; Karpathy et al., 2014; Fang et al., 2014; Kiros et al., 2014; Vinyals et al., 2014; Kuznetsova et al., 2014).”
- Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge (2016, methodology): “An “encoder” RNN reads the source sentence and transforms it into a rich fixed-length vector representation, which in turn in used as the initial hidden state of a “decoder” RNN that generates the target sentence.”
- Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, methodology): “…followed by a step mapping these outputs to linguistic structures (‘sentence plans’ of the sort discussed in Section 2 and 3), such as trees or templates (e.g. Kulkarni et al., 2011; Yang et al., 2011; Mitchell et al., 2012; Elliott & De Vries, 2015; Yatskar et al., 2014; Kuznetsova et al., 2014).”
- What Value Do Explicit High Level Concepts Have in Vision to Language Problems? (2015, methodology): “Instead of using fixed templates, more powerful language models based on language parsing have been developed, such as [1, 27, 28, 39].”
- A Comprehensive Survey of Deep Learning for Image Captioning (2018, background): “Moreover, later on, parsing-based language models have been introduced in image captioning [2, 32, 76, 77, 101], which are more powerful than fixed template-based methods.”
- StyleNet: Generating Attractive Visual Captions with Styles (2017, methodology): “These approaches first retrieve the visually similarity images from a large database, and then transfer the captions of retrieved images to fit the query image [28, 36, 20, 41].”
- Toward Abstractive Summarization Using Semantic Representations (2018, background): “…for people to understand (e.g., website privacy policies; Sadeh et al., 2013), but also for non-textual media (e.g., videos and image collections; Kim et al., 2014; Kuznetsova et al., 2014; Zhao and Xing, 2014), where extractive and compressive summarization techniques simply do not suffice.”
- Deep correlation for matching images and text (2015, background): “Recently, maximum entropy based [10] and tree based [27] language models have been proposed for caption generation.”
- Deep Visual-Semantic Alignments for Generating Image Descriptions (2017, background): “A number of approaches pose the task as a retrieval problem, where the most compatible annotation in the training set is transferred to a test image [15] [42] [7] [36] [17], or where training annotations are broken up and stitched together [24] [28] [25].”