Decoupled Novel Object Captioner

By Yuehua Wu, Linchao Zhu, Lu Jiang, Yi Yang. 2018

63 citations recorded by the source corpus.

Image captioning is a challenging task where the machine automatically describes an image by sentences or phrases. It often requires a large number of paired image-sentence annotations for training. However, a pre-trained captioning model can hardly be applied to a new domain in which some novel object categories exist, i.e., the objects and their description words are unseen during model training. To correctly caption the novel object, it requires professional human workers to annotate the images by sentences with the novel words. It is labor expensive and thus limits its usage in real-world applications. In this paper, we introduce the zero-shot novel object captioning task where the machine generates descriptions without extra training sentences about the novel object. To tackle the challenging problem, we propose a Decoupled Novel Object Captioner (DNOC) framework that can fully decouple the language sequence model from the object descriptions. DNOC has two components. 1) A Sequence Model with the Placeholder (SM-P) generates a sentence containing placeholders. The placeholder represents an unseen novel object. Thus, the sequence model can be decoupled from the novel object descriptions. 2) A key-value object memory built upon the freely available detection model, contains the visual information and the corresponding word for each object. A query generated from the SM-P is used to retrieve the words from the object memory. The placeholder will further be filled with the correct word, resulting in a caption with novel object descriptions. The experimental results on the held-out MSCOCO dataset demonstrate the ability of DNOC in describing novel concepts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Mohit Bansal
3. Zhe Gan
4. Lijuan Wang
5. Zicheng Liu
6. Devi Parikh
7. Xiaowei Hu
8. Trevor Darrell

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Trevor Darrell
4. Xiaogang Wang
5. Yi Yang
6. Cordelia Schmid
7. Jianfeng Gao
8. Jitendra Malik

In Multimodal Learning

1. Lijuan Wang
2. Zhe Gan
3. Zicheng Liu
4. Jianfeng Gao
5. Xiaowei Hu
6. Devi Parikh
7. Jianfeng Wang
8. Rita Cucchiara

Citations in the corpus

• Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, background): “…our approach works more robustly across in-and out-of-domain scenarios and is simpler than the state-of-the-art techniques that utilize constrained beam search (CBS) [5], finite state machine construction plus CBS [6], generating slot-filling templates [54, 76], and copying mechanisms [78].”
• nocaps: novel object captioning at scale (2019, methodology): “Such objects which have detection annotations but are not present in caption corpora are referred to as novel objects and the task of describing images containing novel objects is termed novel object captioning [2,3,13,25,40,43,47].”
• VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-Training (2020, background): “Neural Baby Talk (Lu et al. 2018) and Decoupled Novel Object Captioner (Wu et al. 2018) generate template sentences that are later filled in with visual concepts recognized by object detectors.”
• Pointing Novel Objects in Image Captioning (2019, unknown): “(8) DNOC [30] generates the caption template with placeholder and then fill in the placeholder with the detected objects via key-value object memory.”
• Joint Syntax Representation Learning and Visual Cue Translation for Video Captioning (2019, methodology): “Our method also differs from the two-stage image cap-tioning methods [24, 27, 42] which first generate sentence templates with slots tied to object entities, and then filling the slots using object detectors.”
• Learning to Learn Words from Visual Scenes (2019, background): “Previous work on captioning [28,31,61,59] produces words never seen in the ground truth captions.”
• From Show to Tell: A Survey on Deep Learning-Based Image Captioning (2021, background): “generate a caption template with placeholders to be filled with novel objects [106], [181].”
• Entangled Transformer for Image Captioning (2019, background): “Desipite the efforts [41, 29, 44, 12, 25, 2, 40] investigate the attention over monomodal information, many works also try to combine visual and semantic information semoutanouly.”
• Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning (2018, methodology): “But zero-shot video captioning is a more challenging task: images are static scenes, and methods based on noun word replacement can perform well on novel object captioning (Anderson et al. 2017; Wu et al. 2018; Lu et al. 2018); While describing novel activities in videos requires both temporal under”
• Egoshots, an ego-vision life-logging dataset and semantic fidelity metric to evaluate diversity in image captioning models (2020, unknown): “2a. Decoupled Novel Object Captioner (DNOC) (Wu et al., 2018) follows a two-step process for generating sentences.”