Guided Open Vocabulary Image Captioning with Constrained Beam Search

By Peter Anderson, Basura Fernando, Mark Johnson, Stephen Gould. 2016

253 citations recorded by the source corpus.

Existing image captioning models do not generalize well to out-of-domain images containing novel scenes or objects. This limitation severely hinders the use of these models in real world applications dealing with images in the wild. We address this problem using a flexible approach that enables existing deep captioning architectures to take advantage of image taggers at test time, without re-training. Our method uses constrained beam search to force the inclusion of selected tag words in the output, and fixed, pretrained word embeddings to facilitate vocabulary expansion to previously unseen tag words. Using this approach we achieve state of the art results for out-of-domain captioning on MSCOCO (and improved results for in-domain captioning). Perhaps surprisingly, our results significantly outperform approaches that incorporate the same tag predictions into the learning algorithm. We also show that we can significantly improve the quality of generated ImageNet captions by leveraging ground-truth labels.

Subfields: Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (2022, methodology): “Therefore, we also develop ALICE, a variant of constrained beam search (CBS; Anderson et al., 2017; Hokamp and Liu, 2017; Holtzman et al., 2018; Lu et al., 2021) during decoding that generates statements that are adversarial to a given pre-trained toxicity classifier.”
Autoregressive Entity Retrieval (2020, methodology): “Lexically constrained decoding has been used to force the inclusion of pre-specified words for machine translation (Hokamp & Liu, 2017; Post & Vilar, 2018), and image captioning (Anderson et al., 2017).”
ClipCap: CLIP Prefix for Image Captioning (2021, methodology): “Therefore, we do not employ a constrained beam search [2].”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, methodology): “…our approach works more robustly across in-and out-of-domain scenarios and is simpler than the state-of-the-art techniques that utilize constrained beam search (CBS) [5], ﬁnite state machine construction plus CBS [6], generating slot-ﬁlling templates [54, 76], and copying mechanisms [78].”
Neural Baby Talk (2018, methodology): “G means greedy decoding, and T1−2 means using constrained beam search [2] with 1−2 top detected concepts.”
VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-Training (2020, methodology): “Similarly, Constrained Beam Search (Anderson et al. 2017) is exploited to generate captions that contain detected novel objects (Agrawal et al. 2019).”
Meshed-Memory Transformer for Image Captioning (2019, methodology): “We compare with Up-Down [4] and Neural Baby Talk [27], when using GloVe word embed-dings and Constrained Beam Search (CBS) [3] to address the generation of out-of-vocabulary words and constrain the presence of categories detected by an object detector.”
Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation (2018, methodology): “A related approach was that of Anderson et al. (2017), who similarly extended beam search with a ﬁnite state machine whose states marked completed subsets of the set of constraints, at an exponential cost in the number of constraints.”
Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects (2019, background): “RQ2: How does aspect planning affect generation? To mitigate the trade-off between diversity and relevance, one approach is to add more constraints during generation such as constrained Beam Search (Anderson et al., 2017).”
Universal Captioner: Long-Tail Vision-and-Language Model Training through Content-Style Separation (2021, background): “Traditionally, this has been addressed through the Novel Object Captioning protocol [1,2,64], which requires the captioner to name objects unseen in training captions, relying on object detection tags [20] or uni-modal pre-training [18].”
Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning (2018, background): “But zero-shot video captioning is a more challenging task: images are static scenes, and methods based on noun word replacement can perform well on novel object captioning (Anderson et al. 2017; Wu et al. 2018; Lu et al. 2018); While describing novel activities in videos requires both temporal under”
Decoupled Novel Object Captioner (2018, unknown): “Novel Object Captioner (NOC) to further improve the DCC to an end-to-end system by jointly training the visual classification model, language sequence model, and the captioning model. Anderson et al. [2] leveraged an approximate search algorithm to forcibly guarantee the inclusion of selected words ”
Parallel Sentence Mining by Constrained Decoding (2020, methodology): “In our work, using NMT as a similarity scorer relies on constrained decoding (Hokamp and Liu, 2017), which has been applied on image captioning (Anderson et al., 2017) and keyword generation (Lian et al., 2019).”
Injecting Semantic Concepts into End-to-End Image Captioning (2021, background): “Re-sults are reported with constrained beam search (CBS) decoding [2]. we train the ViTCAP on the training split, which consists of ∼ 3 .”
EDITOR: An Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints (2020, background): “(Hokamp and Liu, 2017) and caption generation (Anderson et al., 2017), they suffer from several issues: Constrained training requires building dedicated models for constrained language generation, while constrained decoding adds significant computational overhead and treats all constraints as hard c”
nocaps : novel object captioning at scale (2019, methodology): “In addition, to handle unseen objects in training data, following [2], we initialize the softmax layer matrix ( W p , b p ) using word embedding and keep this layer ﬁxed during training.”