Neural Baby Talk

By Jiasen Lu, Jianwei Yang, Dhruv Batra, Devi Parikh. 2018

463 citations recorded by the source corpus.

We introduce a novel framework for image captioning that can produce natural language explicitly grounded in entities that object detectors find in the image. Our approach reconciles classical slot filling approaches (that are generally better grounded in images) with modern neural captioning approaches (that are generally more natural sounding and accurate). Our approach first generates a sentence 'template' with slot locations explicitly tied to specific image regions. These slots are then filled in by visual concepts identified in the regions by object detectors. The entire architecture (sentence template generation and slot filling with object detectors) is end-to-end differentiable. We verify the effectiveness of our proposed model on different image captioning tasks. On standard image captioning and novel object captioning, our model reaches state-of-the-art on both COCO and Flickr30k datasets. We also demonstrate that our model has unique advantages when the train and test distributions of scene compositions - and hence language priors of associated captions - are different. Code has been made available at: https://github.com/jiasenlu/NeuralBabyTalk.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

VideoBERT: A Joint Model for Video and Language Representation Learning (2019, background): “, [11, 8, 15]), which is a model of the form p(y|x), where y is the manually provided caption and x is the image.”
VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning (2021, background): “Novel object captioning aims to describe objects that do not exist in the training data [1, 23, 41, 50, 70].”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, background): “…our approach works more robustly across in-and out-of-domain scenarios and is simpler than the state-of-the-art techniques that utilize constrained beam search (CBS) [5], ﬁnite state machine construction plus CBS [6], generating slot-ﬁlling templates [54, 76], and copying mechanisms [78].”
Unified Vision-Language Pre-Training for Image Captioning and VQA (2019, methodology): “Existing works (Zhou et al. 2019; Lu et al. 2018) regard region object labels (probabilities) ( C i ) as an important auxiliary to enrich image region features and here we follow a similar design.”
nocaps: novel object captioning at scale (2019, methodology): “In this work, we investigate two different approaches: Neural Baby Talk (NBT) [25] and Constrained Beam Search (CBS) [2] on our challenging benchmark – both of which recently claimed state-of-the-art on the proof-ofconcept novel object captioning dataset [13].”
CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning (2020, background): “There are a few recent works that incorporate common-sense knowledge in language generation tasks such as essay generation (Guan et al., 2019; Yang et al., 2019a), image captioning (Lu et al., 2018), video storytelling (Yang et al., 2019b), and conversational systems (Zhang et al., 2020a).”
SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, background): “The ability to describe images and videos is one of the core challenges in this domain and a crucial achievement towards machine intelligence [6], [7], [8], [9].”
Connecting Vision and Language with Localized Narratives (2019, background): “Yet, many researchers want deeper visual grounding which links specific words in the caption to specific regions in the image [36,37,49,50].”
TVQA+: Spatio-Temporal Grounding for Video Question Answering (2019, methodology): “Inspired by previous work on grounded image and video captioning [24, 47, 46], we propose methods that explicitly localize video moments as well as spatial regions for answering video-based questions.”
VIVO: Surpassing Human Performance in Novel Object Captioning with Visual Vocabulary Pre-Training (2020, background): “Neural Baby Talk (Lu et al. 2018) and Decoupled Novel Object Captioner (Wu et al.”
Auto-Encoding Scene Graphs for Image Captioning (2018, methodology): “Modern image captioning models employ an end-to-end encoder-decoder framework [10, 30, 29, 2, 27], i.”
Meshed-Memory Transformer for Image Captioning (2019, methodology): “We compare with Up-Down [4] and Neural Baby Talk [27], when using GloVe word embed-dings and Constrained Beam Search (CBS) [3] to address the generation of out-of-vocabulary words and constrain the presence of categories detected by an object detector.”
Grounded Video Description (2018, methodology): “For image captioning, we compare against two SotA methods, Neural Baby Talk (NBT) [15] and BUTD [1].”
TextCaps: a Dataset for Image Captioning with Reading Comprehension (2020, methodology): “Apart from that, unlike conventional VQA models where a prediction is made via classification, it enables iterative answer decoding with a dynamic pointer network [25,36], allowing the model to generate a multi-word answer, which is not limited to a fixed vocabulary.”
Object Hallucination in Image Captioning (2018, methodology): “This is in part because when generating sentences we use the detector outputs provided by Lu et al. (2018).”
Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded (2019, background): “For the dominant paradigm of vision-and-language models that compute an explicit question-guided attention over image regions [28, 33, 14, 35, 20, 3], a seemingly straight-”
XGPT: Cross-modal Generative Pre-Training for Image Captioning (2020, neutral): “3 NBT (with BBox) [31] 57.”
Look Before you Speak: Visually Contextualized Utterances (2020, background): “Thanks to off-theshelf high-quality object detection [30, 64] and simple bounding box representations, object-centric features have been widely used in a number of image and language problems [4, 51].”