Learning Like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images

By Junhua Mao, Xu Wei, Yi Yang, Jiang Wang, Zhiheng Huang, Alan L. Yuille. 2015

161 citations recorded by the source corpus.

In this paper, we address the task of learning novel visual concepts, and their interactions with other concepts, from a few images with sentence descriptions. Using linguistic context and visual features, our method is able to efficiently hypothesize the semantic meaning of new words and add them to its word dictionary so that they can be used to describe images which contain these novel concepts. Our method has an image captioning module based on with several improvements. In particular, we propose a transposed weight sharing scheme, which not only improves performance on image captioning, but also makes the model more suitable for the novel concept learning task. We propose methods to prevent overfitting the new concepts. In addition, three novel concept datasets are constructed for this new task, and are publicly available on the project page. In the experiments, we show that our method effectively learns novel visual concepts from a few examples without disturbing the previously learned concepts. The project page is: www.stat.ucla.edu/junhua. mao/projects/child_learning.html.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Meta & Continual Learning

Citations in the corpus

Long-term recurrent convolutional networks for visual recognition and description (2014, methodology): “e only one 9 Generation Performance (COCO 2014 [33] Test Set) Method B1 B2 B3 B4 C M R [38] NIC 0.895 0.802 0.694 0.587 0.946 0.346 0.682 [39] MSR Captivator 0.907 0.819 0.710 0.601 0.937 0.339 0.680 [40] m-RNN (2015) 0.890 0.798 0.687 0.575 0.935 0.325 0.666 (Ours) * LRCN, this work (sample) 0.895 ”
Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question (2015, methodology): “This strategy will reduce nearly half of the parameters in the model and is shown to have better performance in image captioning and novel visual concept learning tasks [25].”
Generating Visual Explanations (2016, background): “For example, [40,41] propose introducing new vocabulary words into a captioning system.”
Image Captioning with Semantic Attention (2016, background): “The E term is inspired by the transposed weight sharing trick [25] for parameter reduction.”
Few-Shot Image Recognition by Predicting Parameters from Activations (2017, methodology): “30% Learning like a Child [24] 100% 1 Y 76.”
Deal or No Deal? End-to-End Learning of Negotiation Dialogues (2017, methodology): “The token at each time step is predicted with a softmax, which uses weight tying with the embedding matrix E (Mao et al., 2015):”
Guided Open Vocabulary Image Captioning with Constrained Beam Search (2016, methodology): “Our approach to out-of-domain image captioning could be applied to any existing CNN-RNN cap-tioning model that can be decoding using beam search, e.g., (Donahue et al., 2015; Mao et al., 2015; Karpathy and Fei-Fei, 2015; Vinyals et al., 2015; Devlin et al., 2015).”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, methodology): “Finally, Mao et al. (2015a) experiment with various mrnn configurations, obtaining their best results with an architecture in which there are two word embedding layers preceding the recurrent layer, which is in turn projected into a multimodal layer where linguistic features are combined with cnn…”
Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data (2015, methodology): “However, [21] do not consider how to describe objects when no paired imagesentence data is available.”
Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures (2016, methodology): “Mason and Charniak’s (2014) description generation approach differs from the models discussed above in that it formulates description generation as an extractive summarization problem, and it selects the output description by considering only the textual information in the final re-ranking step.”
A Comprehensive Survey of Deep Learning for Image Captioning (2018, methodology): “The top-down paradigms [23, 31, 65, 93, 94, 142, 152] are used for extracting visual features first and then convert them into words.”
Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval (2019, methodology): “Incremental learning methods [24, 2, 29, 38, 34] adapted models to gradually available data and required overcoming catastrophic forgetting.”
Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images (2016, methodology): “Model A is inspired by the CNN-RNN based image captioning models [36, 23].”
From Red Wine to Red Tomato: Composition with Context (2017, background): “As such explicit knowledge is not always available, another line of work [10, 21, 38– 40, 49] relies on the finding such similarity in the linguistic space.”
Pointing Novel Objects in Image Captioning (2019, unknown): “The task of novel object captioning has received increasing attention most recently, which leverages additional image-sentence paired data [15] or unpaired image/text data [8, 26] to describe novel objects.”
Title Generation for User Generated Videos (2016, methodology): “Recently, image captioning methods [15,16,17,18,19,20] begin to adopt the Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) approaches.”