Is this a wampimuk? Cross-modal mapping between distributional semantics and the visual world

By Angeliki Lazaridou, Elia Bruni, Marco Baroni. 2014

112 citations recorded by the source corpus.

Following up on recent work on establishing a mapping between vector-based semantic embeddings of words and the visual representations of the corresponding objects from natural images, we first present a simple approach to cross-modal vector-based semantics for the task of zero-shot learning, in which an image of a previously unseen object is mapped to a linguistic representation denoting its word. We then introduce fast mapping, a challenging and more cognitively plausible variant of the zero-shot task, in which the learner is exposed to new objects and the corresponding words in very limited linguistic contexts. By combining prior linguistic and visual knowledge acquired about words and their objects, as well as exploiting the limited new evidence available, the learner must learn to associate new objects with words. Our results on this task pave the way to realistic simulations of how children or robots could use existing knowledge to bootstrap grounded semantic knowledge about new concepts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Microsoft COCO Captions: Data Collection and Evaluation Server (2015, background): “The automatic generation of captions for images is a long standing and challenging problem in artiﬁcial intelligence [1], [2], [3], [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18], [19].”
Learning Like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images (2015, methodology): “[31] developed this idea by only showing the novel words a few times.”
Weakly-Supervised Learning of Visual Relations (2017, background): “Zero-shot learning has been mostly explored for object classiﬁcation [12, 28, 45, 47] and recently for the task of describing images with novel objects [18, 46].”
Distributional vectors encode referential attributes (2015, background): “…and other representational spaces, such as subject-generated concept properties (Johns and Jones, 2012; Hill et al., 2014; F˘ag˘ar˘as¸an et al., 2015), visual features (Frome et al., 2013; Socher et al., 2013b; Lazaridou et al., 2014) or brain signals (Mitchell et al., 2008; Murphy et al., 2012).”
Combining Language and Vision with a Multimodal Skip-gram Model (2015, background): “The recent work on so-called zero-shot learning to address the annotation bottleneck in image labeling (Frome et al., 2013; Lazaridou et al., 2014; Socher et al., 2013) looks at image- and text-based vectors from a different perspective.”
Improving zero-shot learning by mitigating the hubness problem (2014, methodology): “A straightforward and performant choice (Lazaridou et al., 2014a; Mikolov et al., 2013b) is to assume the mapping function is a linear map W , and use a l2-regularized least-squares error objective: ˆ W = arg min where X and Y are matrices obtained through the concatenation of train source vectors…”
Captioning Images with Diverse Objects (2016, background): “…takes advantage of distributional semantics to learn a joint embedding space using visual and textual infor- mation for zero-shot labeling of novel object categories (Frome et al., 2013; Norouzi et al., 2013), as well as retrieval of images with text (Socher et al., 2014; Lazaridou et al., 2014).”
Sentiment Analysis: Detecting Valence, Emotions, and Other Affectual States from Text (2016, background): “First Socher, Perelygin, Wu, Chuang, Manning, Ng, and Potts (2013), then Le and Mikolov (2014), obtained significant improvements in valence classification on a movie reviews dataset (Pang & Lee, 2008) using word embeddings.”
Feuding Families and Former Friends: Unsupervised Learning for Dynamic Fictional Relationships (2016, unknown): “Other applications for which researchers have prioritized interpretable vector representations include text-to-vision mappings (Lazaridou et al., 2014) and word embeddings (Fyshe et al.”
Learning Image Embeddings using Convolutional Neural Networks for Improved Multi-Modal Semantics (2014, background): “Multimodal models with deep learning components have also successfully been employed in crossmodal tasks (Lazaridou et al., 2014).”
Building a shared world: mapping distributional to model-theoretic semantic spaces (2015, background): “…al. (2013) learn a linear regression to transform vector-based image representations onto vectors representing the same concepts in a linguistic semantic space, and Lazaridou et al. (2014) explore mapping techniques to learn a cross-modal mapping between text and images with promising performance.”
A Survey of Zero-Shot Learning (2019, methodology): “The regression function can be a linear regression function [82, 102, 113] or a nonlinear regression function [82, 135].”
Like a Baby: Visually Situated Neural Language Acquisition (2018, unknown): “…either to achieve a better model of language as it exists in the minds of humans (Baroni, 2016; Johns and Jones, 2012; Kievit-Kylar and Jones, 2011; Lazaridou et al., 2014) or to improve performance on machine learning tasks such as object recognition (Frome et al., 2013; Lazaridou et al., 2015a),…”
Deep Multimodal Fusion by Channel Exchanging (2020, background): “Another categorization of multimodal fusion can be specified as early, middle, and late fusion, depending on when to fuse, which have been discussed in earlier works [2, 7, 17, 41] and also in the deep learning literature [4, 26, 27, 45].”
Multimodal Machine Learning: A Survey and Taxonomy (2017, methodology): “[123] present a fast mapping method for ZSL by mapping extracted visual feature vectors to text-based vectors through a neural network.”