Order-Embeddings of Images and Language

By Ivan Vendrov, Ryan Kiros, Sanja Fidler, Raquel Urtasun. 2015

600 citations recorded by the source corpus.

Hypernymy, textual entailment, and image captioning can be seen as special cases of a single visual-semantic hierarchy over words, sentences, and images. In this paper we advocate for explicitly modeling the partial order structure of this hierarchy. Towards this goal, we introduce a general method for learning ordered representations, and show how it can be applied to a variety of tasks involving images and language. We show that the resulting representations improve performance over current approaches for hypernym prediction and image-caption retrieval.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

A Structured Self-attentive Sentence Embedding (2017, neutral): “…encoders (Bowman et al., 2016) 83.2% 300D NTI-SLSTM-LSTM encoders (Munkhdalai & Yu, 2016a) 83.4% 1024D GRU encoders with SkipThoughts pre-training (Vendrov et al., 2015) 81.4% 300D NSE encoders (Munkhdalai & Yu, 2016b) 84.6% Our method 84.4% We ﬁnd that compared to other published approaches, our…”
An efficient framework for learning sentence representations (2018, methodology): “Image Search Model R@1 R@5 R@10 Med r R@1 R@5 R@10 Med The purely supervised models are respectively from (Karpathy & Fei-Fei, 2015), (Klein et al., 2015), (Mao et al., 2014) and (Vendrov et al., 2015).”
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, neutral): “8 3 Order-embeddings (Vendrov et al., 2016) 46.”
Root Mean Square Layer Normalization (2019, methodology): “We train an order-embedding model (OE) proposed by Vendrov et al. [32] on the Microsoft COCO dataset [17] using their public source code in Theano.”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, methodology): “We compare our results with several representative image-text retrieval approaches, i.e., Embedding-Net [53], 2Way-Net [7], Sm-LSTM [19], Order-Embedding [52], SAE [13], VSE [30] and VSEPP [8].”
Grounded Language Learning in a Simulated 3D World (2017, background): “These mappings can also yield plausible descriptions of visual scenes that were not observed during training (Xu et al., 2015; Vendrov et al., 2015).”
Flow Contrastive Estimation of Energy-Based Models (2019, background): “NCE shows effectiveness in typical NLP tasks such as word embeddings [47] and order embeddings [71].”
Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering (2017, methodology): “We then train on Visual Genome [21] data.”
Enhanced LSTM for Natural Language Inference (2016, background): “The approach proposed by Munkhdalai and Yu (2016a) (2) 300D LSTM encoders (Bowman et al., 2016) 3.0M 83.9 80.6 (3) 1024D pretrained GRU encoders (Vendrov et al., 2015) 15M 98.8 81.4 (4) 300D tree-based CNN encoders (Mou et al., 2016) 3.5M 83.3 82.1 (5) 300D SPINN-PI encoders (Bowman et al., 2016)…”
Stacked Cross Attention for Image-Text Matching (2018, background): “A rich line of studies have explored mapping whole images and full sentences to a common semantic vector space for image-text matching [22, 38, 2, 39, 23, 27, 44, 10, 33, 13, 9, 8, 11].”
A Joint Sequence Fusion Model for Video Question Answering and Retrieval (2018, neutral): “Visual information retrieval with natural language queries has long been tackled via joint visuallanguage embedding models [12,19,24,25,26,27,28].”
Learning Natural Language Inference with LSTM (2015, methodology): “Vendrov et al. (2015) used a Skip-Thought model proposed by Kiros et al. (2015) to the NLI task and reported an accuracy of 81.5% on the test data.”
Natural Language Inference over Interaction Space (2017, methodology): “…representation for each sentence and classiﬁes the relation by using the concatenation of two vector representation along with their absolute element-wise difference and element-wise product(Bowman et al., 2016; Vendrov et al., 2015; Mou et al., 2015; Liu et al., 2016; Munkhdalai & Yu, 2016).”
Natural Language Inference by Tree-Based Convolution and Heuristic Matching (2015, methodology): “Wan et al. (2015) build sentence pair models upon recurrent neural networks (RNNs) to iteratively integrate information along a sentence.”
Learning Language-Visual Embedding for Movie Understanding with Natural-Language (2016, methodology): “The contrastive terms for the minibatch are computed by using 199 contrastive videos for each caption and 199 contrastive captions for each video, similar to [24].”
The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision (2019, background): “…descriptive languages, such as image-captioning or visually-grounded question-answer pairs (Kiros et al., 2014; Shi et al., 2018; Mao et al., 2016; Vendrov et al., 2016; Ganju et al., 2017), dense language descriptions for scenes (Johnson et al., 2016), video-captioning (Donahue et al., 2015) and…”
Guided Open Vocabulary Image Captioning with Constrained Beam Search (2016, neutral): “While recent approaches to image caption generation have used template-based methods [15, 6], nearest neighbor retrieval [20, 16, 4], and order embeddings [27], a large body of work is dedicated to neural network approaches [5, 19, 13, 29].”