Associating neural word embeddings with deep image representations using Fisher Vectors

By Benjamin Klein, Guy Lev, Gil Sadeh, Lior Wolf. 2015

302 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

An efficient framework for learning sentence representations (2018, methodology): “The purely supervised models are respectively from (Karpathy & Fei-Fei, 2015), (Klein et al., 2015), (Mao et al.”
Skip-Thought Vectors (2015, methodology): “Thus the method of [33] is a strong baseline to compare our sentence representations with.”
Self-Supervised MultiModal Versatile Networks (2020, background): “With larger datasets [48, 69, 71, 92, 96], many works have profited from learning such a joint visual-textual space [17, 19, 29, 30, 41, 53, 57, 63, 68, 84, 85, 87, 88, 93].”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, background): “HGLMM FV CCA [42] ImNet + K400 + YouCook2 4.”
HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, methodology): “Examples of tasks that still represent a significant challenge for current artificial systems include text-to-video retrieval [25, 32, 54, 55, 63], text-based action or event localization [15], video captioning [36, 61], and video question answering [51, 63].”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, background): “, cross-modal retrieval [18, 31, 38, 55], image captioning [23, 36], image classication [10, 12, 20] video summarization [4, 41].”
Fine-Grained Image Classification via Combining Vision and Language (2017, methodology): “Zhai et al. [24] propose to learn projection functions by the metric learning, and this method is further improved as Joint Representation Learning (JRL) [ 25] by adding other information such as semantic categories and semi-supervised information.”
ActBERT: Learning Global-Local Video-Text Representations (2020, neutral): “HGLMM [15] YouCook2 4.”
UniViLM: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generation (2020, methodology): “0 500 C+LSTM+SA (Klein et al., 2015) No 4.”
Stacked Cross Attention for Image-Text Matching (2018, background): “A rich line of studies have explored mapping whole images and full sentences to a common semantic vector space for image-text matching [22, 38, 2, 39, 23, 27, 44, 10, 33, 13, 9, 8, 11].”
Order-Embeddings of Images and Language (2015, methodology): “CNN ( i ) is the same image feature used by Klein et al. (2015): we rescale images to have smallest side 256 pixels, we take 224 × 224 crops from the corners, center, and their horizontal reﬂections, run the 10 crops through the 19-layer VGG network of Simonyan & Zisserman (2015) (weights…”
Localizing Moments in Video with Natural Language (2017, methodology): “Canonical correlation analysis (CCA) achieves competitive results for both natural language image [15] and object [24] retrieval tasks.”
Dual Attention Networks for Multimodal Reasoning and Matching (2016, methodology): “[8] apply canonical correlation analysis (CCA) to find embeddings that maximize the correlation between images and sentences, which is further improved by incorporating deep neural networks [14, 34].”
Multimodal Convolutional Neural Networks for Matching Image and Sentence (2015, methodology): “We compared our models with recently developed models on the performances of the bidirectional image and sentence retrieval, specifically DeViSE [6], SDT-RNN [32], DCCA [37], FV [21], STV [20], RTP [28], Deep Fragment [15], m-RNN [24, 25], MNLM [18], RVP [3], DVSA [14], NIC [36], and LRCN [5].”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, background): “CCA (FV HGLMM) [20] (same features) 7.”
Multi-modal Transformer for Video Retrieval (2020, neutral): “2 36 CCA [11] (rep.”
Learning Joint Embedding with Multimodal Cues for Cross-Modal Video-Text Retrieval (2018, background): “Joint visual-semantic embeddings [9, 15, 20, 26, 33] underpin the building of most cross-modal retrieval methods as they can bridge the gap between different modalities.”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, neutral): “The result on Youcook2 dataset R@1 ↑ R@5 ↑ R@10 ↑ S UPERVISED HGLMM(Klein et al., 2015) 4.6 14.3 21.6 Coot(Ging et al., 2020) 16.7 40.2 52.3 UniVL (FT-Joint)(Luo et al., 2020) 22.2 52.2 66.2 VideoCLIP (Fine-tuned) 32.2 62.6 75.0 Z ERO - SHOT Random 0.0 0.2 0.3 HowTo100M(Miech et al., 2019) 6.1 17.3 ”
Use What You Have: Video retrieval using representations from collaborative experts (2019, neutral): “9 50 CCA [31] (rep.”