Fisher Vectors Derived from Hybrid Gaussian-Laplacian Mixture Models for Image Annotation

By Benjamin Klein, Guy Lev, Gil Sadeh, Lior Wolf. 2014

103 citations recorded by the source corpus.

In the traditional object recognition pipeline, descriptors are densely sampled over an image, pooled into a high dimensional non-linear representation and then passed to a classifier. In recent years, Fisher Vectors have proven empirically to be the leading representation for a large variety of applications. The Fisher Vector is typically taken as the gradients of the log-likelihood of descriptors, with respect to the parameters of a Gaussian Mixture Model (GMM). Motivated by the assumption that different distributions should be applied for different datasets, we present two other Mixture Models and derive their Expectation-Maximization and Fisher Vector expressions. The first is a Laplacian Mixture Model (LMM), which is based on the Laplacian distribution. The second Mixture Model presented is a Hybrid Gaussian-Laplacian Mixture Model (HGLMM) which is based on a weighted geometric mean of the Gaussian and Laplacian distribution. An interesting property of the Expectation-Maximization algorithm for the latter is that in the maximization step, each dimension in each component is chosen to be either a Gaussian or a Laplacian. Finally, by using the new Fisher Vectors derived from HGLMMs, we achieve state-of-the-art results for both the image annotation and the image search by a sentence tasks.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models (2015, methodology): “Consistent with Klein et al. (2014), we set the CCA output dimensionality to 4096.”
Learning Deep Structure-Preserving Image-Text Embeddings (2015, methodology): “In image-sentence retrieval experiments, to represent images, we follow the implementation details in [26, 37].”
Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding (2016, background): “Some of such embeddings are based on Canonical Correlation Analysis (Hardoon et al., 2004) e.g. (Gong et al., 2014; Klein et al., 2015; Plummer et al., 2015), linear models with ranking loss (Frome et al., 2013; Karpathy and Fei-Fei, 2015; Socher et al., 2014; Weston et al., 2011) or non-linear…”
Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question (2015, background): “In particular, many studies have made rapid progress on the task of image captioning [26, 15, 14, 40, 6, 8, 4, 19, 16, 42].”
Natural Language Object Retrieval (2015, methodology): “In image retrieval, a ranking function is learned through a recurrent neural network [24, 4], metric learning [13], correlation analysis [20] and other methods [6, 19].”
Learning Two-Branch Neural Networks for Image-Text Matching Tasks (2017, methodology): “For example, in bi-directional image-sentence search [1], [2], [3] one aims to retrieve the corresponding images given a sentence query, and vice versa.”
Probabilistic Embeddings for Cross-Modal Retrieval (2021, methodology): “Early works [11, 23] relied on Canonical Correlation Analysis (CCA) [13] to build joint embedding spaces.”
Learning Social Image Embedding with Deep Multimodal Attention Networks (2017, methodology): “Canonical Correlation Analysis (CCA) [11] and its kernel version (KCCA) [11] are the popular methods used in many works [9, 10, 14], which finds a projection to maximize the correlation between the vectors projected from different views.”
Word2VisualVec: Image and Video to Sentence Matching by Visual Feature Prediction (2016, background): “[16], and region KCCA by Plummer et al.”
Learning to Scale Multilingual Representations for Vision-Language Tasks (2020, methodology): “To address this trade-off between multilingual capacity and performance, we propose a Scalable Multilingual Aligned Language Representation (SMALR) model, which we demonstrate achieves strong task performance while also being highly compact compared to state-of-the-art word embedding methods [11,22,”
ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic (2021, methodology): “The first deep captioning methods applied RNNs to generate sequences of words [37, 48].”
Multimodal Machine Learning: A Survey and Taxonomy (2017, methodology): “CCA models have been used extensively for cross-modal retrieval [76], [106], [169] and audiovisual signal analysis [177], [187].”
Multi-cue Zero-Shot Learning with Strong Supervision (2016, background): “Similar embedding principles, often combined with recurrent neural networks [20] or a dependency parser [8], have recently been applied to image-to-text retrieval [30, 22, 24], language generation [42, 23, 10], and question answering about images [29, 18, 38].”