Improving zero-shot learning by mitigating the hubness problem

By Georgiana Dinu, Marco Baroni. International Conference on Learning Representations, 2014

402 citations recorded by the source corpus.

The zero-shot paradigm exploits vector-based word representations extracted from text corpora with unsupervised methods to learn general mapping functions from other feature spaces onto word space, where the words associated to the nearest neighbours of the mapped vectors are used as their linguistic labels. We show that the neighbourhoods of the mapped elements are strongly polluted by hubs, vectors that tend to be near a high proportion of items, pushing their correct labels down the neighbour list. After illustrating the problem empirically, we propose a simple method to correct it by taking the proximity distribution of potential neighbours across many mapped vectors into account. We show that this correction leads to consistent improvements in realistic zero-shot experiments in the cross-lingual, image labeling and image retrieval domains.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/b9da09d4828902ab8cb02aceb5495dc6064b220b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Mitigating the Hubness Problem for Zero-Shot Learning of 3D Objects (2019, background): “In recent studies [11, 33, 44], the hubness problem in ZSL is investigated.”
Learning principled bilingual mappings of word embeddings while preserving monolingual invariance (2016, background): “Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2289–2294, Austin, Texas, November 1-5, 2016. c©2016 Association for Computational Linguistics”
Baselines and Test Data for Cross-Lingual Inference (2017, methodology): “We map the target language embeddings to English as Mikolov et al. (2013a), using the Dinu et al. (2014) implementation 8 and Wiktionary data.”
Is Attribute-Based Zero-Shot Learning an Ill-Posed Strategy? (2016, methodology): “When an exact match is not found, one can carry out a nearest neighbor search [7,9,15,17,20,21] or a maximum a posteriori estimation method [13,14].”
A Review of Generalized Zero-Shot Learning Methods (2020, background): “[49] observed that a large number of different map vectors are surrounded by many common items, in which the presence of such items causes problem in high-dimensional spaces.”
Bilingual Lexicon Induction through Unsupervised Machine Translation (2019, methodology): “, 2010a,b) refers to the phenomenon of a few points being the nearest neighbors of many other points in high-dimensional spaces, which has been reported to severely affect cross-lingual embedding mappings (Dinu et al., 2015).”
A Framework for the Computational Linguistic Analysis of Dehumanization (2020, methodology): “After training word2vec, we zero-center and normalize all embeddings to alleviate the hubness problem (Dinu et al., 2014).”
Knowledge Distillation for Bilingual Dictionary Induction (2017, methodology): “• The method Ridge is used in a number of prior work (Mikolov et al., 2013; Dinu et al., 2014; Vulic and Korhonen, 2016).”
Unsupervised Parallel Sentence Extraction with Parallel Segment Detection Helps Machine Translation (2019, methodology): “To measure similarity of words we use the co-sine similarity based Cross-Domain Similarity Local Scaling (CSLS) metric (Conneau et al., 2018) which aims to overcome the hubness problem of high dimensional spaces (Dinu et al., 2015).”
Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints (2017, background): “ well-known, high-quality word vector collections: a) The Common Crawl GloVe English vectors from Pennington et al. (2014); b) German vectors from Vulic´ and Korhonen (2016a); c) Italian vectors from Dinu et al. (2015); and d) Russian vectors from Kutuzov and Andreev (2015). In addition, for each of”
Learning a Deep Embedding Model for Zero-Shot Learning (2016, methodology): “Among them, the method in [7] relies on the modelling of the global distribution of test unseen data ranks w.”
Word Translation Without Parallel Data (2017, methodology): “l translation tool to alleviate this issue. We make these dictionaries publicly available as part of the MUSE library3. We report results on these bilingual dictionaries, as well on those released by Dinu et al. (2015) to allow for a direct comparison with previous approaches. For each language pair”
Loss in Translation: Learning Bilingual Word Mapping with a Retrieval Criterion (2018, background): “A common observation is that nearest neighbor search for bilingual lexicon inference suffers from the “hubness problem” (Doddington et al., 1998; Dinu et al., 2014).”
A Survey of Cross-lingual Word Embedding Models (2017, methodology): “(2015) LMMHL ΩMSE Matrix factorization Dinu et al. (2015) LSGNS ΩMSE Better neighbour retrieval Gouws et al. (2015) LSGNS ΩMSE X Sentence-level Vyas and Carpuat (2016) LGloVe ΩMSE Sparse matrix factorization Hauer et al.”
Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces (2018, methodology): “For deciding mutual nearest neighbors, we use the Cross-Domain Similarity Local Scaling proposed in [14] to mitigate the so-called hubness problem [39] (points tending to be nearest neighbors of many points in high-dimensional spaces).”