Word Embeddings through Hellinger PCA

By Rémi Lebret, Ronan Collobert. 2013

233 citations recorded by the source corpus.

Word embeddings resulting from neural language models have been shown to be a great asset for a large variety of NLP tasks. However, such architecture might be difficult and time-consuming to train. Instead, we propose to drastically simplify the word embeddings computation through a Hellinger PCA of the word co- occurence matrix. We compare those new word embeddings with some well-known embeddings on named entity recognition and movie review tasks and show that we can reach similar or even better performance. Although deep learning is not really necessary for generating good word embeddings, we show that it can provide an easy way to adapt embeddings to specific tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

GloVe: Global Vectors for Word Representation (2014, background): “More recently, a square root type transformation in the form of Hellinger PCA (HPCA) (Lebret and Collobert, 2014) has been suggested as an effective way of learning word representations.”
Neural Text Generation from Structured Data with Application to the Biography Domain (2016, background): “This representation has shown to be helpful for various applications (Lebret and Collobert, 2014).”
How to evaluate word embeddings? On importance of data efficiency and simple supervised tasks (2017, unknown): “Setup includes following “shallow” pretrained embeddings: GloVe (100 and 300 dimensions) [Pen-nington et al., 2014], Hellinger PCA (HPCA) [Le-bret and Collobert, 2014], PDC (100 and 300 dimensions) and HDC (300 dimensions) [Sun et al., 2015], Additionally following “deep” embeddings are evaluated: N”
Generating Text from Structured Data with Application to the Biography Domain (2016, background): “This representation has shown to be helpful for various applications (Lebret and Collobert, 2014).”
Multilingual Visual Sentiment Concept Matching (2016, methodology): “The preprocessed text of this corpus was obtained from [23].”
How to Generate a Good Word Embedding (2015, background): “In addition to the neural network approaches, another line of research regarding word embedding is based on the word-context matrix [35, 14], in which each row corresponds to a word and each column corresponds to a context.”
N-gram-Based Low-Dimensional Representation for Document Classification (2014, background): “Good word vector representations can be obtained very quickly with many different recent approaches (Mikolov et al., 2013b; Mnih & Kavukcuoglu, 2013; Lebret & Collobert, 2014; Pennington et al., 2014).”
Bidirectional LSTM-CRF for Clinical Concept Extraction (2016, methodology): “Given the promising performance of deep neural networks and the recent success of unsupervised word embeddings in general NLP tasks (Pennington et al., 2014a; Mikolov et al., 2013b; Lebret and Collobert, 2014), this paper sets to explore the use of a state-of-the-art deep sequential model initialize”
Iterative Refinement for Machine Translation (2016, methodology): “All weights were initialized randomly apart from the word embedding layers, which were pre-computed with Hellinger Principal Component Analysis (Lebret and Collobert, 2014) applied to the bilingual co-occurrence matrix constructed on the training set.”
Evaluation methods for unsupervised word embeddings (2015, methodology): “CBOW in general performed the best and random projection the worst (p-value 0.05 for all pairs except H-PCA and C&W in comparing un-normalized score differences for the ALL - FREQ condition with a randomized permutation test).”
Bloom-Eval: A Hierarchical Evaluation Benchmark for Automatic Survey Generation Based on Bloom's Taxonomy (2026, unknown): “P P The DS score is then calculated as a multi-metric fusion of three distances empirically validated in NLP contexts (Coronel, 2016; Lebret and Collobert, 2014; Ji et al., 2023): the Jensen-Shannon Diver-gence d JS , the Hellinger Distance d H , and the Total Variation Distance d TV .”