Learning Word Representations with Hierarchical Sparse Coding

By Dani Yogatama, Manaal Faruqui, Chris Dyer, Noah A. Smith. 2014

61 citations recorded by the source corpus.

We propose a new method for learning word representations using hierarchical regularization in sparse coding inspired by the linguistic study of word meanings. We show an efficient learning algorithm based on stochastic proximal methods that is significantly faster than previous approaches, making it possible to perform hierarchical sparse coding on a corpus of billions of word tokens. Experiments on various benchmark tasks---word similarity ranking, analogies, sentence completion, and sentiment analysis---demonstrate that the method outperforms or is competitive with state-of-the-art methods. Our word representations are available at \url{this http URL}.

Subfields: Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings (2016, background): “, [23, 30, 40, 9]), and the difficulty of evaluating embedding quality (as compared to supervised learning) parallels the difficulty of defining bias in an embedding.”
Investigating Language Universal and Specific Properties in Word Embeddings (2016, background): “Yogatama et al. (2014) projects word embedding into a sparse vector.”
Sparse Overcomplete Word Vector Representations (2015, background): “…useful in NLP problems (Kazama and Tsujii, 2003; Friedman et al., 2008; Goodman, 2004), like POS tagging (Ganchev et al., 2009), dependency parsing (Mar-tins et al., 2011), text classiﬁcation (Yogatama and Smith, 2014), and representation learning (Ben-gio et al., 2013; Yogatama et al., 2015).”
Analyzing Linguistic Knowledge in Sequential Model of Sentence (2016, background): “Currently, there have been a few attempts (Yogatama et al., 2014; Köhn, 2015; Faruqui and Dyer, 2015) at understanding what is embedded in the word vectors or building linguistically interpretable embeddings.”
Machine Learning in Medicine (2015, methodology): “In addition to image recognition, sparse coding has been applied successfully to natural language processing.(34) We will later discuss if such approaches might be of use in patient classification for the purposes of precision medicine.”