Better Word Representations with Recursive Neural Networks for Morphology

By Thang Luong, Richard Socher, Christopher D. Manning. 2013

911 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Using the Output Embedding to Improve Language Models (2016, methodology): “We use: Simlex999 (Hill et al., 2016), Verb-143 (Baker et al., 2014), MEN (Bruni et al., 2014), Rare-Word (Luong et al., 2013) and MTurk-771 (Halawi et al., 2012).”
GloVe: Global Vectors for Word Representation (2014, methodology): “These include WordSim-353 (Finkelstein et al., 2001), MC (Miller and Charles, 1991), RG (Rubenstein and Goodenough, 1965), SCWS (Huang et al., 2012), and RW (Luong et al., 2013).”
Enriching Word Vectors with Subword Information (2016, methodology): “Recently, several works have proposed different composition functions to derive representations of words from morphemes (Lazaridou et al., 2013; Luong et al., 2013; Botha and Blunsom, 2014; Qiu et al., 2014).”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “For example, some datasets are dedicated for speciﬁc word classes such as verbs (Gerz et al., 2016) or rare words (Luong et al., 2013), or for evaluating compositional knowledge in sentence embeddings (Marelli et al., 2014).”
Learning Gender-Neutral Word Embeddings (2018, background): “Improvements have been made by leveraging semantic lexicons and morphology (Luong et al., 2013; Faruqui et al., 2014), disambiguating multiple senses (ˇSuster et al., 2016; Arora et al., 2018; Upadhyay et al., 2017), and modeling contextualized information by deep neural networks (Peters et al.,…”
ConceptNet 5.5: An Open Multilingual Graph of General Knowledge (2016, methodology): “In practice, recent word embeddings have instead been evaluated on simpler, synthetic analogy data sets (Mikolov et al. 2013), and have not usually been compared to existing non-embedding-based methods of solving analogies.”
Character-Aware Neural Language Models (2015, background): “Unlike previous works that utilize subword information via morphemes (Botha and Blunsom 2014; Luong et al. 2013), our model does not require morphological tagging as a pre-processing step.”
Word Translation Without Parallel Data (2017, background): “The embedding quality of rare words is generally not as good as the one of frequent words (Luong et al., 2013), and we observed that feeding the discriminator with rare words had a small, but not negligible negative impact.”
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech (2018, positive): “We used 13 benchmarks [30] to measure word similarity, including WS-353 [31], WS-353-REL [32], WS-353-SIM , MC-30 [33], RG-65 [34], Rare-Word [35], MEN [36], MTurk-287 [37], MTurk-771 [38], YP-130 [31], SimLex-999 [39], Verb-143 [40], and SimVerb-3500 [41].”
Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation (2015, methodology): “This idea has been explored in prior work by composing morphemes into representations of words (Luong et al., 2013; Botha and Blunsom, 2014; Soricut and Och, 2015).”
Advances in Pre-Training Distributed Word Representations (2017, methodology): “We also report very strong performance on the Rare Words dataset (Luong et al., 2013), again outperforming GloVe models by a large margin.”
A Character-level Decoder without Explicit Segmentation for Neural Machine Translation (2016, background): “, 2010) but also with finer units, such as subwords (Sennrich et al., 2015; Botha and Blunsom, 2014; Luong et al., 2013) as well as symbols resulting from compression/encoding (Chitnis and DeNero, 2015).”
Bayesian Neural Word Embedding (2016, methodology): “The word similarity evaluation includes several different datasets: WordSim353 [16], SCWS [19], Rare Words (RW) [20], MEN [21] and Sim-Lex999 [22].”
Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models (2016, critical): “Our work takes inspiration from (Luong et al., 2013) and (Li et al.”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “…and distributed, making them especially well-suited to capturing grammatical and semantic generalisations (see Mikolov et al., 2013; Luong et al., 2013; Pennington et al., 2014, inter alia). nns have also scored notable successes in sequential modelling using feedforward networks…”
Bilingual Word Embeddings for Phrase-Based Machine Translation (2013, background): “Recently, morphology is explored to learn better word representations through Recursive Neural Networks (Luong et al., 2013).”