Learning to Understand Phrases by Embedding the Dictionary

By Felix Hill, Kyunghyun Cho, Anna Korhonen, Yoshua Bengio. 2015

202 citations recorded by the source corpus.

Distributional models that learn rich semantic word representations are a success story of recent NLP research. However, developing models that learn useful representations of phrases and sentences has proved far harder. We propose using the definitions found in everyday dictionaries as a means of bridging this gap between lexical and phrasal semantics. Neural language embedding models can be effectively trained to map dictionary definitions (phrases) to (lexical) representations of the words defined by those definitions. We present two applications of these architectures: reverse dictionaries that return the name of a concept given a definition or description and general-knowledge crossword question answerers. On both tasks, neural language embedding models trained on definitions from a handful of freely-available lexical resources perform as well or better than existing commercial systems that rely on significant task-specific engineering. The results highlight the effectiveness of both neural embedding architectures and definition-based training for developing models that understand phrases and sentences.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features (2017, methodology): “(Hill et al., 2016a) propose a Sequential (Denoising) Au-toencoder, S(D)AE .”
Learning Distributed Representations of Texts and Entities from Knowledge Base (2017, methodology): “Methods capable of learning distributed representations of arbitrary-length texts (i.e., fixed-length continuous vectors that encode the semantics of texts), such as sentences and paragraphs, have recently attracted considerable attention (Le and Mikolov, 2014; Kiros et al., 2015; Li et al., 2015; W”
Zero-Shot Activity Recognition with Verb Attribute Induction (2017, methodology): “BGRU pretaining We pretrain the BGRU model on the Dictionary Challenge, a collection of 800,000 word-definition pairs obtained from Wordnik and Wikipedia articles (Hill et al., 2016); the objective is to obtain a word’s embedding given one of its definitions.”
Memory, Show the Way: Memory Based Few Shot Word Representation Learning (2018, background): “Hill et al. (2016) learn an embedding of a dictionary deﬁnition to match the pre-trained headword vector, while Weissenborn (2017) re-ﬁnes the word embeddings with explicit background knowledge from a commonsense knowledge base.”
emoji2vec: Learning Emoji Representations from their Description (2016, background): “In addition, our work relates to the work of Hill et al. (2016) who built word representations for words and concepts based on their description in a dictionary.”
Learning to Compute Word Embeddings On the Fly (2017, background): “tems. source of auxiliary data. In this respect, the approach presented here subsumes theirs, and can be seen as a generalization to other types of deﬁnitions. The closest to our work is the study by Hill et al. (2016), in which a network is trained to produce an embedding of a dictionary deﬁnition ”