Ultradense Word Embeddings by Orthogonal Transformation

By Sascha Rothe, Sebastian Ebert, Hinrich Schütze. 2016

116 citations recorded by the source corpus.

Embeddings are generic representations that are useful for many NLP tasks. In this paper, we introduce DENSIFIER, a method that learns an orthogonal transformation of the embedding space that focuses the information relevant for a task in an ultradense subspace of a dimensionality that is smaller by a factor of 100 than the original space. We show that ultradense embeddings generated by DENSIFIER reach state of the art on a lexicon creation task in which words are annotated with three types of lexical information - sentiment, concreteness and frequency. On the SemEval2015 10B sentiment analysis task we show that no information is lost when the ultradense subspace is used, but training is an order of magnitude more efficient due to the compactness of the ultradense space.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

SemEval-2018 Task 1: Affect in Tweets (2018, methodology): “Word embeddings trained from Spanish tweets, such as the ones provided in (Rothe et al., 2016), were used as the basis for training deep learning models.”
Uncovering Divergent Linguistic Information in Word Embeddings with Lessons for Intrinsic and Extrinsic Evaluation (2018, background): “Rothe et al. (2016) propose an orthogonal transformation to concentrate the information relevant for a task in a lower dimensional subspace, and Rothe and Sch¨utze (2016) extend this work to decompose embeddings into four subspaces speciﬁcally capturing polarity, concreteness, frequency and…”
Word Emotion Induction for Multiple Languages as a Deep Multi-Task Learning Problem (2018, methodology): “Rothe et al. (2016) train an orthogonal matrix Q ∈ R n n ( n being the dimensionality of the word embeddings) such that applying Q to an embedding vector e i concentrates all the polarity information in its ﬁrst dimension such that the polarity of a word w i can be computed as where p = (1 , 0 , 0…”
Learning and Evaluating Emotion Lexicons for 91 Languages (2020, unknown): “More recent approaches focus heavily on word embeddings, either using semi-supervised graph-based approaches (Wang et al., 2016; Hamilton et al., 2016; Sedoc et al., 2017) or fully supervised methods (Rosenthal et al., 2015; Li et al., 2017; Rothe et al., 2016; Du and Zhang, 2016).”
Inducing Domain-Specific Sentiment Lexicons from Unlabeled Corpora (2016, methodology): “2 The SOCIALSENT package provides a benchmark toolkit for inducing sentiment lexicons, including implementations of previously published algorithms (Velikovich et al., 2010; Rothe et al., 2016), which are not otherwise publicly available.”