Unsupervised Post-processing of Word Vectors via Conceptor Negation

By Tianlin Liu, Lyle Ungar, João Sedoc. 2018

22 citations recorded by the source corpus.

Word vectors are at the core of many natural language processing tasks. Recently, there has been interest in post-processing word vectors to enrich their semantic information. In this paper, we introduce a novel word vector post-processing technique based on matrix conceptors (Jaeger 2014), a family of regularized identity maps. More concretely, we propose to use conceptors to suppress those latent features of word vectors having high variances. The proposed method is purely unsupervised: it does not rely on any corpus or external linguistic database. We evaluate the post-processed word vectors on a battery of intrinsic lexical evaluation tasks, showing that the proposed method consistently outperforms existing state-of-the-art alternatives. We also show that post-processed word vectors can be used for the downstream natural language processing task of dialogue state tracking, yielding improved results in different dialogue domains.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. João Sedoc
2. Lyle Ungar
3. Tianlin Liu
4. S. Karve
5. Peter Christen

Citations in the corpus

• Continual Learning for Sentence Representations Using Conceptors (2019, methodology): “In this section, we briefly introduce matrix conceptors, drawing heavily on (Jaeger, 2017; He and Jaeger, 2018; Liu et al., 2019).”
• Conceptor Debiasing of Word Representations Evaluated on WEAT (2019, background): “2 To describe matrix conceptors, we draw heavily on (Jaeger, 2014; He and Jaeger, 2018; Liu et al., 2019b,a).”
• The Role of Protected Class Word Lists in Bias Identification of Contextualized Word Representations (2019, background): “As described in the orignal work on matrix conceptors (Jaeger, 2014; He and Jaeger, 2018; Liu et al., 2019b,a) C has a closed form solution: Intuitively, C is a soft projection matrix on the linear subspace that gives the largest shrinkage where the word embeddings Z have the highest variance.”