Counter-fitting Word Vectors to Linguistic Constraints

By Nikola Mrksic, Diarmuid Ó Séaghdha, Blaise Thomson, Milica Gasic, Lina Maria Rojas-Barahona, Pei-hao Su, David Vandyke, Tsung-Hsien Wen, Susannah Young. 2016

520 citations recorded by the source corpus.

In this work, we present a novel counter-fitting method which injects antonymy and synonymy constraints into vector space representations in order to improve the vectors' capability for judging semantic similarity. Applying this method to publicly available pre-trained word vectors leads to a new state of the art performance on the SimLex-999 dataset. We also show how the method can be used to tailor the word vector space for the downstream task of dialogue state tracking, resulting in robust improvements across different dialogue domains.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Towards Open Intent Discovery for Conversational Text (2019, background): “Moreover, adding linguistic structure to existing models has been shown to improve their performance across a wide range of NLU tasks, such as word embedding [38], machine translation [4], named entity recognition [21], and semantic role labelling [16].”
ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translations (2017, methodology): “PPDB has been used to improve word embeddings (Faruqui et al., 2015; Mrkšić et al., 2016).”
Intent detection using semantically enriched word embeddings (2016, methodology): “In this work, we extend the word embedding enrichment in [9] by incorporating the enrichment mechanism via related words from Macmillan Dictionary, and build a bidirectional”
Unsupervised Post-processing of Word Vectors via Conceptor Negation (2018, background): “, synonym relations) on word vectors (Faruqui et al. 2015; Mrksic et al. 2016; 2017), where the linguistic constraints are extracted from an external linguistic knowledge base such as WordNet (Miller 1995) and PPDB (Pavlick et al.”
Certified Robustness to Adversarial Word Substitutions (2019, methodology): “Alzantot et al. (2018) define the neighbors N(w) of a word w as the n = 8 nearest neighbors of w in a “counter-fitted” word vector space where antonyms are far apart (Mrkšić et al., 2016).3 The neighbors must also lie within some Euclidean distance threshold.”
BERT: A Review of Applications in Natural Language Processing and Understanding (2021, methodology): “Text embeddings at the word level can be used here [54].”
Gender and Representation Bias in GPT-3 Generated Stories (2021, methodology): “For power, we take a different approach, because antonyms tend be close in semantic space (Mrkšić et al., 2016).”
A Survey on Data Augmentation for Text Classification (2021, unknown): “Counter-fitting is an approach that depicts word embeddings on the basis of a target function in a way that similarities between synonyms are rewarded and similarities between antonyms are sanctioned [74].”
BAE: BERT-based Adversarial Examples for Text Classification (2020, methodology): “As a baseline, we consider TextFooler (Jin et al., 2019) which performs synonym replacement using a ﬁxed word embedding space (Mrkˇsi´c et al., 2016).”
Finnish resources for evaluating language model semantics (2017, background): “More recently, it has been argued that the average correlation of one human rater with the average of all the other raters is a fairer measure for evaluating computational models performance compared to inter-rater agreement et al., 2016).”
AdvEntuRe: Adversarial Training for Textual Entailment with Knowledge-Guided Examples (2018, methodology): “, 2015) or counterfitted (Mrkšić et al., 2016) to such resources.”
Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment (2019, methodology): “To represent the words, we use word embeddings from (Mrkšić et al. 2016).”