SimLex-999: Evaluating Semantic Models With (Genuine) Similarity Estimation

By Felix Hill, Roi Reichart, Anna Korhonen. 2014

1,352 citations recorded by the source corpus.

We present SimLex-999, a gold standard resource for evaluating distributional semantic models that improves on existing resources in several important ways. First, in contrast to gold standards such as WordSim-353 and MEN, it explicitly quantifies similarity rather than association or relatedness so that pairs of entities that are associated but not actually similar (Freud, psychology) have a low rating. We show that, via this focus on similarity, SimLex-999 incentivizes the development of models with a different, and arguably wider, range of applications than those which reflect conceptual association. Second, SimLex-999 contains a range of concrete and abstract adjective, noun, and verb pairs, together with an independent rating of concreteness and (free) association strength for each pair. This diversity enables fine-grained analyses of the performance of models on concepts of different types, and consequently greater insight into how architectures can be improved. Further, unlike existing gold standard evaluations, for which automatic approaches have reached or surpassed the inter-annotator agreement ceiling, state-of-the-art models perform well below this ceiling on SimLex-999. There is therefore plenty of scope for SimLex-999 to quantify future improvements to distributional semantic models, guiding the development of the next generation of representation-learning architectures.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Using the Output Embedding to Improve Language Models (2016, positive): “We use: Simlex999 (Hill et al., 2016), Verb-143 (Baker et al., 2014), MEN (Bruni et al., 2014), Rare-Word (Luong et al., 2013) and MTurk-771 (Halawi et al., 2012).”
Learning Distributed Representations of Sentences from Unlabelled Data (2016, methodology): “Deeper, more complex models are preferable for representations to be used in supervised systems, but shallow log-bilinear models work best for building representation spaces that can be decoded with simple spatial distance metrics.”
WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations (2018, methodology): “…Similarity (SCWS) dataset (Huang et al., 2012) comprises 2003 word pairs and is analogous to standard word similarity datasets, such as RG-65 (Rubenstein and Goodenough, 1965) and Sim-Lex ( Hill et al., 2015), in which the task is to automatically estimate the semantic similarity of word pairs.”
SentEval: An Evaluation Toolkit for Universal Sentence Representations (2018, background): “On the one hand, people have measured performance on intrinsic evaluations, e.g. of human judgments of word or sentence similarity ratings (Agirre et al., 2012; Hill et al., 2016b) or of word associations (Vuli´c et al., 2017).”
Language Models as Knowledge Bases? (2019, background): “Hill et al. (2015) investigate to what degree word representations capture semantic meaning as measured by similarity between word pairs.”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “Datasets containing such similarity scores are often used to evaluate word embeddings (Finkelstein et al., 2002; Bruni et al., 2012; Hill et al., 2015, inter alia) or sentence embeddings; see the many shared tasks on semantic textual similarity in Se-mEval (Cer et al., 2017, and previous editions).”
It’s All in the Name: Mitigating Gender Bias with Name-Based Counterfactual Data Substitution (2019, background): “The SimLex-999 dataset (Hill et al., 2015) provides a ground-truth measure of similarity produced by 500 native English speakers.”
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech (2018, positive): “We used 13 benchmarks [30] to measure word similarity, including WS-353 [31], WS-353-REL [32], WS-353-SIM , MC-30 [33], RG-65 [34], Rare-Word [35], MEN [36], MTurk-287 [37], MTurk-771 [38], YP-130 [31], SimLex-999 [39], Verb-143 [40], and SimVerb-3500 [41].”
NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing (2020, background): “SimLex-999 [17] is a larger and more challenging benchmark, consisting of similarity judgments for 999 word pairs.”
Charagram: Embedding Words and Sentences via Character n-grams (2016, methodology): “On multiple word similarity datasets, CHARAGRAM outperforms RNNs and CNNs, achieving state-ofthe-art performance on SimLex-999 (Hill et al., 2015).”
Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection (2020, methodology): “Additionally, we perform a semantic evaluation of the debiased embeddings using multiple word similarity datasets (e.g. SimLex-999 (Hill et al., 2015)).”
Commonsense Knowledge Base Completion (2016, methodology): “We use the GloVe (Penning-ton et al., 2014) embeddings trained on 840 billion tokens of Common Crawl web text and the P ARAGRAM -SimLex embeddings of Wieting et al. (2015), which were tuned to have strong performance on the SimLex-999 task (Hill et al., 2015).”
Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text (2020, methodology): “49 with human ratings from the Simlex-999 dataset [Hill et al., 2015].”
How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddings (2019, background): “These tasks cover semantic similarity, analogy solving, and concept categorization: SimLex999 (Hill et al., 2015), MEN (Bruni et al.”
Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation (2021, background): “The ultimate goal of automatic machine learning model evaluation is usually stated as capturing human judgements (Callison-Burch et al., 2006; Hill et al., 2015; Vedantam et al., 2015; Liu et al., 2016).”