SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity

By Daniela Gerz, Ivan Vulic, Felix Hill, Roi Reichart, Anna Korhonen. 2016

272 citations recorded by the source corpus.

Verbs play a critical role in the meaning of sentences, but these ubiquitous words have received little attention in recent distributional semantics research. We introduce SimVerb-3500, an evaluation resource that provides human ratings for the similarity of 3,500 verb pairs. SimVerb-3500 covers all normed verb types from the USF free-association database, providing at least three examples for every VerbNet class. This broad coverage facilitates detailed analyses of how syntactic and semantic phenomena together influence human understanding of verb meaning. Further, with significantly larger development and test sets than existing benchmarks, SimVerb-3500 enables more robust evaluation of representation learning architectures and promotes the development of methods tailored to verbs. We hope that SimVerb-3500 will enable a richer understanding of the diversity and complexity of verb semantics and guide the development of systems that can effectively represent and interpret this meaning.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, background): “For example, some datasets are dedicated for speciﬁc word classes such as verbs (Gerz et al., 2016) or rare words (Luong et al., 2013), or for evaluating compositional knowledge in sentence embeddings (Marelli et al., 2014).”
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech (2018, positive): “We used 13 benchmarks [30] to measure word similarity, including WS-353 [31], WS-353-REL [32], WS-353-SIM , MC-30 [33], RG-65 [34], Rare-Word [35], MEN [36], MTurk-287 [37], MTurk-771 [38], YP-130 [31], SimLex-999 [39], Verb-143 [40], and SimVerb-3500 [41].”
All-but-the-Top: Simple and Effective Postprocessing for Word Representations (2017, background): “, 2016) where the score measures “genuine” similarity; and lastly the SimVerb-3500 (SimVerb) dataset (Gerz et al., 2016), a newly released large dataset focusing on similarity of verbs.”
Learning Word Embeddings from Speech (2017, unknown): “…RG-65 [Rubenstein and Goodenough, 1965], Rare-Word [Luong et al., 2013], MEN [Bruni et al., 2012], MTurk-287 [Radinsky et al., 2011], MTurk-771 [Halawi et al., 2012], YP-130 [Yang and Powers, 2006], SimLex-999 [Hill et al., 2015], Verb-143 [Baker et al., 2014], and SimVerb-3500 [Gerz et al., 2016].”
Conceptor Debiasing of Word Representations Evaluated on WEAT (2019, background): “The word similarity sets are: the RG65 (Rubenstein and Goodenough, 1965), the WordSim-353 (WS) (Finkelstein et al., 2002), the rare-words (RW) (Luong et al., 2013), the MEN dataset (Bruni et al., 2014), the MTurk (Radinsky et al., 2011), the SimLex-999 (SimLex) (Hill et al., 2015), and the SimVerb-3”
Distributional Models of Word Meaning (2018, background): “4), and are often outperformed by nondistributional similarity measures (Gerz et al. 2016).”
Evaluation by Association: A Systematic Study of Quantitative Word Association Evaluation (2017, background): “The construction of ground truth evaluation sets that reﬂect these relations, such as SimLex-999 (Hill et al., 2015), SimVerb-3500 (Gerz et al., 2016), MEN (Bruni et al., 2014) or Rare Words (Luong et al., 2013), relies on manually constructed guidelines that trigger subjective human interpretation…”
The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling (2020, unknown): “…and Charles, 1991), rg-65 Rubenstein and Goodenough (1965), Rare-Word (or rw) (Luong et al., 2013), simLex999 (Hill et al., 2015), simverb-3500 (Gerz et al., 2016), verb-143 (Baker et al., 2014) , YP-130 Yang and Powers (2006) and the relatedness-based datasets include MEN (Bruni et al.,…”
Semantic Specialisation of Distributional Word Vector Spaces using Monolingual and Cross-Lingual Constraints (2017, methodology): “For English vectors, we also report Spearman’s correlation with SimVerb-3500 (Gerz et al., 2016), a semantic similarity dataset that focuses on verb pair similarity.”
A Causal Inference Method for Reducing Gender Bias in Word Embedding Relations (2019, unknown): “2012), SimLex-999 (Hill, Reichart, and Korhonen 2015), and SimVerb-3500 (Gerz et al. 2016).”
Unsupervised Post-processing of Word Vectors via Conceptor Negation (2018, background): “We use the publicly available pre-trained Google News Word2Vec (Mikolov et al. 2013) Word similarity We test the performance of CN on seven benchmarks that have been widely used to measure word similarity: the RG65 (Rubenstein and Goodenough 1965), the WordSim-353 (WS) (Finkelstein et al. 2002), the”
Associative Multichannel Autoencoder for Multimodal Word Representation (2018, unknown): “…evaluation benchmarks, covering two different tasks: (i) Semantic relatedness: Men-3000 (Bruni et al., 2014) and Wordrel-252 (Agirre et al., 2009); (ii) Semantic similarity: Simlex-999 (Hill et al., 2016), Semsim-7576 (Silberer and Lap-ata, 2014), Wordsim-203 and Simverb-3500 (Gerz et al., 2016).”
All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality (2021, methodology): “We evaluate this using 4 word similarity/relatedness judgment datasets: RG65 (Ruben-stein and Goodenough, 1965), WS353 (Agirre et al., 2009), S IM L EX 999 (Hill et al., 2015) and S IM V ERB 3500 (Gerz et al., 2016).”
SemEval-2017 Task 2: Multilingual and Cross-lingual Semantic Word Similarity (2017, background): “Given that multiple large and high-quality verb similarity datasets have been created in recent years (Yang and Powers, 2006; Baker et al., 2014; Gerz et al., 2016), we decided to focus on nominal words.”
The Zero Resource Speech Challenge 2021: Spoken Language Modelling (2021, unknown): “We used a set of 13 existing semantic similarity and relatedness tests: WordSim-353 [10], WordSim-353-SIM [11], mc-30 [12], rg-65 [13], Rare-Word (or rw) [14], simLex999 [15], simverb-3500 [16], verb-143 [17] , YP-130 [10] and the relatedness-based”