Community Evaluation and Exchange of Word Vectors at wordvectors.org

By Manaal Faruqui, Chris Dyer. 2014

123 citations recorded by the source corpus.

Vector space word representations are useful for many natural language processing applications. The diversity of techniques for computing vector representations and the large number of evaluation benchmarks makes reliable comparison a tedious task both for researchers developing new vector space models and for those wishing to use them. We present a website and suite of offline tools that that facilitate evaluation of word vectors on standard lexical semantics benchmarks and permit exchange and archival by users who wish to find good vectors for their applications. The system is accessible at: www.wordvectors.org.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, methodology): “In addition, we also evaluate the quality of our learned individual word representations using standard benchmarks (Faruqui & Dyer, 2014; Tsvetkov et al., 2015).”
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech (2018, positive): “We used 13 benchmarks [30] to measure word similarity, including WS-353 [31], WS-353-REL [32], WS-353-SIM , MC-30 [33], RG-65 [34], Rare-Word [35], MEN [36], MTurk-287 [37], MTurk-771 [38], YP-130 [31], SimLex-999 [39], Verb-143 [40], and SimVerb-3500 [41].”
Word Representations via Gaussian Embedding (2014, methodology): “This is achieved by adding a hard constraint that the eigenvalues λi lie within the hypercube [m,M ]d for constants m and M . mI ≺ Σi ≺MI, ∀i For diagonal covariances, this simply involves either applying the min or max function to each element of the diagonal to keep it within the hypercube, Σii ←…”
A Survey of Cross-lingual Word Embedding Models (2017, methodology): “Faruqui and Dyer (2014) later applied CCA to project words from two languages into a shared embedding space.”
Learning Word Embeddings from Speech (2017, methodology): “We evaluated the vector representations learned by the proposed Seq2seq Audio2vec model on 13 different benchmarks [Faruqui and Dyer, 2014] that have been widely used to measure word similarity.”
Context Mover's Distance & Barycenters: Optimal transport of contexts for building representations (2018, methodology): “Nevertheless, Table 3, shows the results on standard WS tasks [81] for GloVe embeddings (with cosine similarity) and using our Context Mover’s Distance (CMD) on top.”
Interpreting Word-Level Hidden State Behaviour of Character-Level LSTM Language Models (2018, methodology): “We assessed these embeddings us-ing the 13 word similarity tasks of http://wordvectors.org (Faruqui and Dyer, 2014), achieving the results shown in Table 4.”
A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors (2018, methodology): “An overview of widely used datasets is given by Faruqui and Dyer (2014).”
Layer-Wise Analysis of a Self-Supervised Speech Representation Model (2021, methodology): “We also measure performance on two downstream tasks, word-discrimination [13] and semantic word similarity [14], using non-parametric predictors based on the cosine similarities between representations.”
Evaluating Word Embeddings Using a Representative Suite of Practical Tasks (2016, methodology): “A thorough such lexical semantics evaluation was created by Faruqui and Dyer (2014)1.”
Problems With Evaluation of Word Embeddings Using Word Similarity Tasks (2016, background): “Faruqui and Dyer (2014b), and Lu et al. (2015) evaluate the word embeddings exclusively on word similarity and word analogy tasks.”
Evaluation methods for unsupervised word embeddings (2015, background): “Faruqui and Dyer (2014) provide a website that allows the automatic evaluation of embeddings on a number of query inventories.”
Intrinsic Evaluation of Word Vectors Fails to Predict Extrinsic Performance (2016, unknown): “(Mikolov et al., 2013a; Mikolov et al., 2013b; Fernández et al., 2014).”
Word Embeddings and Gender Stereotypes in Swedish and English (2019, methodology): “A method based on word analogies was developed to measure the quality of word embeddings and this method is now the de facto standard method of evaluating embeddings together with various word similarity tasks [12].”
LINSPECTOR WEB: A Multilingual Probing Suite for Word Representations (2019, methodology): “Early works for evaluating word representations (Faruqui and Dyer, 2014a,b; Nayak et al., 2016) have mostly focused on English and used either the word similarity or a set of downstream tasks.”