SemEval-2017 Task 2: Multilingual and Cross-lingual Semantic Word Similarity

By José Camacho-Collados, Mohammad Taher Pilehvar, Nigel Collier, Roberto Navigli. 2017

161 citations recorded by the source corpus.

This paper introduces a new task on Multilingual and Cross-lingual SemanticThis paper introduces a new task on Multilingual and Cross-lingual Semantic Word Similarity which measures the semantic similarity of word pairs within and across five languages: English, Farsi, German, Italian and Spanish. High quality datasets were manually curated for the five languages with high inter-annotator agreements (consistently in the 0.9 ballpark). These were used for semi-automatic construction of ten cross-lingual datasets. 17 teams participated in the task, submitting 24 systems in subtask 1 and 14 systems in subtask 2. Results show that systems that combine statistical knowledge from text corpora, in the form of word embeddings, and external knowledge from lexical resources are best performers in both subtasks. More information can be found on the task website: http://alt.qcri.org/semeval2017/task2/

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Word Translation Without Parallel Data (2017, methodology): “We use the SemEval 2017 competition data (Camacho-Collados et al. (2017)) which provides large, high-quality and well-balanced datasets composed of nominal pairs that are manually scored according to a well-deﬁned similarity scale.”
Examining Gender Bias in Languages with Grammatical Gender (2019, methodology): “Word Similarity We test the quality of the embeddings after mitigation on the SemEval 2017 word similarity task (Camacho-Collados et al., 2017) for monolingual embeddings.”
Unsupervised Multilingual Word Embeddings (2018, unknown): “All of the unsupervised methods outperform the supervised systems from SemEval-2017 on language pairs involving Farsi, which is not covered by the Europarl bitexts.”
Context-Aware Cross-Lingual Mapping (2019, methodology): “We also evaluated on the SemEval’17 cross-lingual word similarity task (Camacho-Collados et al., 2017), which is mea- sured using the average of Pearson and Spearman correlation coefficients against human judgements.”
From Word to Sense Embeddings: A Survey on Vector Representations of Meaning (2018, methodology): “Other similar approaches are those by Pilehvar and Collier (2017) and Goikoetxea, Soroa, and Agirre (2015), which analyze the structure of semantic networks via Personalized Page Rank (Haveliwala, 2002) for extending the coverage and quality of pre-trained word embeddings, respectively.”
L2F/INESC-ID at SemEval-2017 Tasks 1 and 2: Lexical and semantic features in word and textual similarity (2017, methodology): “We applied all metrics to the train, test and trial examples of the SICK corpus (Marelli et al., 2014) and train and test examples from previous Semantic Textual Similarity in SemEval, as compiled by Tan et al. (2015).”