Task-Oriented Intrinsic Evaluation of Semantic Textual Similarity

By Nils Reimers, Philip Beyer, Iryna Gurevych. 2016

75 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, methodology): “14e.g., Reimers et al. (2016) report success using STS labels with alternative metrics such as normalized Cumulative Gain (nCG), normalized Discounted Cumulative Gain (nDCG) and F1 to more accurately predict performance on the downstream tasks: text reuse detection, binary classification of…”
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019, methodology): “We also provide the Pearson correlation r to make the results comparable to Misra et al. However, we showed (Reimers et al., 2016) that Pearson correlation has some serious drawbacks and should be avoided for comparing STS systems.”
SimCSE: Simple Contrastive Learning of Sentence Embeddings (2021, background): “Reimers et al. (2016) argue that Spearman correlation, which measures the rankings instead of the actual scores, better suits the need of evaluating sentence embeddings.”
Making Monolingual Sentence Embeddings Multilingual Using Knowledge Distillation (2020, methodology): “pute cosine similarity and, as recommended in (Reimers et al., 2016), compute the Spearman’s rank correlation ρ between the computed score and the gold score.”
SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding (2022, methodology): “We do not use Pearson correlation as it has been shown to be a poor indicator of performance on STS tasks (Reimers et al., 2016).”
Measuring Similarity of Opinion-bearing Sentences (2021, background): “However, Pearson correlation can be misleading because it is a measure of linear relationship, sensitive to outliers and requires the two variables to be approximately normally distributed (Reimers et al., 2016).”
MTEB: Massive Text Embedding Benchmark (2022, methodology): “Spearman correlation based on cosine similarity serves as the main metric (Reimers et al., 2016).”
An Unsupervised Sentence Embedding Method by Mutual Information Maximization (2020, methodology): “As pointed out in Reimers et al. (2016) that Pearson correlation is badly suited for STS, Spearman’s rank correlation between the cosine-similarity of the sentence embeddings and the gold labels is instead used as the evaluation metric.”
Length-Induced Embedding Collapse in PLM-based Models (2024, unknown): “Calculate the Spearman correlation based on cosine similarity (Reimers et al., 2016) as the main metric.”