SentEval: An Evaluation Toolkit for Universal Sentence Representations

By Alexis Conneau, Douwe Kiela. 2018

695 citations recorded by the source corpus.

We introduce SentEval, a toolkit for evaluating the quality of universal sentence representations. SentEval encompasses a variety of tasks, including binary and multi-class classification, natural language inference and sentence similarity. The set of tasks was selected based on what appears to be the community consensus regarding the appropriate evaluations for universal sentence representations. The toolkit comes with scripts to download and preprocess datasets, and an easy interface to evaluate sentence encoders. The aim is to provide a fairer, less cumbersome and more centralized way for evaluating sentence representations.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, background): “Many of the SentEval tasks are closely re-lated to sentiment analysis, such as MR (Pang and Lee, 2005), SST (Socher et al., 2013), CR (Hu and Liu, 2004), and SUBJ (Pang and Lee, 2004).”
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, methodology): “We constructed a sentence evaluation tool(2) called SentEval (Conneau and Kiela, 2018) to automate evaluation on all the tasks mentioned in this paper.”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “…et al., 2018; Yang et al., 2019; Dong et al., 2019], unlabeled datasets [Yang et al., 2019; Liu et al., 2019c; Zellers et al., 2019], benchmarks [Wang et al., 2019b, 2018; Conneau and Kiela, 2018], fine-tuning methods [Howard and Ruder, 2018; Houlsby et al., 2019; Peters et al., 2019], and more.”
Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, methodology): “’s implementation of InferSent (Conneau and Kiela, 2018) trained on SNLI.”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, background): “Other similarly-motivated benchmarks include SentEval (Conneau and Kiela, 2018), which evaluates fixed-size sentence embeddings, and DecaNLP (McCann et al.”
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, neutral): “, 2018) and SentEval (Conneau and Kiela, 2018).”
Do Better ImageNet Models Transfer Better? (2018, background): “More recent evaluation efforts have investigated transfer from modern CNNs to medical image datasets [58], and transfer of sentence embeddings to language tasks [12].”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, background): “The first approach is commonly used as a strong baseline for monolingual sentence embeddings (Arora et al., 2017; Conneau and Kiela, 2018; Gouews et al., 2014).”
AI and the Everything in the Whole Wide World Benchmark (2021, background): “1,2,3,5] as a main differentiator from predecessors’ benchmarks, SentEval [Conneau and Kiela, 2018] and decaNLP [McCann et al.”
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2023, neutral): “SentEval (Conneau and Kiela, 2018) and De-caNLP (McCann et al., 2018) also set benchmarks for evaluating models’ general capabilities.”
AdapterFusion: Non-Destructive Task Composition for Transfer Learning (2020, neutral): “BoolQ, SICK, CSQA, SST-2, CB, MRPC, RTE, which is in line with previous work (Phang et al., 2018; Conneau and Kiela, 2018; Reimers and Gurevych, 2019).”
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (2018, background): “…downstream tasks Figure 2 reports correlation between performance on our probing tasks and the downstream tasks available in the SentEval 5 suite (Conneau and Kiela, 2018), which consists of classiﬁcation (MR, CR, SUBJ, MPQA, SST2, SST5, TREC), natural language inference (SICK-E), semantic…”
CLUE: A Chinese Language Understanding Evaluation Benchmark (2020, methodology): “SentEval (Conneau and Kiela, 2018) addresses the above problem by introducing a standard evaluation pipeline on a set of popular sentence embedding evaluation datasets.”
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (2019, background): “SentEval (Conneau and Kiela, 2018) is a popular toolkit to evaluate the quality of sentence embed-dings.”
Dynabench: Rethinking Benchmarking in NLP (2021, methodology): “More recently, multi-task benchmarks such as SentE-val (Conneau and Kiela, 2018), DecaNLP (McCann et al., 2018), GLUE (Wang et al., 2018), and Super-GLUE (Wang et al., 2019) were proposed with the aim of measuring general progress across several tasks.”
What do you learn from context? Probing for sentence structure in contextualized word representations (2019, background): “…sentence encoders, such as InferSent (Conneau et al., 2017), speciﬁcally their ability to capture surface properties of sentences such as length, word content, and word order (Adi et al., 2017), as well as a broader set of syntactic features, such as tree depth and tense (Conneau et al., 2018).”
Continual Learning for Sentence Representations Using Conceptors (2019, background): “First, it is worthwhile to incorporate more benchmarks such as GLUE (Wang et al., 2019) and SentEval (Con-neau and Kiela, 2018) into the continual sentence representation task.”
What Does BERT Learn about the Structure of Language? (2019, methodology): “We use the SentEval toolkit (Conneau and Kiela, 2018) along with the recom-”