Evaluation of sentence embeddings in downstream and linguistic probing tasks

By Christian S. Perone, Roberto Silveira, Thomas S. Paula. 2018

160 citations recorded by the source corpus.

Despite the fast developmental pace of new sentence embedding methods, it is still challenging to find comprehensive evaluations of these different techniques. In the past years, we saw significant improvements in the field of sentence embeddings and especially towards the development of universal sentence encoders that could provide inductive transfer to a wide variety of downstream tasks. In this work, we perform a comprehensive evaluation of recent methods using a wide variety of downstream and linguistic feature probing tasks. We show that a simple approach using bag-of-words with a recently introduced language model for deep context-dependent word embeddings proved to yield better results in many tasks when compared to sentence encoders trained on entailment datasets. We also show, however, that we are still far away from a universal encoder that can perform consistently across several downstream tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Finding Generalizable Evidence by Learning to Convince Q&A Models (2019, methodology): “We compute the cosine similarity between the embeddings for two texts X and Y using: fastText(X,Y) = cos(BoWFT(X),BoWFT(Y)) This method has proven to be a strong baseline for evaluating the similarity between two texts (Perone et al., 2018).”
Latent Retrieval for Weakly Supervised Open Domain Question Answering (2019, methodology): “We experiment with unsupervised pooled representations from neural language models (LM), which has been shown to be state-of-the-art unsupervised representations (Perone et al., 2018).”
Contextual and Non-Contextual Word Embeddings: an in-depth Linguistic Investigation (2020, background): “…provided evidences that contextualized Neural Language Models (NLMs) are able to capture a wide range of linguistic phenomena (Adi et al., 2016; Perone et al., 2018; Ten-ney et al., 2019b) and even to organize this information in a hierarchical manner (Belinkov et al., 2017; Lin et al., 2019;…”
In Other News: a Bi-style Text-to-speech Model for Synthesizing Newscaster Voice with Limited Data (2019, background): “They are able to capture both semantic and syntactic relations between words (Perone et al., 2018).”
Multi-View Domain Adapted Sentence Embeddings for Low-Resource Unsupervised Duplicate Question Detection (2019, methodology): “For f G , we either use SIF-weighted averaged GloVe vectors (Section 3.2), or unweighted averaged ParaNMT 17 word and trigram vectors (Wieting and Gimpel, 2018), which are the current State of the Art on the unsupervised STS Benchmark test set (Ethayarajh, 2018).”
Context-Aware Cross-Lingual Mapping (2019, background): “Sentence embeddings can be obtained from ELMo by averaging the contextualized word embeddings (Perone et al., 2018).”
Efficient Few-Shot Learning Without Prompts (2022, unknown): “Perone et al. (2018) introduced the idea of using sentence embeddings for text classiﬁcation.”
Language Model Pre-training for Hierarchical Document Representations (2019, unknown): “Perone et al. (2018) have shown that ELM O pool produces strong sentence representations and outperforms many other sentence representation models including skip-thought vectors.”
DirectProbe: Studying Representations without Classifiers (2021, background): “…calls for training classiﬁers on them to predict linguistic properties such as syntax, or cognitive skills like numeracy (e.g. Kassner and Schütze, 2020; Perone et al., 2018; Yaghoobzadeh et al., 2019; Krasnowska-Kiera´s and Wróblewska, 2019; Wallace et al., 2019; Pruksachatkun et al., 2020).”
AISA: AI Safety Assistant Framework for Continuous Improvement of Highway Construction (2026, unknown): “A probe is a lightweight classification layer, here an MLP, trained on top of frozen embeddings to perform a target linguistic tasks [16].”