Still a Pain in the Neck: Evaluating Text Representations on Lexical Composition

By Vered Shwartz, Ido Dagan. 2019

92 citations recorded by the source corpus.

Building meaningful phrase representations is challenging because phrase meanings are not simply the sum of their constituent meanings. Lexical composition can shift the meanings of the constituent words and introduce implicit information. We tested a broad range of textual representations for their capacity to address these issues. We found that, as expected, contextualized word representations perform better than static word embeddings, more so on detecting meaning shift than in recovering implicit information, in which their performance is still far from that of humans. Our evaluation suite, consisting of six tasks related to lexical composition effects, can serve future research aiming to improve representations.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

oLMpics-On What Language Model Pre-training Captures (2019, background): “revealing that they encode substantial amounts of syntax and semantics (Linzen et al., 2016b; Tenney et al., 2019b,a; Shwartz and Dagan, 2019; Lin et al., 2019; Coenen et al., 2019).”
Diversify Your Datasets: Analyzing Generalization via Controlled Variance in Adversarial Datasets (2019, background): “Specifically, several works showed that they capture various types of linguistic knowledge, from syntactic to semantic and discourse relations (e.g. Peters et al., 2018a; Tenney et al., 2019; Shwartz and Dagan, 2019).”
SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding (2022, background): “However, as these embeddings encode contextual information, supervised approaches using these representations tend to obtain better results in different tasks dealing with (non-)compositional semantics (Shwartz and Dagan, 2019; Fakharian and Cook, 2021; Zeng and Bhat, 2021).”
On the Systematicity of Probing Contextualized Word Representations: The Case of Hypernymy in BERT (2020, methodology): “We follow the work on diagnostic classiﬁers (Shi et al., 2016; Adi et al., 2017; Conneau et al., 2018; Hupkes et al., 2018; Liu et al., 2019; Shwartz and Dagan, 2019) and construct minimal embed-interact-predict probes to assess taxonomic knowledge in pretrained representations.”