Contextual and Non-Contextual Word Embeddings: an in-depth Linguistic Investigation
By Alessio Miaschi, Felice Dell'Orletta. 2020
90 citations recorded by the source corpus.
In this paper we present a comparison between the linguistic knowledge encoded in the internal representations of a contextual Language Model (BERT) and a contextual-independent one (Word2vec). We use a wide set of probing tasks, each of which corresponds to a distinct sentence-level feature extracted from different levels of linguistic annotation. We show that, although BERT is capable of understanding the full context of each word in an input sequence, the implicit knowledge encoded in its aggregated sentence representations is still comparable to that of a contextual-independent model. We also find that BERT is able to encode sentence-level properties even within single-word embeddings, obtaining comparable or even superior results than those obtained with sentence representations.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
Citations in the corpus
- Exploring the Potential of Large Language Models (LLMs)in Learning on Graphs (2023, background): “We also consider non-contextualized shallow embeddings [41] including TF-IDF and Word2vec [23] as a comparison.”
- TurkEmbed: Turkish Embedding Model on Natural Language Inference & Sentence Text Similarity Tasks (2025, unknown): “This limitation makes non-contextual embeddings inadequate for advanced tasks like Natural Language Inference (NLI) and Semantic Textual Similarity (STS), which require contextual understanding [8].”