What Context Features Can Transformer Language Models Use?

By J. O’Connor, Jacob Andreas. 2021

86 citations recorded by the source corpus.

Transformer-based language models benefit from conditioning on contexts of hundreds to thousands of previous tokens. What aspects of these contexts contribute to accurate model prediction? We describe a series of experiments that measure usable information by selectively ablating lexical and structural information in transformer language models trained on English Wikipedia. In both mid- and long-range contexts, we find that several extremely destructive context manipulations—including shuffling word order within sentences and deleting all words other than nouns—remove less than 15% of the usable information. Our results suggest that long contexts, but not their detailed syntactic and propositional content, are important for the low perplexity of current transformer language models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Representation & Self-Supervised Learning

Citations in the corpus

Understanding Dataset Difficulty with V-Usable Information (2021, positive): “Such transformations were applied by O’Connor & Andreas (2021) to understand what syntactic features Transformers use in next-token prediction; we take this a step further, aiming to discover annotation artefacts, compare individual instances, and ultimately understand the dataset itself.”
Can language models learn from explanations in context? (2022, background): “First, we ensured that the beneﬁts of explanations are not due simply to word-level features, as smaller unsupervised text models have been found to ignore word order in some contexts (e.g., Pham et al., 2020; O’Connor and Andreas, 2021, but cf. Pa-padimitriou et al., 2022).”
AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts (2021, background): “They tend to favor certain prompt formats, paraphrases [45, 51], or even certain information in the input.”
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “10 in Appendix B, this recency bias is a phenomenon that has been previously documented in LLMs (O’Connor and Andreas, 2021).”
Do Prompt-Based Models Really Understand the Meaning of Their Prompts? (2021, neutral): “Our main argument throughout the paper shares the same logic as a recent line of studies (Sinha et al., 2021; O’Connor and Andreas, 2021; Pham et al., 2021; Gupta et al., 2021) which argue that the fact that LMs achieve good performance under ideal conditions is insufﬁcient to establish language…”
Lost in the Middle: How Language Models Use Long Contexts (2023, background): “For example, Transformer language models may have varying sensitivity to different linguistic features in their input (O’Connor and Andreas, 2021).”
Emergent Structures and Training Dynamics in Large Language Models (2022, methodology): “Similarly, O’Connor and Andreas (2021) use syntax-level perturbations and ablations to con-466 clude that the information in context windows most useful to language models are local ordering statis-468 tics and content words, e.g. nouns, verbs, adverbs, and adjectives.”
A Bayesian Framework for Information-Theoretic Probing (2021, methodology): “(5)Cross mutual information (XMI) has been used in several previous work such as (Pimentel et al., 2019, 2020b; Bugliarello et al., 2020; McAllester and Stratos, 2020; Torroba Hennigen et al., 2020; Fernandes et al., 2021; O’Connor and Andreas, 2021).”
When classifying arguments, BERT doesn’t care about word order...except when it matters (2022, background): “…that they are surprisingly invariant to scrambling word order (Sinha et al., 2021; Hessel and Schofield, 2021; Pham et al., 2020; Gupta et al., 2021; O’Connor and Andreas, 2021) and that grammatical knowledge like part of speech, often attributed to contextual embeddings, is actually also captured…”
Information-Theoretic Measures of Dataset Difficulty (2021, methodology): “In NLP, V - information has been used to study what context features Transformers actually use (O’Connor and Andreas, 2021), as well as to condition out information during probing (Hewitt et al., 2021), but not for estimating difﬁculty.”
When and why vision-language models behave like bags-of-words, and what to do about it? (2022, background): “Order information in language and vision: Several works have highlighted the lack of word order sensitivity in large language models (Hessel & Schoﬁeld, 2021; O’Connor & Andreas, 2021; Pham et al., 2021; Sinha et al., 2021).”