What if This Modified That? Syntactic Interventions with Counterfactual Embeddings

By Mycal Tucker, Peng Qian, Roger Levy. 2021

51 citations recorded by the source corpus.

Neural language models exhibit impressive performance on a variety of tasks, but their internal reasoning may be difficult to understand. Prior art aims to uncover meaningful properties within model representations via probes, but it is unclear how faithfully such probes portray information that the models actually use. To overcome such limitations, we propose a technique, inspired by causal analysis, for generating counterfactual embeddings within models. In experiments testing our technique, we produce evidence that suggests some BERT-based models use a tree-distance-like representation of syntax in downstream prediction tasks.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Neel Nanda
2. Andrew Lee
3. Martin Wattenberg
4. Been Kim
5. Michael R Douglas

Citations in the corpus

• Eliciting Latent Predictions from Transformers with the Tuned Lens (2023, unknown): “One common approach is to train classifiers to extract specific concepts from hidden states, like part-of-speech and syntactic structure (Hewitt and Manning, 2019; Tucker et al., 2021; Li et al., 2022).”
• Emergent Linear Representations in World Models of Self-Supervised Sequence Models (2023, background): “Intervention is commonly used to study causality (Giulianelli et al., 2018; Tucker et al., 2021), but have often reached mixed conclusions (Belinkov, 2022).”