A Tale of a Probe and a Parser

By Rowan Hall Maudslay, Josef Valvoda, Tiago Pimentel, Adina Williams, Ryan Cotterell. 2020

63 citations recorded by the source corpus.

Measuring what linguistic information is encoded in neural models of language has become popular in NLP. Researchers approach this enterprise by training “probes”—supervised models designed to extract linguistic structure from another model’s output. One such probe is the structural probe (Hewitt and Manning, 2019), designed to quantify the extent to which syntactic information is encoded in contextualised word representations. The structural probe has a novel design, unattested in the parsing literature, the precise benefit of which is not immediately obvious. To explore whether syntactic probes would do better to make use of existing techniques, we compare the structural probe to a more traditional parser with an identical lightweight parameterisation. The parser outperforms structural probe on UUAS in seven of nine analysed languages, often by a substantial amount (e.g. by 11.1 points in English). Under a second less common metric, however, there is the opposite trend—the structural probe outperforms the parser. This begs the question: which metric should we prefer?

Subfields: Artificial Intelligence, Computational Linguistics & Semantics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Linguistics & Semantics

Citations in the corpus

Probing Classifiers: Promises, Shortcomings, and Advances (2021, background): “In particular, they randomize the labels in D P , creating a new keeping track of it, it should be possible to extract this information from its internal state space” (Hupkes, Veldhoen, and Zuidema 2018).”
Probing Classifiers: Promises, Shortcomings, and Alternatives (2021, background): “Some studies advocate for using simple probes, such as linear classiﬁers (Alain and Bengio 2016; Liu et al. 2019; Hewitt and Manning 2019; Hall Maudslay et al. 2020).”
Unsupervised Distillation of Syntactic Information from Contextualized Word Representations (2020, background): “This approach can introduce confounding between extracting information and learning it by the probe (Hewitt and Liang, 2019; Ravichander et al., 2020; Maudslay et al., 2020; Elazar et al., 2020).”
An Information Theoretic View on Selecting Linguistic Probes (2020, background): “Recently, various probing tasks have queried the representations of, e.g., contextualized word embeddings (Tenney et al., 2019a,b) and sentence embeddings (Linzen et al., 2016; Chen et al., 2019; Alt et al., 2020; Kassner and Sch¨utze, 2020; Maudslay et al., 2020; Chi et al., 2020).”
Finding Universal Grammatical Relations in Multilingual BERT (2020, unknown): “See Maudslay et al. (2020) for further discussion.”
What Happens To BERT Embeddings During Fine-tuning? (2020, unknown): “Note that Hall Maudslay et al. (2020) has recently raised concern about these metrics, but we follow the original method of Hewitt and Manning (2019) for the most comparable results.”
What if This Modified That? Syntactic Interventions with Counterfactual Embeddings (2021, methodology): “Probes, tools built to reveal properties of a trained model, are a favored approach (Hall Maudslay et al., 2020; Conneau et al., 2018).”
Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little (2021, background): “On the one hand, proponents of simple linear probes argue that having explicit, focused classifiers should provide better signal as the function itself is too weak on its own (Alain and Bengio, 2016; Hewitt and Manning, 2019; Maudslay et al., 2020).”
A Bayesian Framework for Information-Theoretic Probing (2021, background): “For this reason, many works limit themselves to studying only linearly encoded information (e.g. Alain and Bengio, 2016; Hewitt and Manning, 2019; Hall Maudslay et al., 2020) or a subset of neurons at a time (e.g. Torroba Hennigen et al., 2020; Mu and Andreas, 2020; Durrani et al., 2020).”
Pareto Probing: Trading-Off Accuracy and Complexity (2020, background): “Much work in probing advocates for the need for simple probes (Hewitt and Manning, 2019; Hall Maudslay et al., 2020).”