Low-Complexity Probing via Finding Subnetworks

By Steven Cao, Victor Sanh, Alexander M. Rush. 2021

77 citations recorded by the source corpus.

The dominant approach in probing neural networks for linguistic properties is to train a new shallow multi-layer perceptron (MLP) on top of the model’s internal representations. This approach can detect properties encoded in the model, but at the cost of adding new parameters that may learn the task directly. We instead propose a subtractive pruning-based probe, where we find an existing subnetwork that performs the linguistic task of interest. Compared to an MLP, the subnetwork probe achieves both higher accuracy on pre-trained models and lower accuracy on random models, so it is both better at finding properties of interest and worse at learning on its own. Next, by varying the complexity of each probe, we show that subnetwork probing Pareto-dominates MLP probing in that it achieves higher accuracy given any budget of probe complexity. Finally, we analyze the resulting subnetworks across various tasks to locate where each task is encoded, and we find that lower-level tasks are captured in lower layers, reproducing similar findings in past work.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Parameter-Efficient Transfer Learning with Diff Pruning (2020, background): “…work on learning to mask parts of deep networks with differentiable relaxations of binary masks for model pruning and parameter sharing (Wang et al., 2019b; Zhao et al., 2020; Sanh et al., 2020; Radiya-Dixit & Wang, 2020; Mallya et al., 2018; Guo et al., 2019; Sun et al., 2020a; Cao et al., 2021).”
Towards Automated Circuit Discovery for Mechanistic Interpretability (2023, methodology): “To this end, we study the task-specific metrics on models with permuted weights (which we call reset networks (Zhang and Bowman, 2018; Cao, Sanh, and Rush, 2021)) and verify that the circuit recovery algorithms perform worse on these models that do not have underlying algorithms.”
Finding Neurons in a Haystack: Case Studies with Sparse Probing (2023, background): “…sparse probing is situated within the broader category of probing methods (Hennigen et al., 2020; Voita & Titov, 2020; Pimentel et al., 2020; Cao et al., 2021), which itself is just one paradigm among other localization techniques (Bau et al., 2018; Dhamdhere et al., 2018; Amjad et al.,…”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “This causal perspective on interpretability has led to the development of various intervention techniques, including activation patching (Section 4.2.1), causal abstraction (Section 4.2.2), and hypothesis testing methods (Section 4.2.3).”
Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task (2022, background): “In the following section, we adhere to Belinkov (2016)’s recommendation, performing a set of interventional experiments to determine the causal relationship between model predictions and the emergent world representations.”
Conditional probing: measuring usable information beyond a baseline (2021, background): “Pimentel et al. (2020a) argues for choosing multiple points along an axis of expressivity, while Cao et al. (2021) deﬁne the family through the weights of the neural network.”