Representation Engineering: A Top-Down Approach to AI Transparency

By Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski and 11 more. arXiv.org, 2023

1,173 citations recorded by the source corpus.

In this paper, we identify and characterize the emerging area of representation engineering (RepE), an approach to enhancing the transparency of AI systems that draws on insights from cognitive neuroscience. RepE places population-level representations, rather than neurons or circuits, at the center of analysis, equipping us with novel methods for monitoring and manipulating high-level cognitive phenomena in deep neural networks (DNNs). We provide baselines and an initial analysis of RepE techniques, showing that they offer simple yet effective solutions for improving our understanding and control of large language models. We showcase how these methods can provide traction on a wide range of safety-relevant problems, including honesty, harmlessness, power-seeking, and more, demonstrating the promise of top-down transparency research. We hope that this work catalyzes further exploration of RepE and fosters advancements in the transparency and safety of AI systems.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering

View Paper: https://www.semanticscholar.org/paper/58fdf550600fc3873729d466601c5d08a51ba8a0

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

Citations in the corpus

Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (2024, neutral): “While our study provides critical information about widely deployed AI tools in legal practice, it comes with certain limitations.”
The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning (2024, methodology): “Inspired by representation engineering (Zou et al., 2023a), RMU perturbs model activations on hazardous data while preserving model activations on benign data (Section 4).”
Steering Llama 2 via Contrastive Activation Addition (2023, positive): “Li et al. (2023) employ linear probes to predict truthfulness on a contrastive question-answering dataset to identify as sparse sets of “truthful” attention heads.”
Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (2024, neutral): “On the other hand, prior methods for analyzing behaviors in terms of fine-grained units (Kim et al., 2018; Belinkov, 2022; Geiger et al., 2023; Zou et al., 2023) attempt to fit model internals to researcher-specified mechanistic hypotheses using researcher-curated data.”
Refusal in Language Models Is Mediated by a Single Direction (2024, positive): “Extracting feature directions from contrastive pairs of inputs is an established technique (Burns et al., 2022; Panickssery et al., 2023; Zou et al., 2023a).”
Safety Alignment Should Be Made More Than Just a Few Tokens Deep (2024, positive): “…that aligned models can still be made to respond to harmful requests via adversarially optimized inputs (Qi et al., 2023a; Carlini et al., 2023; Zou et al., 2023a; Chao et al., 2023; Andriushchenko et al., 2024), a few gradient steps of fine-tuning (Qi et al., 2023c; Zhan et al., 2023), or…”
Many-shot Jailbreaking (2024, neutral): “The effectiveness of the attack takes a hit on most domains, but improves steadily as a function of number of demonstrations.”
Improving Alignment and Robustness with Circuit Breakers (2024, positive): “This includes research ranging from uncovering emergent interpretable structures in intermediate representations [77, 46, 10], to the identification and modification of embedded knowledge [48, 42, 43], as well as steering model outputs [66, 7, 33, 24, 65].”