Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet
By Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, E. Ameisen, Andy Jones and 16 more. arXiv.org, 2026
690 citations recorded by the source corpus.
We demonstrate that sparse autoencoders can extract interpretable features from Claude 3 Sonnet, a production-scale language model, addressing the open question of whether dictionary learning methods scale beyond small transformers. We trained sparse autoencoders with up to 34 million features on the model's middle layer residual stream, using scaling laws to guide hyperparameter selection. The resulting features are multilingual and multimodal (generalizing to images despite text-only training), respond to both concrete instances and abstract discussions of concepts, and can be used to steer model behavior in ways consistent with their interpretations. We find features corresponding to famous entities and locations, as well as more abstract concepts like sarcasm or errors in code. We also identify features relevant to ways in which language models might cause harm--including features representing deception, power-seeking, sycophancy, and bias--and show that these causally influence model outputs when manipulated. Additionally, we conduct analyses of feature interpretability, geometry, and computational function. However, significant limitations remain: our suite of features is incomplete, and we lack rigorous methods for evaluating whether our features faithfully capture model computations.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, General Social Sciences
View Paper: https://www.semanticscholar.org/paper/1b6e16cc72aee038eb1879b8686b53776be3d51f
Citations in the corpus
- Sparse Prototype Code Underlies Classification and Prediction Across Modalities (2026, unknown): “A growing body of work in mechanistic interpretability addresses this challenge by extracting sparse, human-interpretable features from network activations [Elhage et al., 2022, Bricken et al., 2023, Huben et al., 2024, Templeton et al., 2026].”
- Latent Terms: Dense Retrievers Contain Trivially Extractable BM25-ready Zipfian Vocabularies (2026, unknown): “…neural activations into features that are more localized and interpretable than individual coordinates of the original representation (Cunning-ham et al., 2023), thus facilitating the process of interpreting otherwise “black-boxed” neural activations (Lieberum et al., 2024; Templeton et al., 2025).”
- Navigating the Concept Space of Language Models (2026, unknown): “Contemporary SAEs frequently contain hundreds of thousands to millions of latent dimensions (Templeton et al., 2024; Gao et al., 2025).”
- Dataset Featurization: Uncovering Natural Language Features through Unsupervised Data Reconstruction (2025, unknown): “While these descriptions can be refined through self-improvement loops (Pan et al., 2023; Gero et al., 2023), expert feedback (Templeton et al., 2024), or reconstruction-based optimization (Singh et al., 2024b), they remain limited to dataset-level insights without capturing properties of…”
- Understanding Generative AI Content with Embedding Models (2024, unknown): “Recent work has explored the use of sparse autoencoders to decompose the activations of a model into interpretable components [17–19].”