Explainability for Large Language Models: A Survey
By Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Mengnan Du. ACM Transactions on Intelligent Systems and Technology, 2023
887 citations recorded by the source corpus.
Large language models (LLMs) have demonstrated impressive capabilities in natural language processing. However, their internal mechanisms are still unclear and this lack of transparency poses unwanted risks for downstream applications. Therefore, understanding and explaining these models is crucial for elucidating their behaviors, limitations, and social impacts. In this article, we introduce a taxonomy of explainability techniques and provide a structured overview of methods for explaining Transformer-based language models. We categorize techniques based on the training paradigms of LLMs: traditional fine-tuning-based paradigm and prompting-based paradigm. For each paradigm, we summarize the goals and dominant approaches for generating local explanations of individual predictions and global explanations of overall model knowledge. We also discuss metrics for evaluating generated explanations and discuss how explanations can be leveraged to debug models and improve performance. Lastly, we examine key challenges and emerging opportunities for explanation techniques in the era of LLMs in comparison to conventional deep learning models.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/26089bdfdbca1e6eaaceca71e3116b715bec6d47
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- Knowledge Conflicts for LLMs: A Survey (2024, positive): “Furthermore, LLMs tend to encode superficial associations rather than true comprehension of training data (Li et al., 2022b; Kang and Choi, 2023; Zhao et al., 2023a; Kandpal et al., 2023), leading to predetermined responses based on spurious correlations and potentially divergent answers for…”
- Large language models in healthcare and medical domain: A review (2023, background): “The potential of large language models (LLMs) in healthcare extends into the realms of explainable medical AI [104] and the utilization of multi-modal models incorporating sensor data.”
- A Comprehensive Overview of Large Language Models (2023, neutral): “This paper will serve as a valuable resource for researchers, offering insights into the recent advancements in LLMs and providing fundamental concepts and details to develop improved LLMs.”
- Multimodal Large Language Models in Health Care: Applications, Challenges, and Future Outlook (2024, neutral): “This challenge necessitates the development of advanced methods for explaining transformer-based language models [167,168], particularly methods for local explanations, such as feature attribution explanation, attention-based explanation, example-based explanation, and natural language explanation…”
- A Survey on the Memory Mechanism of Large Language Model-based Agents (2024, neutral): “AWM [179] proposes to generate workflows from previous trajectories by identifying reusable routines, subsequently incorporating these workflows into agent memory to direct future task-solving procedures.”
- Tool learning with large language models: a survey (2024, neutral): “Moreover, by inter-facing with project management and workflow tools, LLMs can aid users in managing tasks, monitoring progress, and optimizing work processes [18].”
- Large language models empowered agent-based modeling and simulation: a survey and perspectives (2023, positive): “Therefore, they propose to benchmark and continuously improve LLM agents ’ interpretability (Zhao et al., 2024).”
- KnowThyself: An Agentic Assistant for LLM Interpretability (2025, unknown): “However, their black-box nature makes it difficult to interpret internal decision processes, raising concerns about transparency, trust, and accountability [2, 3].”