Post-hoc Interpretability for Neural NLP: A Survey

By Andreas Madsen, Siva Reddy, A. P. Sarath Chandar. ACM Computing Surveys, 2021

321 citations recorded by the source corpus.

Neural networks for NLP are becoming increasingly complex and widespread, and there is a growing concern if these models are responsible to use. Explaining models helps to address the safety and ethical concerns and is essential for accountability. Interpretability serves to provide these explanations in terms that are understandable to humans. Additionally, post-hoc methods provide explanations after a model is learned and are generally model-agnostic. This survey provides a categorization of how recent post-hoc interpretability methods communicate explanations to humans, it discusses each method in-depth, and how they are validated, as the latter is often a common concern.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/eadb1e7da375939e25083ae3936c4f4ef1f2a719

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Nazneen Rajani
2. Q. Liao
3. Danqi Chen
4. Yejin Choi
5. Jesse Vig
6. Suchin Gururangan
7. Mohit Bansal
8. Sanjeev Arora

Citations in the corpus

• LESS: Selecting Influential Data for Targeted Instruction Tuning (2024, methodology): “Our formulation of influence has been used in identifying mislabeled examples (Pruthi et al., 2020), analyzing memorization effects (Feldman & Zhang, 2020), and deriving various inter-pretability insights (Madsen et al., 2022), but it does not permit counterfactual reasoning.”
• AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, methodology): “There are many types of techniques to generate feature attribution scores for neural NLP models, as summarized in several recent survey papers on explainability for NLP [39, 132, 135].”
• Universal and Independent: Multilingual Probing Framework for Exhaustive Model Interpretation and Evaluation (2022, background): “In the survey of post-hoc language model interpretation (Madsen et al., 2021), the linguistic information-based tasks fall into the groups of the highest abstraction and the top-informativeness of properties used.”
• Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022, background): “In an effort to mitigate the risks for harm in AI/ML systems, researchers and developers have proposed strategies to make systems more explainable, transparent, and contestable (e.g., [3, 24, 34, 41, 45, 47, 74]).”
• Trust in AI: progress, challenges, and future directions (2024, unknown): “Post-hoc interpretable methods refer to methods that utilize the power of complex black-box models in accurate predictions and try to add global or local interpretability to their decisions (Du et al., 2019; Madsen et al., 2021; Molnar et al., 2019; Peake and Wang, 2018).”