Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI

By Richard J. Tomsett, Alun David Preece, Dave Braines, Federico Cerutti, Supriyo Chakraborty, Mani B. Srivastava, G. Pearson, Lance M. Kaplan. 2020

175 citations recorded by the source corpus.

Artificial intelligence (AI) systems hold great promise as decision-support tools, but we must be able to identify and understand their inevitable mistakes if they are to fulfill this potential. This is particularly true in domains where the decisions are high-stakes, such as law, medicine, and the military. In this Perspective, we describe the particular challenges for AI decision support posed in military coalition operations. These include having to deal with limited, low-quality data, which inevitably compromises AI performance. We suggest that these problems can be mitigated by taking steps that allow rapid trust calibration so that decision makers understand the AI system's limitations and likely failures and can calibrate their trust in its outputs appropriately. We propose that AI services can achieve this by being both interpretable and uncertainty-aware. Creating such AI systems poses various technical and human factors challenges. We review these challenges and recommend directions for future research.

Subfields: Artificial Intelligence, Surgery, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lesia Semenova
2. Adrian Weller
3. Chudi Zhong
4. Zhi Chen
5. Haiyang Huang
6. Q. Liao
7. Javier Antorán
8. Umang Bhatt

In Surgery

1. Q. Liao
2. Cynthia Rudin

In Safety, Robustness & Interpretability

1. Lesia Semenova
2. Zhi Chen
3. Adrian Weller
4. Umang Bhatt
5. Chudi Zhong
6. Haiyang Huang
7. Javier Antorán
8. Q. Liao

Citations in the corpus

• Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “Tomsett et al. (2020) discusses the importance of uncertainty quantification and interpretability, and Antoràn et al. (2020) discusses interpretable uncertainty estimation.”
• Trust in AI: progress, challenges, and future directions (2024, unknown): “In order to address this, Tomsett et al. (2020) explain the concept of trust calibration, which involves making AI systems interpretable and uncertainty-aware.”
• Uncertainty as a Form of Transparency: Measuring, Communicating, and Using Uncertainty (2020, background): “If well-calibrated and effectively communicated, uncertainty can help stakeholders understand when they should trust model predictions and help developers address fairness issues in their models [123, 138].”