Automation and Accountability in Decision Support System Interface Design

By M. Cummings. 2006

146 citations recorded by the source corpus.

When the human element is introduced into decision support system design, entirely new layers of social and ethical issues emerge but are not always recognized as such. This paper discusses those ethical and social impact issues specific to decision support systems and highlights areas that interface designers should consider during design with an emphasis on military applications. Because of the inherent complexity of socio-technical systems, decision support systems are particularly vulnerable to certain potential ethical pitfalls that encompass automation and accountability issues. If computer systems diminish a user’s sense of moral agency and responsibility, an erosion of accountability could result. In addition, these problems are exacerbated when an interface is perceived as a legitimate authority. I argue that when developing human computer interfaces for decision support systems that have the ability to harm people, the possibility exists that a moral buffer, a form of psychological distancing, is created which allows people to ethically distance themselves from their actions.

Subfields: Information Systems, Safety Research, Social Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

1. Yiling Chen
2. Supriyo Chakraborty
3. Mani B. Srivastava
4. Alun David Preece
5. Lance M. Kaplan
6. Madeleine Clare Elish
7. Madalina Busuioc
8. Jeroen van den Hoven

In Safety Research

1. Ben Green
2. Yiling Chen
3. Madalina Busuioc
4. Filippo Santoni de Sio
5. Alun David Preece
6. Madeleine Clare Elish
7. Jeroen van den Hoven
8. Chenhao Tan

In Social Psychology

1. Yiling Chen
2. Filippo Santoni de Sio
3. Ben Green
4. Madeleine Clare Elish
5. Madalina Busuioc
6. Jeroen van den Hoven

Citations in the corpus

• Rapid Trust Calibration through Interpretable and Uncertainty-Aware AI (2020, background): “Various studies have looked at different factors affecting automation bias, including the cognitive load of the user, 58 the accountability of the user in the decision process, 59,60 and their level of expertise and training.”
• Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction (2019, background): “Such issues have been looked at from the diverse perspectives of law and policy (Calo, Froomkin and Kerr 2016) human factors engineering (Cummings 2006; Sheridan and Parasuraman 2005), systems design (Friedman 1997; Leveson 2011), ethics (Lin, Abney, and Bekey 2014; Coeckelbergh 2011; Bryson,…”
• Meaningful Human Control over Autonomous Systems: A Philosophical Account (2018, background): “It has been argued that the tendency for human beings increasingly to depend on computer systems for their decision-making can lead to a reduced sense of responsibility for the consequences of those decisions (Cummings, 2006; Coeckelbergh, 2013; Saxon, 2016).”
• Disparate Interactions: An Algorithm-in-the-Loop Analysis of Fairness in Risk Assessments (2019, background): “Heavy reliance on automated systems can alter people’s relationship to a task by creating a “moral bu � er” between their decisions and the impacts of those decisions [11].”
• Accountable Artificial Intelligence: Holding Algorithms to Account (2020, background): “…reliance on algorithmic outputs could conceivably lead to unjustified deference of the decision-maker to algorithm predictions: “Automation bias” (Cummings 2006; Lyell and Coiera 2017; Skitka, Mosier, and Burdick 1999) is a well-documented phenomenon reported in psychological studies manifest in…”
• The Principles and Limits of Algorithm-in-the-Loop Decision Making (2019, unknown): “Prior work has found that using automated systems can generate a “moral bu er” that prompts people to feel less responsible and accountable for their actions [17].”