Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction

By Madeleine Clare Elish. Engaging Science, Technology, and Society, 2019

305 citations recorded by the source corpus.

As debates about the policy and ethical implications of AI systems grow, it will be increasingly important to accurately locate who is responsible when agency is distributed in a system and control over an action is mediated through time and space. Analyzing several high-profile accidents involving complex and automated socio-technical systems and the media coverage that surrounded them, I introduce the concept of a moral crumple zone to describe how responsibility for an action may be misattributed to a human actor who had limited control over the behavior of an automated or autonomous system. Just as the crumple zone in a car is designed to absorb the force of impact in a crash, the human in a highly complex and automated system may become simply a component—accidentally or intentionally—that bears the brunt of the moral and legal responsibilities when the overall system malfunctions. While the crumple zone in a car is meant to protect the human driver, the moral crumple zone protects the integrity of the technological system, at the expense of the nearest human operator. The concept is both a challenge to and an opportunity for the design and regulation of human-robot systems. At stake in articulating moral crumple zones is not only the misattribution of responsibility but also the ways in which new forms of consumer and worker harm may develop in new complex, automated, or purported autonomous technologies.

Subfields: Cognitive Neuroscience, Safety Research, Social Psychology

View Paper: https://www.semanticscholar.org/paper/4fc0d366d9ffd491d5955e7252edc4bf5f274291

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Safety Research

In Social Psychology

Citations in the corpus

The Principles and Limits of Algorithm-in-the-Loop Decision Making (2019, background): “…of “moral crumple zones” that place undue responsibility on the human operators of computer systems rather than on the creators of those systems [25], the people developing algorithmic decision aids must feel responsibility and be accountable for how their design choices a ect the nal decision…”
Artificial intelligence policy in India: a framework for engaging the limits of data-driven decision-making (2018, background): “loop have a risk of simply turning into a moral crumple zone [66]?”
Four Responsibility Gaps with Artificial Intelligence: Why they Matter and How to Address them (2021, background): “…thereby avoiding two undesired results: first, scapegoating, i.e. agents being held culpable without having a fair capacity to avoid wrongdoing (Elish, 2019): in the example of the automated driving systems above, for instance, the drivers’ relevant technical and motivational capacities not…”
Machine learning, social learning and the governance of self-driving cars (2018, background): “In systems where machines and humans interact, users often become the ‘moral crumple zone’ (Elish, 2016) for technological failure.”
Pretrained AI Models: Performativity, Mobility, and Change (2019, background): “An alternative is an ethics of co-responsibility, where producers and lead users assume shared responsibility [27] for intended and unintended consequences, rather than producers being cast as a kind of moral crumple zone [25].”