Prolegomena to any future artificial moral agent

By C. Allen, G. Varner, Jason Zinser. Journal of experimental and theoretical artificial intelligence (Print), 2000

324 citations recorded by the source corpus.

As artificial intelligence moves ever closer to the goal of producing fully autonomous agents, the question of how to design and implement an artificial moral agent (AMA) becomes increasingly pressing. Robots possessing autonomous capacities to do things that are useful to humans will also have the capacity to do things that are harmful to humans and other sentient beings. Theoretical challenges to developing artificial moral agents result both from controversies among ethicists about moral theory itself, and from computational limits to the implementation of such theories. In this paper the ethical disputes are surveyed, the possibility of a ‘moral Turing Test’ is considered and the computational difficulties accompanying the different types of approach are assessed. Human-like performance, which is prone to include immoral actions, may not be acceptable in machines, but moral perfection may be computationally unattainable. The risks posed by autonomous machines ignorantly or deliberately harming people and other sentient beings are great. The development of machines with enough intelligence to assess the effects of their actions on sentient beings and act accordingly may ultimately be the most important task faced by the designers of artificially intelligent automata.

Subfields: Cognitive Neuroscience, Safety Research

View Paper: https://www.semanticscholar.org/paper/67ca442b1662171733d234e52b3c4af929619c45

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Safety Research

Citations in the corpus

A Conceptual and Computational Model of Moral Decision Making in Human and Artificial Agents (2010, background): “The possibility of building AI systems with moral decision making faculties has stepped beyond the stories of science fiction writers such as Isaac Asimov and is being seriously considered by philosophers and engineers, (Gips 1991, Clark 1994, Allen et al. 2000).”
Artificial Morality: Top-down, Bottom-up, and Hybrid Approaches (2005, background): “We pick up where Allen et al. (2000) left off when they wrote, ‘Essential to building a morally praiseworthy agent is the task of giving it enough intelligence to assess the effects of its actions on sentient beings, and to use those assessments to make appropriate choices’.”
Moral competence in social robots (2014, background): “, [10], [11]; for example, they do not include a deep, reflective self-concept, and they don’t presuppose “free will.”
On the Morality of Artificial Agents (2004, background): “The notion of a ‘transition system’ provides a convenient vehicle to support our criteria for agenthood, being general enough to embrace the usual notions like automaton and process.”
Ethics of artificial intelligence and robotics (2020, methodology): “Several authors use ‘artificial moral agent’ in a less demanding sense, borrowing from the software ‘agent’ use (Allen, Varner, and Zinser 2000), in which case matters of responsibility and rights will not arise.”
Social choice ethics in artificial intelligence (2017, background): “The other line of thinking is the concept of ‘‘bottom–up’’ ethics (Allen et al. 2000; 2005; Wallach et al. 2008; Wallach and Allen 2008).”
Sacrifice One For the Good of Many? People Apply Different Moral Norms to Human and Robot Agents (2015, background): “Since Allen and colleagues [17] offered a prescient discussion of morality in artificial agents, there has been growing interest 1 in issues of ethics and social robotics [18]–[24].”
Against the moral Turing test: accountable design and the moral reasoning of autonomous systems (2016, background): “While the term can serve as shorthand for a wide range of ways to test how humans regard machines (as seen cinematically most recently in 2014’s ‘‘Ex Machina’’), the idea of a ‘‘moral’’ Turing test still leans on the premise and method of the original version (Allen et al. 2000).”