Concrete Problems in AI Safety

By Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dandelion Mané. 2016

3,295 citations recorded by the source corpus.

Rapid progress in machine learning and artificial intelligence (AI) has brought increasing attention to the potential impacts of AI technologies on society. In this paper we discuss one such potential impact: the problem of accidents in machine learning systems, defined as unintended and harmful behavior that may emerge from poor design of real-world AI systems. We present a list of five practical research problems related to accident risk, categorized according to whether the problem originates from having the wrong objective function ("avoiding side effects" and "avoiding reward hacking"), an objective function that is too expensive to evaluate frequently ("scalable supervision"), or undesirable behavior during the learning process ("safe exploration" and "distributional shift"). We review previous work in these areas as well as suggesting research directions with a focus on relevance to cutting-edge AI systems. Finally, we consider the high-level question of how to think most productively about the safety of forward-looking applications of AI.

Subfields: Artificial Intelligence, Safety, Risk, Reliability and Quality, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Sergey Levine
  2. Yoshua Bengio
  3. Mohit Bansal
  4. Percy Liang
  5. Quoc V. Le
  6. Jianfeng Gao
  7. Pieter Abbeel
  8. Ruslan Salakhutdinov

In Safety, Risk, Reliability and Quality

  1. Yarin Gal
  2. Marcel A. J. van Gerven
  3. Gabrielle Ras
  4. James Sharp
  5. Xiaowei Huang
  6. Finale Doshi-Velez
  7. José Miguel Hernández-Lobato
  8. Bo Li

In Safety, Robustness & Interpretability

  1. Nicholas Carlini
  2. Matthias Hein
  3. Dan Hendrycks
  4. Nicolas Papernot
  5. Bo Li
  6. Tom Everitt
  7. Dawn Song
  8. Yixuan Li

Citations in the corpus