Preventing undesirable behavior of intelligent machines

By Philip S. Thomas, Bruno Castro da Silva, Andrew Barto, Stephen Giguere, Yuriy Brun, Emma Brunskill. 2019

189 citations recorded by the source corpus.

Making well-behaved algorithms Machine learning algorithms are being used in an ever-increasing number of applications, and many of these applications affect quality of life. Yet such algorithms often exhibit undesirable behavior, from various types of bias to causing financial loss or delaying medical diagnoses. In standard machine learning approaches, the burden of avoiding this harmful behavior is placed on the user of the algorithm, who most often is not a computer scientist. Thomas et al. introduce a general framework for algorithm design in which this burden is shifted from the user to the designer of the algorithm. The researchers illustrate the benefits of their approach using examples in gender fairness and diabetes management. Science, this issue p. 999 A machine learning algorithm design framework shifts the burden of avoiding undesirable behavior from user to designer. Intelligent machines using machine learning algorithms are ubiquitous, ranging from simple data analysis and pattern recognition tools to complex systems that achieve superhuman performance on various tasks. Ensuring that they do not exhibit undesirable behavior—that they do not, for example, cause harm to humans—is therefore a pressing problem. We propose a general and flexible framework for designing machine learning algorithms. This framework simplifies the problem of specifying and regulating undesirable behavior. To show the viability of this framework, we used it to create machine learning algorithms that precluded the dangerous behavior caused by standard machine learning algorithms in our experiments. Our framework for designing machine learning algorithms simplifies the safe and responsible application of machine learning.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Industrial and Manufacturing Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Industrial and Manufacturing Engineering

Citations in the corpus

MOReL : Model-Based Offline Reinforcement Learning (2020, background): “This abstraction is particularly relevant for safety sensitive applications like healthcare and industrial automation where careful oversight by a domain expert is necessary for taking exploratory actions or deploying new policies [12, 13].”
Provably Good Batch Reinforcement Learning Without Great Exploration (2020, background): “However, such work either assumes that the policy class can be enumerated [38], which is infeasible in a number of important cases; or uses regularization with a behavior policy [37] as a heuristic, which can disallow significantly different but better policies.”
Interpretable Off-Policy Evaluation in Reinforcement Learning by Highlighting Influential Transitions (2020, background): “However, while these bounds are theoretically sound, they are often too conservative to be useful in practice (though see e.g. Thomas et al. (2019) for an exception).”
What are the Statistical Limits of Offline RL with Linear Function Approximation? (2020, background): “Also important is that in some safety-critical settings, we seek guarantees when oﬄine-trained policies can be eﬀective [Thomas, 2014, Thomas et al., 2019].”
Baldur: Whole-Proof Generation and Repair with Large Language Models (2023, background): “Probabilistic verification has successfully provided guarantees for such properties for machine learning systems [27, 57, 89, 96].”
Towards Safe Policy Improvement for Non-Stationary MDPs (2020, unknown): “To ensure safe policy improvement, we adapt the generic template of the Seldonian framework [61] to the non-stationary setting.”
The Real-World-Weight Cross-Entropy Loss Function: Modeling the Costs of Mislabeling (2020, methodology): “Techniques have been developed to reduce social bias in some neural networks designs such as adjusting the high dimensional vectors representing individual words to remove differences in the distance from the concepts of male and female in word2vec [11] and the Seldonian approach of describing and r”