Hidden Incentives for Auto-Induced Distributional Shift

By David Krueger, Tegan Maharaj, Jan Leike. 2020

57 citations recorded by the source corpus.

Decisions made by machine learning systems have increasing influence on the world, yet it is common for machine learning algorithms to assume that no such influence exists. An example is the use of the i.i.d. assumption in content recommendation. In fact, the (choice of) content displayed can change users' perceptions and preferences, or even drive them away, causing a shift in the distribution of users. We introduce the term auto-induced distributional shift (ADS) to describe the phenomenon of an algorithm causing a change in the distribution of its own inputs. Our goal is to ensure that machine learning systems do not leverage ADS to increase performance when doing so could be undesirable. We demonstrate that changes to the learning algorithm, such as the introduction of meta-learning, can cause hidden incentives for auto-induced distributional shift (HI-ADS) to be revealed. To address this issue, we introduce `unit tests' and a mitigation strategy for HI-ADS, as well as a toy environment for modelling real-world issues with HI-ADS in content recommendation, where we demonstrate that strong meta-learners achieve gains in performance via ADS. We show meta-learning and Q-learning both sometimes fail unit tests, but pass when using our mitigation strategy.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Management Science and Operations Research, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Percy Liang
3. Mohit Bansal
4. Jacob Steinhardt
5. Pieter Abbeel
6. Chelsea Finn
7. Richard Ngo
8. Quoc V. Le

In Electrical and Electronic Engineering

1. Adrian Weller
2. Ryan J. Lowe
3. Jan Leike
4. Dan Hendrycks
5. John Schulman
6. Jacob Steinhardt

In Management Science and Operations Research

1. Dylan Hadfield-Menell
2. Anca Dragan
3. Umang Bhatt
4. Richard Ngo
5. Jan Leike
6. David Krueger
7. Claudia Shi
8. Nitarshan Rajkumar

In Safety, Robustness & Interpretability

1. Jacob Steinhardt
2. Dan Hendrycks
3. Richard Ngo
4. Nicholas Carlini
5. Dylan Hadfield-Menell
6. John Schulman
7. Umang Bhatt
8. Matthias Hein

Citations in the corpus

• Recursively Summarizing Books with Human Feedback (2021, methodology): “Auto-induced distributional shift (ADS, see Section 2.3.1) may also be making training significantly more difficult, and curriculum choice may matter a lot as a result.”
• Unsolved Problems in ML Safety (2021, background): “We also need more complex environments that can manifest diverse unintended side effects [192] such as feedback loops, which are a source of hazards to users of recommender systems [105].”
• Harms from Increasingly Agentic Algorithmic Systems (2023, background): “[111] show that long time-horizon systems, such as RLRS, will have incentives to change or manipulate users’ internal states (e.”
• The alignment problem from a deep learning perspective (2022, background): “The first additional reason is that training ML systems to interact with the real world often gives rise to feedback loops not captured by ML formalisms, which can incentivize behavior with larger-scale effects than developers intended [Krueger et al., 2020].”
• Objective Robustness in Deep Reinforcement Learning (2021, background): “As it trains, the agent is able to visit new states due to newly learned capabilities and thus effectively induces its own distributional shift [22].”
• Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “On the other hand, if reward and policy are learned jointly by gathering feedback from policy samples, the system will be prone to “auto-induced distributional shift” [Krueger et al., 2020, Carroll et al., 2022].”