Categorizing Variants of Goodhart's Law

By David Manheim, Scott Garrabrant. 2018

183 citations recorded by the source corpus.

There are several distinct failure modes for overoptimization of systems on the basis of metrics. This occurs when a metric which can be used to improve a system is used to an extent that further optimization is ineffective or harmful, and is sometimes termed Goodhart's Law. This class of failure is often poorly understood, partly because terminology for discussing them is ambiguous, and partly because discussion using this ambiguous terminology ignores distinctions between different failure modes of this general type. This paper expands on an earlier discussion by Garrabrant, which notes there are "(at least) four different mechanisms" that relate to Goodhart's Law. This paper is intended to explore these mechanisms further, and specify more clearly how they occur. This discussion should be helpful in better understanding these types of failures in economic regulation, in public policy, in machine learning, and in Artificial Intelligence alignment. The importance of Goodhart effects depends on the amount of power directed towards optimizing the proxy, and so the increased optimization power offered by artificial intelligence makes it especially critical for that field.

Subfields: Statistics and Probability, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Statistics and Probability

In Safety, Robustness & Interpretability

Citations in the corpus

Scaling Laws for Reward Model Overoptimization (2022, methodology): “One useful taxonomy for various Goodhart effects is presented in Manheim and Garrabrant [2018], categorizing Goodhart’s Law into 4 categories: Regressional, Extremal, Causal, and Adversarial.”
Risks from Learned Optimization in Advanced Machine Learning Systems (2019, background): “”[22] In “Categorizing Variants of Goodhart’s Law,” Manheim and Garrabrant describe adversarial Goodhart as when “an agent with different goals than the regulator” causes a “collapse of the statistical relationship between a goal which the optimizer intends and the proxy used for that goal.”
Unsolved Problems in ML Safety (2021, background): “Since proxies “will tend to collapse once pressure is placed upon” them by optimizers [64, 127, 169], proxies can often be gamed.”
Low-N protein engineering with data-efficient deep learning (2020, methodology): “Here efforts to use high-throughput proxy assays for the endpoint in question may in fact generate worse candidates for later-stage development 13,15 by over-optimizing a biased metric 53 .”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “Applying strong optimization pressure for an imperfect proxy measure for a goal tends to cause poor performance on the underlying target goal [Hoskin, 1996, Manheim and Garrabrant, 2018, Gao et al., 2022].”
Aligning AI Optimization to Community Well-Being (2020, background): “Gaming Any measure that becomes a target will change meaning as people change their behavior, a very general problem that is sometimes known as Goodhart ’ s law (Manheim and Garrabrant 2018).”
Defining and Characterizing Reward Hacking (2022, unknown): “Scholars have also criticized the assumption that human values can be encoded as rewards (Dobbe et al., 2021), and challenged the use of metrics more broadly (O’Neil, 2016; Thomas and Uminsky, 2022), citing Goodhart’s Law (Manheim and Garrabrant, 2018; Goodhart, 1975).”
Disentangling Length from Quality in Direct Preference Optimization (2024, background): “However, prior works have demonstrated that reward models can be exploited (Casper et al., 2023; Gao et al., 2023) due to a Goodhart’s law effect (Clark and Amodei, 2016; Manheim and Garrabrant, 2019; Skalse et al., 2022; Lambert and Calandra, 2023).”
Embedded Agency (2019, background): “As described by Manheim and Garrabrant (2018), there are (at least) four mechanisms behind Goodhart’s law: regressional, causal, extremal, and adversarial.”
Consequences of Misaligned AI (2021, background): “Manheim and Garrabrant (2018) (25) discuss overoptimization in the context of Goodhart’s Law and provide four distinct mechanisms by which systems overoptimize for an objective.”
AGI Safety Literature Review (2018, background): “Manheim and Garrabrant (2018) categorize instances of Goodhart’s law.”
The alignment problem from a deep learning perspective (2022, background): “One underlying reason for this difﬁculty is Goodhart’s law: when a measure becomes a target, it ceases to be a good measure [Manheim and Garrabrant, 2018].”