Quantilizers: A Safer Alternative to Maximizers for Limited Optimization

By Jessica Taylor. 2016

54 citations recorded by the source corpus.

Subfields: Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety, Robustness & Interpretability

Citations in the corpus

Artificial Intelligence, Values, and Alignment (2020, background): “For this reason, researchers at the Machine Intelligence Research Institute have developed the idea of ‘quantilizers’, which represents a way of programming AI that potentially renders an agent indifferent between a top tier of good outcomes (Taylor, 2016).”
Scaling Laws for Reward Model Overoptimization (2022, neutral): “Many approaches have been proposed for reducing overoptimization in general [Taylor, 2016, Everitt et al., 2017], as well as in RMs [Gleave and Irving, 2022], including within the ﬁeld of adversarial robustness [Chakraborty et al., 2018].”
Concrete Problems in AI Safety (2016, background): “The Machine Intelligence Research Institute has studied safety issues that may arise in very advanced AI [57, 56, 36, 154, 142], including a few mentioned above (e.”
The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022, background): “Blindly optimizing the proxy in these cases can lead to catastrophic failure (Zhuang & Hadﬁeld-Menell, 2020; Taylor, 2016).”
AI Safety Gridworlds (2017, methodology): “This method works by combining a “good-enough” reward level with randomization (Taylor, 2016).”
Reinforcement Learning with a Corrupted Reward Channel (2017, methodology): “Applied to RL and CRMDPs, this idea leads to quantilising agents [Taylor, 2016].”
Impossibility and Uncertainty Theorems in AI Value Alignment (or why your AGI should not have a utility function) (2018, background): “This has been termed a “satisficing” or “quantilized” decision rule (Taylor 2016).”
Avoiding Side Effects By Considering Future Tasks (2020, methodology): “Another uncertainty method is quantilization [23], which incorporates uncertainty by sampling from the top quantile of actions rather than taking the optimal action.”
The Problem with Metrics is a Fundamental Problem for AI (2020, unknown): “The issue of over optimizing single metrics is so worrisome that recent work (Taylor 2016) has suggested replacing direct optimization with “q-quantilizers” which practically attempt to replace offering the deterministic highest utility maximizer with random alternatives weighted by their utility.”
Penalizing Side Effects using Stepwise Relative Reachability (2018, background): “Quantilization [Taylor, 2016] incorporates uncertainty by taking actions from the top quantile of actions.”
Alignment for Advanced Machine Learning Systems (2020, background): “Taylor (2015) suggests an alternative, which she calls “quantilization.””
Embedded Agency (2019, background): “Ifyou have a probability distribution Q ( y ) such that the proxy X is only a boundedly bad approximation of Y on average, quantilization avoids extremal Goodhart by selecting randomly from Q ( y | x ≥ c ) for some threshold c (Taylor 2015).”
Defining and Characterizing Reward Hacking (2022, unknown): “Our notion of simplification bears a close resemblance to quantilization (Taylor, 2016).”
AGI Safety Literature Review (2018, methodology): “r when a target function is optimized in the extreme (such as turning the earth into paperclips). Quantilization can avoid over12 optimization under some assumptions (Everitt, Krakovna, et al., 2017; Taylor, 2016). Another more specic method is to \regularize" reward by the impact the policy is”