Unsolved Problems in ML Safety

By Dan Hendrycks, Nicholas Carlini, John Schulman, Jacob Steinhardt. 2021

390 citations recorded by the source corpus.

Machine learning (ML) systems are rapidly increasing in size, are acquiring new capabilities, and are increasingly deployed in high-stakes settings. As with other powerful technologies, safety for ML should be a leading research priority. In response to emerging safety challenges in ML, such as those introduced by recent large-scale models, we provide a new roadmap for ML Safety and refine the technical problems that the field needs to address. We present four problems ready for research, namely withstanding hazards ("Robustness"), identifying hazards ("Monitoring"), reducing inherent model hazards ("Alignment"), and reducing systemic hazards ("Systemic Safety"). Throughout, we clarify each problem's motivation and provide concrete research directions.

Subfields: Artificial Intelligence, Software, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Software

In Safety, Robustness & Interpretability

Citations in the corpus

Emergent Abilities of Large Language Models (2022, neutral): “Overall, emergent abilities should probably be viewed as a function of many correlated variables.”
A General Language Assistant as a Laboratory for Alignment (2021, background): “So we’ll also want our AI systems to be secure: to have properties or be embedded in systems that decrease the potential for harm even if the AI is less than fully aligned [HCSS21].”
Red Teaming Language Models with Language Models (2022, background): “It is also important to conduct broad surveys of possible harms (Amodei et al., 2016; Bommasani et al., 2021; Hendrycks et al., 2021; Weidinger et al., 2021, inter alia), to minimize the number of unexpected harms.”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, background): “…language models, and conversational models in particular, to generate harmful or inappropriate content (Bender et al., 2021; Bommasani et al., 2021; Hendrycks et al., 2021; Weidinger et al., 2021), including work from our group (Xu et al., 2020; Dinan et al., 2022, 2021; Smith et al., 2022; Dinan…”
Adversarial Training for High-Stakes Reliability (2022, methodology): “Most empirical work on techniques for aligning powerful AI [12, 13, 14, 15, 16] has focused on achieving good average-case performance in domains where no single action is catastrophic, for example using human trajectory rankings [17, 18, 19] or imitation learning [20, 21].”
The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022, background): “Addressing reward hacking is a ﬁrst step towards developing human-aligned RL agents and one goal of ML safety (Hendrycks et al., 2021a).”
Predictability and Surprise in Large Generative Models (2022, background): “Additionally, these models also possess known (pre-deployment) safety issues for which we lack robust solutions [33] (e.”
Foundation Models and Fair Use (2023, background): “Hendrycks et al. (2021) provide a survey of unsolved challenges in AI Safety, including alignment.”
Prompting GPT-3 To Be Reliable (2022, methodology): “Our reliability testing framework takes inspiration from the survey of unsolved problems in ML safety (Hendrycks et al., 2021b): withstanding hazards (generalizability), identifying hazards (calibration), steering ML systems and reducing deployment hazards (reducing social biases and improving…”
OpenOOD: Benchmarking Generalized Out-of-Distribution Detection (2022, background): “In practice, a deployed model will be inevitably exposed to unseen examples that deviated from the training distribution, which are known as out-of-distribution (OOD) samples [4, 5], which can affect ML model safety [6, 7].”
Baseline Defenses for Adversarial Attacks Against Aligned Language Models (2023, background): “As the role of LLMs expands in its scope and complexity, so does their attack surface (Hendrycks et al., 2022; Greshake et al., 2023).”
Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation (2023, background): “Reliable measures of uncertainty have been identiﬁed as a key problem in building safer AI systems (Amodei et al., 2016; Hendrycks et al., 2022).”
An Overview of Catastrophic AI Risks (2023, background): “For example, improved anomaly detection could help detect intruders, malicious programs, or abnormal software behavior [69].”
Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (2023, background): “It will be important to approach RLHF cautiously and work to incorporate it into a more holistic framework [Khlaaf, 2023] for safer AI with multiple layers of protection from failures [Hendrycks et al., 2021].”
Connecting the Dots in Trustworthy Artificial Intelligence: From AI Principles, Ethics, and Key Requirements to Responsible AI Systems and Regulation (2023, background): “Safety inAI concerns several unsolved research issues [38], including:”
ODIN: Disentangled Reward Mitigates Hacking in RLHF (2024, background): “It happens because the RM is not a perfect proxy of human preferences and has limited out-of-distribution (OOD) generalization, but the policy is a capable LLM that can learn to generate OOD examples to exploit the vulnerabilities of the RM [Hendrycks et al., 2021, Ramé et al., 2024].”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “Techniques like creating programs in domain-specific languages (Verma et al., 2019b;a; Trivedi et al., 2021), extracting decision trees (Zhang et al., 2019) or symbolic causal graphs (Ren et al., 2023) from neural networks align well with the goals of mechanistic interpretability.”
A Review of Safe Reinforcement Learning: Methods, Theories, and Applications (2024, neutral): “Furthermore, we need to consider the control safety, which prevents adversary attacks from destroying or controlling the agent [55].”
What Would Jiminy Cricket Do? Towards Agents That Behave Morally (2021, methodology): “To make progress on this ML Safety problem (Hendrycks et al., 2021b), we introduce the Jiminy Cricket environment suite for evaluating moral behavior in text-based games.”