VALUATION : A N ADVERSARIAL APPROACH TO UNCOVER CATASTROPHIC FAILURES

By Jonathan Uesato, Ananya Kumar, Csaba Szepesvari, Tom Erez, Avraham Ruderman, Keith Anderson, Krishnamurthy (Dj) Dvijotham, Nicolas Manfred Otto Heess, Pushmeet Kohli. 2018

94 citations recorded by the source corpus.

This paper addresses the problem of evaluating learning systems in safety critical domains such as autonomous driving, where failures can have catastrophic consequences. We focus on two problems: searching for scenarios when learned agents fail and assessing their probability of failure. The standard method for agent evaluation in reinforcement learning, Vanilla Monte Carlo, can miss failures entirely, leading to the deployment of unsafe agents. We demonstrate this is an issue for current agents, where even matching the compute used for training is sometimes insufficient for evaluation. To address this shortcoming, we draw upon the rare event probability estimation literature and propose an adversarial evaluation approach. Our approach focuses evaluation on adversarially chosen situations, while still providing unbiased estimates of failure probabilities. The key difficulty is in identifying these adversarial situations -- since failures are rare there is little signal to drive optimization. To solve this we propose a continuation approach that learns failure modes in related but less robust agents. Our approach also allows reuse of data already collected for training the agent. We demonstrate the efficacy of adversarial evaluation on two standard domains: humanoid control and simulated driving. Experimental results show that our methods can find catastrophic failures and estimate failures rates of agents multiple orders of magnitude faster than standard evaluation schemes, in minutes to hours rather than days.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “To cover more general cases, we could train agents to explicitly discover weaknesses in the reward model and opportunities for reward hacking as well as the minimal perturbation that leads to an unacceptable outcome (Anonymous, 2019c).”
Learning to Give Checkable Answers with Prover-Verifier Games (2021, unknown): “…of models (such as robustness to adversarial perturbations) can be categorized under three groups [19]: those that test consistency with speciﬁcations [29, 32], those that train speciﬁcation consistent models [28, 36, 23, 7] and those that formally verify adherence to speciﬁcations [17, 7, 30].”
Machine Learning Testing: Survey, Landscapes and Horizons (2019, background): “[94] aimed to find catastrophic failures in safety-critical agents like autonomous driving in reinforcement learning.”
Finding and Visualizing Weaknesses of Deep Reinforcement Learning Agents (2019, methodology): “In our experiments we show that the typical methods fail in the case of RL networks and generate images far outside the manifold of valid game states, even with all typical forms of regularization.”
A Statistical Approach to Assessing Neural Network Robustness (2018, background): “For instance, concurrent work to ours (Uesato et al., 2019) takes a continuation approach to efficiently estimating the probability that an agent fails when this may be a rare event.”
Causal Analysis of Agent Behavior for AI Safety (2021, background): “, 2018; Puiutta & Veith, 2020) and developing safety guarantees (Uesato et al., 2018).”