Programmatically Interpretable Reinforcement Learning

By Abhinav Verma, Vijayaraghavan Murali, Rishabh Singh, Pushmeet Kohli, Swarat Chaudhuri. 2018

417 citations recorded by the source corpus.

We present a reinforcement learning framework, called Programmatically Interpretable Reinforcement Learning (PIRL), that is designed to generate interpretable and verifiable agent policies. Unlike the popular Deep Reinforcement Learning (DRL) paradigm, which represents policies by neural networks, PIRL represents policies using a high-level, domain-specific programming language. Such programmatic policies have the benefits of being more easily interpreted than neural networks, and being amenable to verification by symbolic methods. We propose a new method, called Neurally Directed Program Search (NDPS), for solving the challenging nonsmooth optimization problem of finding a programmatic policy with maximal reward. NDPS works by first learning a neural policy network using DRL, and then performing a local search over programmatic policies that seeks to minimize a distance from this neural "oracle". We evaluate NDPS on the task of learning to drive a simulated car in the TORCS car-racing environment. We demonstrate that NDPS is able to discover human-readable policies that pass some significant performance bars. We also show that PIRL policies can have smoother trajectories, and can be more easily transferred to environments not encountered during training, than corresponding policies discovered by DRL.

Subfields: Artificial Intelligence, Reinforcement Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Safety, Robustness & Interpretability

Citations in the corpus

Interpretable Machine Learning: Fundamental Principles and 10 Grand Challenges (2021, background): “In PIRL (Verma et al., 2018), policies were represented by programming languages.”
Toward Interpretable-AI Policies Using Evolutionary Nonlinear Decision Trees for Discrete-Action Systems (2020, background): “Based on the state vector S t = ( x t , v t ) , the policy π( S ) selects one of the three actions A t : 1) decelerate ( A t = 0) along positive x -axis with a predeﬁned value − a ; 2) do nothing ( A t = 1); or 3) accelerate ( A t = 2) with + a in positive x -axis direction.”
Verifiable Reinforcement Learning via Policy Extraction (2018, background): “There has also been recent work using program synthesis to devise structured policies using imitation learning [35], but their focus is interpretability, and they are outperformed by DNNs even on cart-pole.”
Challenges of Real-World Reinforcement Learning (2019, background): “In the case of (Verma et al., 2018), evaluation is centered on looking at the performance of the programatically de-ﬁned policy, since the policy is inherently deﬁned as a human-readable program.”
Transparency in Deep Reinforcement Learning Networks (2018, background): “[39] who represent policies using a domain specific high level programming language.”
Towards Better Interpretability in Deep Q-Networks (2018, methodology): “More recently, (Verma et al. 2018) introduce a new method that finds interpretable programs that can best explain the policy learned by a neural network- these programs can also be treated as global explanations for the policy networks.”
Counterfactual States for Atari Agents via Generative Deep Learning (2019, background): “…work for explaining RL has looked at explaining policies through t-SNE embeddings [ Mnih and Hassabis, 2015; Khan et al. , 2009 ] , state abstractions [ Zahavy et al. , 2016 ] , human-interpretable predicates [ Hayes and Shah, 2017 ] and a high-level programming language [ Verma et al. , 2018 ] .”
Learning Interpretable Models with Causal Guarantees (2019, positive): “Finally, there has been a wide range of work using an uninterpretable oracle model f∗ to guide the learning of an interpretable model (Lakkaraju et al., 2017; Bastani et al., 2017; Verma et al., 2018; Frosst & Hinton, 2017; Bastani et al., 2018).”
Causal Analysis of Agent Behavior for AI Safety (2021, background): “Other recent approaches for analyzing AI systems have mostly focused on white-box approaches for improving understanding (see for instance Mott et al., 2019; Verma et al., 2018; Montavon et al., 2018; Puiutta & Veith, 2020) and developing safety guarantees (Uesato et al.”
Neurosymbolic Programming (2021, methodology): “In particular, the Pirl (Verma et al., 2018) and Propel (Verma et al., 2019) approaches for programmatic RL (mentioned in Section 3.3) consider formal verification of the learned program.”
Explainable Reinforcement Learning: A Survey (2020, neutral): “ each quadrant 4 and explain them in more detail in the following sections as an example for the diﬀerent XRL methods. 3.1 Method A: Programmatically Interpretable Reinforcement Learning Verma et al. [63] have developed ‘PIRL’, a Programmatically Interpretable Reinforcement Learning framework, as an”
Learning to Synthesize Programs as Interpretable and Generalizable Policies (2021, background): “[14, 15] distill neural network policies into programmatic policies.”
Mechanistic Interpretability for AI Safety - A Review (2024, background): “Techniques like creating programs in domain-specific languages (Verma et al., 2019b;a; Trivedi et al., 2021), extracting decision trees (Zhang et al., 2019) or symbolic causal graphs (Ren et al., 2023) from neural networks align well with the goals of mechanistic interpretability.”
Interpretable Multi-Objective Reinforcement Learning through Policy Orchestration (2018, methodology): “Although the policies themselves may not be directly interpretable (though there is recent work in this area (Verma et al. 2018; Liu et al. 2018)), our system does capture the notion of transparency and interpretability as we can see which policy is being applied in real time.”