Interpretable Policies for Reinforcement Learning by Genetic Programming

By Daniel Hein, Steffen Udluft, Thomas A. Runkler. 2017

156 citations recorded by the source corpus.

The search for interpretable reinforcement learning policies is of high academic and industrial interest. Especially for industrial systems, domain experts are more likely to deploy autonomously learned controllers if they are understandable and convenient to evaluate. Basic algebraic equations are supposed to meet these requirements, as long as they are restricted to an adequate complexity. Here we introduce the genetic programming for reinforcement learning (GPRL) approach based on model-based batch reinforcement learning and genetic programming, which autonomously learns policy equations from pre-existing default state-action trajectory samples. GPRL is compared to a straight-forward method which utilizes genetic programming for symbolic regression, yielding policies imitating an existing well-performing, but non-interpretable policy. Experiments on three reinforcement learning benchmarks, i.e., mountain car, cart-pole balancing, and industrial benchmark, demonstrate the superiority of our GPRL approach compared to the symbolic regression method. GPRL is capable of producing well-performing interpretable reinforcement learning policies from pre-existing default trajectory data.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Toward Interpretable-AI Policies Using Evolutionary Nonlinear Decision Trees for Discrete-Action Systems (2020, methodology): “In [17], genetic programming (GP) is used to obtain interpretable policies on control tasks involving continuous actions space through model-based policy learning.”
Reinforcement Learning in Healthcare: A Survey (2019, background): “…[123], selection of function approximators and its stability issue [226], consideration of safety in optimization criterion/exploration process [56] and robustness facing uncertainties [239], as well as interpretability derived from causal explanation and interpretable/verified policies [16, 80].”
Transparency in Deep Reinforcement Learning Networks (2018, methodology): “[14], genetic programming for reinforcement learning (GPRL) approach using a model-based batch reinforcement learning and genetic programming is used, which autonomously learns policy equations from pre-existing default state-action trajectory samples.”
Overcoming Model Bias for Robust Offline Deep Reinforcement Learning (2020, methodology): “…the dataset (Fujimoto, Meger, and Precup 2018; Kumar et al. 2019), however most other ofﬂine algorithms often implicitly assume that the data in the batch contains sufﬁcient exploration to solve the problem (Ernst et al. 2005; Depeweg et al. 2016; Hein et al. 2016; Hein, Udluft, and Runkler 2018).”