Learning Finite State Representations of Recurrent Policy Networks
By Anurag Koul, Sam Greydanus, Alan Fern. 2018
99 citations recorded by the source corpus.
Recurrent neural networks (RNNs) are an effective representation of control policies for a wide range of reinforcement and imitation learning problems. RNN policies, however, are particularly difficult to explain, understand, and analyze due to their use of continuous-valued memory vectors and observation features. In this paper, we introduce a new technique, Quantized Bottleneck Insertion, to learn finite representations of these vectors and features. The result is a quantized representation of the RNN that can be analyzed to improve our understanding of memory use and general behavior. We present results of this approach on synthetic environments and six Atari games. The resulting finite representations are surprisingly small in some cases, using as few as 3 discrete memory states and 10 observations for a perfect Pong policy. We also show that these finite policy representations lead to improved interpretability.
Subfields: Artificial Intelligence, Reinforcement Learning, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Joelle Pineau
- Anima Anandkumar
- Zachary Chase Lipton
- Kamyar Azizzadenesheli
- Mohit Bansal
- Sergey Levine
- Amy Zhang
- Yoshua Bengio
In Reinforcement Learning
- Anima Anandkumar
- Kamyar Azizzadenesheli
- Zachary Chase Lipton
- Joelle Pineau
- Amy Zhang
- Luis Villaseñor Pineda
- Been Kim
- Demis Hassabis
In Safety, Robustness & Interpretability
- Been Kim
- Thomas McGrath
- Joelle Pineau
- Ulrich Paquet
- Anima Anandkumar
- Adam Pearce
- Tommaso Furlanello
- Amy Zhang
Citations in the corpus
- Acquisition of chess knowledge in AlphaZero (2021, background): “An interesting approach to understanding temporally-extended behaviour in agents with recurrent neural networks is to extract finite-state models of an agent’s recurrent state [62].”
- Learning Causal State Representations of Partially Observable Environments (2019, methodology): “We evaluate our algorithm for approximate causal states reconstruction on a modification of the original VizDoom environment used in [25] as well as multiple GridWorld navigation tasks with partially observable states.”