Reinforcement Learning: A Survey

By Leslie Pack Kaelbling, Michael L. Littman, Andrew W. Moore. 1996

This paper surveys the field of reinforcement learning from a computer-science perspective. It is written to be accessible to researchers familiar with machine learning. Both the historical basis of the field and a broad selection of current work are summarized. Reinforcement learning is the problem faced by an agent that learns behavior through trial-and-error interactions with a dynamic environment. The work described here has a resemblance to work in psychology, but differs considerably in the details and in the use of the word "reinforcement." The paper discusses central issues of reinforcement learning, including trading off exploration and exploitation, establishing the foundations of the field via Markov decision theory, learning from delayed reinforcement, constructing empirical models to accelerate learning, making use of generalization and hierarchy, and coping with hidden state. It concludes with a survey of some implemented systems and an assessment of the practical utility of current methods for reinforcement learning.

Subfields: Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, neutral): “A large number of prior methods (Sutton, 1990; Kaelbling et al., 1996) have proposed to use such a policy distribution as an exploration strategy, but in the context of a reinforcement learning algorithm where the Q-function is learned via the standard (“hard”) max operator, corresponding to a tempe”
Addressing Function Approximation Error in Actor-Critic Methods (2018, background): “Note that overestimation due to function approximation is not a good measure of uncertainty, thus, using an over-estimating critic is not an effective way to implement optimism in the face of uncertainty (Kaelbling et al., 1996), and can lead to suboptimal policies (Thrun & Schwartz, 1993).”
Reinforcement Learning with Deep Energy-Based Policies (2017, background): “A closely related concept to maximum entropy policies is Boltzmann exploration, which uses the exponential of the standard Q-function as the probability of an action (Kaelbling et al., 1996).”
Recurrent World Models Facilitate Policy Evolution (2018, background): “For many reinforcement learning (RL) problems [37, 96, 106], an artiﬁcial RL agent may also beneﬁt from a predictive model (M) of the future [95, 104] (model-based RL).”
Deep Reinforcement Learning with Double Q-Learning (2015, background): “Furthermore, it is known that sometimes it is good to be optimistic: optimism in the face of uncertainty is a well-known exploration technique (Kaelbling et al., 1996).”
Neural Approaches to Conversational AI (2018, neutral): “t learning that is most relevant to the discussions in later chapters. For a comprehensive survey, interested readers are referred to excellent textbooks and reviews, such as Sutton and Barto (2018); Kaelbling et al. (1996); Bertsekas and Tsitsiklis (1996); Szepesvari (2010); Wiering and van Otterlo”
Toward Causal Representation Learning (2021, background): “Particularly relevant for causal leaning are generative world models that capture some of the causal relations underlying the environment and serve as Lorenzian imagined spaces (see I NTRODUCTION above) to train RL agents [48], [99], [127], [178], [214], [231], [248], [268], [271].”
Deep learning in neural networks: An overview (2014, background): “This assumption does not hold in the broader fields of Sequential Decision Making and Reinforcement Learning (RL) (Kaelbling et al., 1996; Sutton and Barto, 1998; Hutter, 2005) (Sec.”
Towards Causal Representation Learning (2021, background): “world models that capture some of the causal relations underlying the environment and serve as Lorenzian imagined spaces (see INTRODUCTION above) to train RL agents [127, 248, 98, 47, 271, 178, 232, 214, 268].”
World Models (2018, background): “In many reinforcement learning (RL) problems (Kaelbling et al., 1996; Sutton & Barto, 1998; Wiering & van Otterlo, 2012), an artiﬁcial agent also beneﬁts from having a good representation of past and present states, and a good predictive model of the future (Werbos, 1987; Silver, 2017), preferably…”
Unmasking Clever Hans predictors and assessing what machines really learn (2019, background): “The interplay of environment, network and memory is a convoluted process where convergence is not guaranteed [116].”
Improving Robot Controller Transparency Through Autonomous Policy Explanation (2017, background): “We demonstrate our approach’s utility within three representative domains (discrete, continuous, and multi-agent) that map to three important classes of robot controllers, including hard-coded, conditional statement-driven policies, and those that rely upon trained reinforcement learning models [18]”
Reinforcement learning in robotics: A survey (2013, background): “As illustrated by Kaelbling et al. (1996), this parameter often qualitatively changes the form of the optimal solution.”
When to Trust Your Model: Model-Based Policy Optimization (2019, background): “Model-based reinforcement learning methods are promising candidates for real-world sequential decision-making problems due to their data efﬁciency (Kaelbling et al., 1996).”
Informing sequential clinical decision-making through reinforcement learning: an empirical study (2010, background): “For example, in (Kaelbling et al. 1996), (Strehl and Littman 2004), and (Strehl and Littman 2005), measures of confidence are used to guide exploration.”
A General Family of Robust Stochastic Operators for Reinforcement Learning (2018, methodology): “Through iterative application of a convergent operator, value-based reinforcement learning (RL) generates successive refinements of an initial value function [13, 20, 19].”
Maximum a Posteriori Policy Optimisation (2018, background): “cit M-step. Finally, the connection between RL and inference has been invoked to motivate work on exploration. The most prominent examples for this are formed by work on Boltzmann exploration such as Kaelbling et al. (1996); Perkins & Precup (2002); Sutton (1990); O’Donoghue et al. (2017), which”
A Multi-Agent Framework for Packet Routing in Wireless Sensor Networks (2015, background): “[37] note, however, that this kind of algorithm, which always takes the highest rewards action and overlooks the trade-off between exploitation and exploration, may converge to a sub-optimal state.”