Linear Off-Policy Actor-Critic

By Thomas Degris, Martha White, Richard S. Sutton. 2012

213 citations recorded by the source corpus.

This paper presents the first actor-critic algorithm for off-policy reinforcement learning. Our algorithm is online and incremental, and its per-time-step complexity scales linearly with the number of learned weights. Previous work on actor-critic algorithms is limited to the on-policy setting and does not take advantage of the recent advances in off-policy gradient temporal-difference learning. Off-policy techniques, such as Greedy-GQ, enable a target policy to be learned while following and obtaining data from another (behavior) policy. For many problems, however, actor-critic methods are more practical than action value methods (like Greedy-GQ) because they explicitly represent the policy; consequently, the policy can be stochastic and utilize a large action space. In this paper, we illustrate how to practically combine the generality and learning potential of off-policy learning with the flexibility in action selection given by actor-critic methods. We derive an incremental, linear time and space complexity algorithm that includes eligibility traces, prove convergence under assumptions similar to previous off-policy algorithms, and empirically show better or comparable performance to existing algorithms on standard reinforcement-learning benchmark problems.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Computer Networks and Communications, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Computer Networks and Communications

In Reinforcement Learning

Citations in the corpus

Sample Efficient Actor-Critic with Experience Replay (2016, methodology): “The existing variants applicable to both continuous and discrete domains, such as the on-policy asynchronous advantage actor critic (A3C) of Mnih et al. (2016), are sample inefficient.”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “…improve off-policy learning differ from our work: They either have focused on scenarios where the policy is able to explore and collect more data (Degris et al., 2012; Riedmiller, 2005) such as learning online from an outdated replay buffer (e.g. (Munos et al., 2016)), or have performed…”
Off-Policy Actor-Critic with Shared Experience Replay (2019, methodology): “Following Degris et al. (2012), a multi-step formulation of the expected return is where E µ denotes the expectation under policy µ up to an episode termination, δ t V = r t + γV ( s t +1 ) − V ( s t ) is the temporal difference error in consecutive states s t +1 , s t , and π t = π t ( a t | s t )…”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “Differentiating the objective and applying a further approximation results in the off-policy policy gradient (Degris et al., 2012):”
Accelerating Online Reinforcement Learning with Offline Datasets (2020, background): “Off-policy RL algorithms are designed to reuse off-policy data during training, and have been studied extensively [21, 6, 27, 13, 9, 5, 34, 52, 48, 4].”
MOPO: Model-based Offline Policy Optimization (2020, background): “While many off-policy RL algorithms [52, 10, 30, 47, 42, 26, 19, 23, 24] can in principle be applied to a batch ofﬂine setting, they perform poorly in practice [18] due to poor extrapolation to out-of-distribution actions.”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, background): “The more common approach to the off-policy problem is to simply ignore it, which is exactly what has been proposed by many existing off-policy policy gradient methods (Degris et al., 2012; Silver et al., 2014).”
Learning When-to-Treat Policies (2019, background): “Adaptions of actor-critic (Degris et al., 2012) and Gaussian processes (Schulam and Saria, 2017) have been proposed for the off-policy setting as well.”
Deep Reinforcement Learning and the Deadly Triad (2018, methodology): “Bootstrapping is also commonly used within policy-gradient and actor-critic methods to learn value functions (Witten, 1977; Barto et al., 1983; Peters and Schaal, 2008; Bhatnagar et al., 2009; van Hasselt, 2012; Degris et al., 2012; Mnih et al., 2016).”
End-to-End Offline Goal-Oriented Dialog Policy Learning via Policy Gradient (2017, methodology): “In this paper, we propose to use off-policy policy gradient [4] to exploit such information.”
Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, methodology): “, 2016, Degris et al., 2012, Wang et al., 2016] or Q-learning [Watkins and Dayan, 1992, Haarnoja et al., 2018, Munos et al., 2016]. Fujimoto et al. [2018a] propose to solve off-policy reinforcement learning by constraining the action space, and Fujimoto et al. [2018b] use double Q-learning [Van Hass”
EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (2020, background): “Leveraging past interactions in order to improve a decision-making process is the hallmark goal of off-policy reinforcement learning (RL) (Precup et al., 2001; Degris et al., 2012).”
Simple random search provides a competitive approach to reinforcement learning (2018, background): “[5], combined such ideas with the actor-critic framework into a method for training deterministic policies, relying on exploratory poli-”
Deterministic Policy Gradient Algorithms (2014, methodology): “…in favour of policy evaluation algorithms that estimate the value function more ef-ﬁciently by temporal-difference learning (Bhatnagar et al., 2007; Degris et al., 2012b; Peters et al., 2005); indeed if both i) and ii) are satisﬁed then the overall algorithm is equivalent to not using a critic at…”
Provably Convergent Off-Policy Actor-Critic with Function Approximation (2019, methodology): “While there have been efforts to combine actor-critic algorithms with off-policy learning (Degris et al., 2012; Imani et al., 2018; Maei, 2018; Zhang et al., 2019; Liu et al., 2019), none of the resulting off-policy actor-critic algorithms is provably convergent under function approximation.”
A Multi-Agent Off-Policy Actor-Critic Algorithm for Distributed Reinforcement Learning (2019, background): “Following the foundational policy gradient theorem of [16] for the on-policy case, some efforts in the off-policy direction include [12], [13], [3], [4], as well as [10], which builds off the off-policy policy gradient theorem of [2] in the tabular case to prove convergence of the actor step under l”
Statistically Efficient Off-Policy Policy Gradients (2020, background): “(4) Following Degris et al. (2012), Chen et al. (2019) replace ν0:t with ν̃t in Eq.”