Incremental multi-step Q-learning

By Jing Peng, Ronald J. Williams. 1994

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Asynchronous Methods for Deep Reinforcement Learning (2016, methodology): “…n-step methods operate in the forward view (Sutton & Barto, 1998) by using corrected n-step returns directly as targets, it has been more common to use the backward view to implicitly combine different returns through eligibility traces (Watkins, 1989; Sutton & Barto, 1998; Peng & Williams, 1996).”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “Our proposed PCL algorithm bears some similiarity to multi-step Q-learning (Peng & Williams, 1996), which rather than minimizing one-step hard-max Bellman error, optimizes a Q-value function approximator by unrolling the trajectory for some number of steps before using a hardmax backup.”
PGQ: Combining policy gradient and Q-learning (2016, methodology): “…at the next state, i.e. , In REINFORCE the critic is the Monte Carlo return from that state on, If the return trace is truncated and a bootstrap is performed after n -steps, this is equivalent to n -step SARSA or n -step Q-learning, depending on the form of the bootstrap (Peng & Williams, 1996).”
Reinforcement Learning with Unsupervised Auxiliary Tasks (2016, methodology): “The UVFA architecture (Schaul et al., 2015a) is a factored representation of a continuous set of optimal value functions, combining features of the state with an embedding of the pseudo-reward function.”
Deep learning in neural networks: An overview (2014, neutral): “…& Parr, 2003; Maei & Sutton, 2010; Mahadevan, 1996; Meuleau, Peshkin, Kim, & Kaelbling, 1999; Moore & Atkeson, 1993; Morimoto & Doya, 2000; Peng & Williams, 1996; Prokhorov &Wunsch, 1997; Rummery &Niranjan, 1994; Santamaría, Sutton, & Ram, 1997; Schwartz, 1993; Singh, 1994; Sutton & Barto, 1998;…”
RUDDER: Return Decomposition for Delayed Rewards (2018, methodology): “Another eligibility trace method is Peng’s Q(λ) [90] which also does not zero out eligibility traces.”
Reinforcement learning in robotics: A survey (2013, methodology): “Q-learning and dynamic programming based approaches have been compared in a pick-n-place task (Kalmár et al., 1998) using modules.”
On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models (2015, methodology): “The parameters or weights of C are trained to maximize reward by a standard RL method such as Q-learning or similar methods [17, 292, 293, 172, 254, 212, 262, 10, 122, 188, 157, 281, 26, 216, 197, 272, 311, 9, 163, 174, 22, 27, 2, 137, 276, 156, 284].”
Q(λ) with Off-Policy Corrections (2016, background): “Q(λ) of Peng and Williams [7] is meant to remedy this, by being a hybrid between SARSA(λ) and Watkins’s Q(λ).”
Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning (2018, background): “Usually, n -step return methods are on-policy: they estimate target Q -values at some state-action pair as ) for a trace of rewards r t in an observed H -step trajectory (Peng & Williams, 1994).”
Reinforcement Learning with Replacing Eligibility Traces (2005, methodology): “The second mechanism that has been widely used for handling delay is the el-igibility trace.1 Introduced by Klopf (1972), eligibility traces have been used in avariety of reinforcement learning systems (e.g., Barto, Sutton & Anderson, 1983;Lin, 1992; Tesauro, 1992; Peng & Williams, 1994).”
Accelerated Methods for Deep Reinforcement Learning (2018, methodology): “…learning was combined with ﬁve other enhancements under the name Rainbow: 1) Double-DQN (Van Hasselt et al., 2016), 2) Dueling Networks (Wang et al., 2016), 3) Priori-tized Replay (Schaul et al., 2016), 4) n-step learning (Peng & Williams, 1994), and 5) NoisyNets (Fortunato et al., 2018).”
Multi-agent Reinforcement Learning: An Overview (2010, background): “A broad spectrum of single-agent RL algorithms exists, e.g., model-free methods based on the online estimation of value functions [6,89,118,120,137], model-based methods (typically called dynamic programming) [8, 96], and model-learning methods that estimate a model, and then learn using…”
Transfer Learning in Deep Reinforcement Learning: A Survey (2020, background): “Examples of Qlearning variations with eligibility traces, called Q(λ), are the work done by [22] [23], and [1].”
Applying reinforcement learning to small scale combat in the real-time strategy game StarCraft:Broodwar (2012, unknown): “Peng’s Q( λ ) [25] on the other hand never cuts traces, thus making it basically a hybrid of Watkins’s Q( λ ) and Sarsa( λ ) [2].”
Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning (2021, methodology): “DDPG is amenable to incorporate n -step returns [Watkins, 1989, eng and Williams, 1996] when estimating TD error beyond a single step [Barth-Maron et al., 2018].”