Off‐Policy Actor‐Criticアルゴリズムによる強化学習

By 木村元, 小林重信. 2004

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

Citations in the corpus

Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, background): “[9, 44]); such as learning online from an outdated replay buffer (e.”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “…improve off-policy learning differ from our work: They either have focused on scenarios where the policy is able to explore and collect more data (Degris et al., 2012; Ried-miller, 2005) such as learning online from an out-dated replay buffer (e.g. (Munos et al., 2016)), or have performed…”
PGQ: Combining policy gradient and Q-learning (2016, methodology): “Vanilla actor-critic methods are on-policy only, although some attempts have been made to extend them to off-policy data (Degris et al., 2012; Levine & Koltun, 2013).”
Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning (2017, methodology): “While it is not necessary for such algorithms to be off-policy, we primarily analyze the off-policy variants, such as (Riedmiller, 2005; Degris et al., 2012; Heess et al., 2015; Lillicrap et al., 2016).”
Generalized Off-Policy Actor-Critic (2019, methodology): “Geoff-PAC signiﬁcantly outperforms the actor-critic algorithms proposed by Degris et al. (2012); Imani et al. (2018), and to our best knowledge, Geoff-PAC is the ﬁrst empirical success of emphatic algorithms in prevailing deep RL benchmarks.”
An Off-policy Policy Gradient Theorem Using Emphatic Weightings (2018, background): “4 For λ a = 0 , the emphatic weighting is simply i , and the gradient with this weighting reduces to the regular off-policy actor critic update [Degris et al., 2012b].”
Batch Policy Gradient Methods for Improving Neural Conversation Models (2017, methodology): “This is similar to objectives used in on-line off-policy policy gradient literature where q is replaced by the limiting distribution of the behaviour policy (Degris et al., 2012).”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, neutral): “2 produces a Q-function such that the expected off-policy policy improvement objective [8] computed using this learned Q-function lower-bounds its actual value.”
P3O: Policy-on Policy-off Policy Optimization (2019, methodology): “Efforts to combine the ease-of-use of on-policy algorithms with the sample efficiency of off-policy algorithms have been fruitful (Gu et al., 2016; O’Donoghue et al., 2016b; Wang et al., 2016; Gu et al., 2017; Nachum et al., 2017; Degris et al., 2012).”
Reinforcement Learning from Imperfect Demonstrations (2018, methodology): “It is tempting to apply various off-policy methods to the problem of learning from demonstration, such as policy gradient variants (Gu et al., 2017; 2016; Degris et al., 2012; Wang et al., 2016), Q-learning (Watkins & Dayan, 1992) and Retrace (Munos et al.”
Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimization (2020, background): “This concept may be seen in contrast to sample efﬁciency or data efﬁciency (Precup et al., 2001; Degris et al., 2012; Gu et al., 2017b; Haarnoja et al., 2018; Lillicrap et al., 2016; Nachum et al., 2018), which measures the amount of environment interactions incurred during training, without regard…”
Off-Policy Policy Gradient with Stationary Distribution Correction (2019, methodology): “Shaded region represents 1 standard deviation over 10 runs of each method. but without the state distribution correction, that is the Off-PAC algorithm (Degris et al., 2012).”
Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning (2018, methodology): “To reconcile the use of off-policy data with an on-policy estimator, Silver et al. (2014) brieﬂy note an analogous off-policy policy improvement theorem in the case of ﬁnite state spaces (Degris et al., 2012).”
Discrete Sequential Prediction of Continuous Actions for Deep RL (2017, methodology): “[10] Thomas Degris, Martha White, and Richard S.”
Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning (2019, background): “These new developments reflect practical interests in deploying reinforcement learning to safety-critical situations [31, 57, 3, 1], and the increasing importance of off-policy learning and counterfactual reasoning [12, 52, 37, 30, 34, 41].”
Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing (2018, background): “There has been various attempts to incorporate off-policy samples within the policy gradient framework to improve the sample efﬁciency of the REINFORCE and actor-critic algorithms ( e.g. , [12, 57, 51, 15]).”
Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents (2018, methodology): “This projection is a common technique for stabilizing the stochastic approximation algorithms (Kushner & Yin, 2003), and is a standard assumption in many analyses for actorcritic algorithms (Bhatnagar et al., 2009; Degris et al., 2012; Prasad et al., 2014).”
Meta-Gradient Reinforcement Learning with an Objective Discovered Online (2020, unknown): “As a field, reinforcement learning (RL) has also largely embraced the transition from handcrafting features to handcrafting objectives: deep function approximation has been successfully combined with ideas such as TD-learning [30, 34], Q-learning [42, 23], double Q-learning [36, 37], n-step updates ”
RLLib: C++ Library to Predict, Control, and Represent Learnable Knowledge Using On/Off Policy Reinforcement Learning (2015, methodology): “We have used GQ(λ) [5] with the GVF question encoding: π(s,a) = 1, r(s) = 0.”