Combining Policy Gradient and Q-Learning

By N. Sanghi. 2021

115 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Experience Replay for Continual Learning (2018, background): “Previous works [7, 22, 31] have explored mixing on-and off-policy updates in RL, though these were focused on speed and stability in individual tasks and did not examine continual learning.”
Generalized Off-Policy Actor-Critic (2019, background): “The target policy can then be derived from the value function directly (O’Donoghue et al., 2016; Nachum et al., 2017a; Schulman et al., 2017a).”
P3O: Policy-on Policy-off Policy Optimization (2019, methodology): “The PGQL algorithm (O’Donoghue et al., 2016a) uses an estimate of the action value function of the target policy to combine on-policy updates with those obtained from the Bellman error objective.”
Deep Reinforcement Learning for Intelligent Transportation Systems: A Survey (2020, background): “There are recent papers that deal with the variations of actor-critic models using the deep RL approach [20], [21], [26], in which function approximators for both actor and critic are based on deep neural networks.”
Text Generation with Efficient (Soft) Q-Learning (2021, background): “Previous work in robotic and game control demonstrated that this formulation provides a substantial improvement in exploration and robustness [70, 57, 58, 69, 50, 41, 17, 36, 53, 37, 13].”
Expected Policy Gradients for Reinforcement Learning (2018, methodology): “For discrete action spaces, softmax Q -learning, a family of methods with a hybrid loss combining sarsa and Q -learning, has recently been linked to policy gradients via an entropy term(O’Donoghue et al., 2016).”
Stochastic Activation Actor Critic Methods (2019, unknown): “…DRL models built upon the maximum entropy framework [45] by augmenting the standard RL objective with an entropy maximization term to achieve probabilistic inference have gained much attention, thanks to the potential of improving exploration and generalization in the face of uncertainty [14, 28].”
Provably Convergent Two-Timescale Off-Policy Actor-Critic with Function Approximation (2019, result): “Other works in this line of research include Nachum et al. (2017a); O’Donoghue et al. (2016); Schulman et al. (2017); Nachum et al. (2017b); Haarnoja et al. (2017; 2018), which are mainly developed in a tabular setting and do not have a convergence analysis under function approximation.”
Stochastic matrix games with bandit feedback (2020, background): “On the other hand, for ﬁxed y and τ ≥ 0 the Lagrangian is concave in x , since entropy is concave [29].”