Ranking Policy Gradient
By Kaixiang Lin, Jiayu Zhou. 2019
8 citations recorded by the source corpus.
Sample inefficiency is a long-lasting problem in reinforcement learning (RL). The state-of-the-art estimates the optimal action values while it usually involves an extensive search over the state-action space and unstable optimization. Towards the sample-efficient RL, we propose ranking policy gradient (RPG), a policy gradient method that learns the optimal rank of a set of discrete actions. To accelerate the learning of policy gradient methods, we establish the equivalence between maximizing the lower bound of return and imitating a near-optimal policy without accessing any oracles. These results lead to a general off-policy learning framework, which preserves the optimality, reduces variance, and improves the sample-efficiency. Furthermore, the sample complexity of RPG does not depend on the dimension of state space, which enables RPG for large-scale problems. We conduct extensive experiments showing that when consolidating with the off-policy learning framework, RPG substantially reduces the sample complexity, comparing to the state-of-the-art.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Yoshua Bengio
- Sergey Levine
- Caglar Gulcehre
- Kyunghyun Cho
- Quoc V. Le
- Ruslan Salakhutdinov
- Ziyun Wang
- Chelsea Finn
Citations in the corpus
- Regularized Behavior Value Estimation (2021, methodology): “The Ranking Policy Gradient Theorem formulates the optimization of long-term reward using a ranking objective as done in (Lin and Zhou, 2020).”
- Simplifying Deep Reinforcement Learning via Self-Supervision (2021, methodology): “Unlike the previous work where the demonstrations are assumed to be perfect (Ziebart et al. 2008), or near optimal (Lin and Zhou 2019), our framework continuously refines the demonstrations in the buffer during the learning process and eventually finds the expert-level demonstrations and discovers a…”
- Revisiting Fundamentals of Experience Replay (2020, unknown): “Quantities similar to the replay ratio have also been identified as important hyperparameters in several recent works on deep RL methods (Wang et al., 2016; Kapturowski et al., 2019; Hamrick et al., 2020; Lin & Zhou, 2020).”