Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
By Che Wang, Zijian Zhou, Keith W. Ross. International Conference on Learning Representations, 2021
397 citations recorded by the source corpus.
Using a high Update-To-Data (UTD) ratio, model-based methods have recently achieved much higher sample efficiency than previous model-free methods for continuous-action DRL benchmarks. In this paper, we introduce a simple model-free algorithm, Randomized Ensembled Double Q-Learning (REDQ), and show that its performance is just as good as, if not better than, a state-of-the-art model-based algorithm for the MuJoCo benchmark. Moreover, REDQ can achieve this performance using fewer parameters than the model-based method, and with less wall-clock run time. REDQ has three carefully integrated ingredients which allow it to achieve its high performance: (i) a UTD ratio>>1; (ii) an ensemble of Q functions; (iii) in-target minimization across a random subset of Q functions from the ensemble. Through carefully designed experiments, we provide a detailed analysis of REDQ and related model-free algorithms. To our knowledge, REDQ is the first successful model-free DRL algorithm for continuous-action spaces using a UTD ratio>>1.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
View Paper: https://www.semanticscholar.org/paper/736590f70e7f2dc464c1c62491cfa8adb4d718f3
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Xiaolong Wang
- Ilya Kostrikov
- Hao Su
- Nicklas Hansen
- Sergey Levine
- Laura M. Smith
- Philip J. Ball
- Kang Liu
In Computer Vision and Pattern Recognition
- Sergey Levine
- Trevor Darrell
- Luc Van Gool
- Abhinav Kumar Gupta
- Xiaolong Wang
- Dacheng Tao
- Jitendra Malik
- Xiaogang Wang
Citations in the corpus
- A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development (2023, background): “(2) Up to now, various RL algorithms have been developed for different situations [78], e.g., TRPO [79], SAC [80], PPO [81], TD3 [82], and REDQ [83].”
- Efficient Online Reinforcement Learning with Offline Data (2023, methodology): “Approaches relying on this objective are empirically impressive (Haarnoja et al., 2018a;b; Chen et al., 2021; Hiraoka et al., 2022).”
- TD-MPC2: Scalable, Robust World Models for Continuous Control (2023, methodology): “Targets are then computed as the minimum of two randomly sub-sampled ¯ Q -functions (Chen et al., 2021).”