Calibrated Model-Based Deep Reinforcement Learning

By Ali Malik, Volodymyr Kuleshov, Jiaming Song, Danny Nemer, Harlan Seymour, Stefano Ermon. 2019

62 citations recorded by the source corpus.

Estimates of predictive uncertainty are important for accurate model-based planning and reinforcement learning. However, predictive uncertainties---especially ones derived from modern deep learning systems---can be inaccurate and impose a bottleneck on performance. This paper explores which uncertainties are needed for model-based reinforcement learning and argues that good uncertainties must be calibrated, i.e. their probabilities should match empirical frequencies of predicted events. We describe a simple way to augment any model-based reinforcement learning agent with a calibrated model and show that doing so consistently improves planning, sample complexity, and exploration. On the \textsc{HalfCheetah} MuJoCo task, our system achieves state-of-the-art performance using 50\% fewer samples than the current leading approach. Our findings suggest that calibration can improve the performance of model-based reinforcement learning with minimal computational and implementation overhead.

Subfields: Artificial Intelligence, Information Systems, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Tengyu Ma
3. Chelsea Finn
4. Pieter Abbeel
5. Michael Janner
6. Percy Liang
7. Vikash Kumar
8. Qiyang Li

In Information Systems

1. Percy Liang
2. Sergey Levine
3. Tengyu Ma
4. Luke S. Zettlemoyer
5. Josh Tenenbaum
6. Michihiro Yasunaga
7. Jacob Andreas
8. Mirella Lapata

In Reinforcement Learning

1. Sergey Levine
2. Michael Janner
3. Vikash Kumar
4. Qiyang Li
5. Pieter Abbeel
6. Tengyu Ma
7. Anusha Nagabandi
8. Chelsea Finn

Citations in the corpus

• Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “However, while these prior methods typically require additional machinery to work well, such as ensembles in the online setting (Kurutach et al., 2018; Buckman et al., 2018; Malik et al., 2019) or conservatism mechanisms in the offline setting (Yu et al.”
• Verified Uncertainty Calibration (2019, background): “Calibration, including the squared calibration error, has been studied in many fields besides machine learning including meteorology [2, 3, 4, 5, 6], fairness [29, 30], healthcare [1, 31, 32, 33], reinforcement learning [34], natural language processing [7, 8], speech recognition [35], econometrics ”
• Deep Dynamics Models for Learning Dexterous Manipulation (2019, methodology): “Model-based RL methods, including recent work in uncertainty estimation [29] and combining policy networks with online planning [30], have unfortunately mostly been studied and shown on lower-dimensional (and often, simulated) benchmark tasks, and scaling these methods to higher dimensional tasks…”
• Reinforcement Learning as One Big Sequence Modeling Problem (2021, background): “However, while these prior methods typically require additional machinery to work well, such as ensembles (in the online setting) (Chua et al., 2018; Kurutach et al., 2018; Buckman et al., 2018; Malik et al., 2019) or conservatism or pessimism mechanisms (in the offline setting) (Yu et al.”