Model-Based Offline Planning with Trajectory Pruning
By Xianyuan Zhan, Xiangyu Zhu, Haoran Xu. 2021
44 citations recorded by the source corpus.
The recent offline reinforcement learning (RL) studies have achieved much progress to make RL usable in real-world systems by learning policies from pre-collected datasets without environment interaction. Unfortunately, existing offline RL methods still face many practical challenges in real-world system control tasks, such as computational restriction during agent training and the requirement of extra control flexibility. The model-based planning framework provides an attractive alternative. However, most model-based planning algorithms are not designed for offline settings. Simply combining the ingredients of offline RL with existing methods either provides over-restrictive planning or leads to inferior performance. We propose a new light-weighted model-based offline planning framework, namely MOPP, which tackles the dilemma between the restrictions of offline learning and high-performance planning. MOPP encourages more aggressive trajectory rollout guided by the behavior policy learned from data, and prunes out problematic trajectories to avoid potential out-of-distribution samples. Experimental results show that MOPP provides competitive performance compared with existing model-based offline planning and RL approaches.
Subfields: Artificial Intelligence, Control and Systems Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Chelsea Finn
- Tianhe Yu
- Aravind Rajeswaran
- Aviral Kumar
- Pieter Abbeel
- Rafael Rafailov
- Abhishek Gupta
In Control and Systems Engineering
- Sergey Levine
- Chelsea Finn
- Aravind Rajeswaran
- Tianhe Yu
- Aviral Kumar
- Pieter Abbeel
- Rafael Rafailov
- Vikash Kumar
Citations in the corpus
- COMBO: Conservative Offline Model-Based Policy Optimization (2021, methodology): “Model-based offline RL. Model-based offline RL methods [11, 9, 24, 26, 67, 39, 3, 60, 48, 33, 68] provide an alternative approach to policy learning that involves the learning of a dynamics model using techniques from supervised learning and generative modeling.”