Model-based Reinforcement Learning for Semi-Markov Decision Processes with Neural ODEs
By Jianzhun Du, Joseph D. Futoma, Finale Doshi-Velez. 2020
63 citations recorded by the source corpus.
We present two elegant solutions for modeling continuous-time dynamics, in a novel model-based reinforcement learning (RL) framework for semi-Markov decision processes (SMDPs), using neural ordinary differential equations (ODEs). Our models accurately characterize continuous-time dynamics and enable us to develop high-performing policies using a small amount of data. We also develop a model-based approach for optimizing time schedules to reduce interaction rates with the environment while maintaining the near-optimal performance, which is not possible for model-free methods. We experimentally demonstrate the efficacy of our methods across various continuous-time domains.
Subfields: Artificial Intelligence, Control and Systems Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yilun Du
- Josh Tenenbaum
- Michael Janner
- Chelsea Finn
- Pieter Abbeel
- Finale Doshi-Velez
- Mohit Bansal
In Control and Systems Engineering
- Josh Tenenbaum
- Michael Janner
- Yilun Du
- Finale Doshi-Velez
- Sergey Levine
- Jianzhun Du
- Chelsea Finn
- Pieter Abbeel
Citations in the corpus
- Identifying Decision Points for Safe and Interpretable Reinforcement Learning in Hypotension Treatment (2021, methodology): “Du et al. (2020) take a model-based approach to continuous-time RL with demonstrations in the healthcare environment, whereas it cannot be applied to the batch RL setting.”
- TO-FLOW: Efficient Continuous Normalizing Flows with Temporal Optimization adjoint with Moving Speed (2022, background): “In addition, Neural ODEs have shown great promise for numerous tasks such as image registration [59], video generation [60], reinforcement learning [61] and system identification [34].”
- Planning with Diffusion for Flexible Behavior Synthesis (2022, background): “Iterative, gradient-based sampling lends itself towards flexible conditioning (Dhariwal & Nichol, 2021) and compositionality (Du et al., 2020b), which we use to recover effective behaviors from heterogeneous datasets and plan for reward functions unseen during training.”