Lipschitz Continuity in Model-based Reinforcement Learning

By Kavosh Asadi, Dipendra Kumar Misra, Michael L. Littman. 2018

196 citations recorded by the source corpus.

Model-based reinforcement-learning methods learn transition and reward models and use them to guide behavior. We analyze the impact of learning models that are Lipschitz continuous---the distance between function values for two inputs is bounded by a linear function of the distance between the inputs. Our first result shows a tight bound on model errors for multi-step predictions with Lipschitz continuous models. We go on to prove an error bound for the value-function estimate arising from such models and show that the estimated value function is itself Lipschitz continuous. We conclude with empirical results that demonstrate significant benefits to enforcing Lipschitz continuity of neural net models during reinforcement learning.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Reinforcement Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Reinforcement Learning

In Statistical Learning Theory

Citations in the corpus

DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, background): “1 and if for all a P A : Several works have studied Lipschitz norm constraints on the transition and reward functions (Hinderer, 2005; Asadi et al., 2018) to provide conditions for value functions to be Lipschitz.”
When to Trust Your Model: Model-Based Policy Optimization (2019, methodology): “, 2017) or by regularizing the model by controlling its Lipschitz constant (Asadi et al., 2018).”
Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees (2018, methodology): “Asadi et al. (2018) also study the discrepancy bounds under Lipschitz condition of the MDP. Prior work explores a variety of ways of combining model-free and model-based ideas to achieve the best of the two methods (Sutton, 1991; 1990; Racanière et al., 2017; Mordatch et al., 2016; Sun et al.,…”
Non-Stationary Markov Decision Processes a Worst-Case Approach using Model-Based Reinforcement Learning (2019, background): “The choice of the Wasserstein distance is motivated by the fact that it quantifies the distance between two distributions in a physical and interpretable manner, respectful of the topology of the measured space (Dabney et al., 2017; Asadi et al., 2018).”
An investigation of model-free planning (2019, background): “Model-based approaches are potentially powerful but have been challenging to scale with learned models in complex and high-dimensional environments (Talvitie, 2014; Asadi et al., 2018), though there has been recent progress in that direction (Buesing et al., 2018; Ebert et al., 2018).”
Learning Markov State Abstractions for Deep Reinforcement Learning (2021, background): “Smoothness is another desirable property and its beneﬁts in RL are well known (Pazis & Parr, 2013; Pirotta et al., 2015; Asadi et al., 2018).”
Block Contextual MDPs for Continual Learning (2021, unknown): “Asadi et al. (2018) present results when bounding the model class to Lipschitz functions but assume that the given MDP is Lipschitz.”
Combating the Compounding-Error Problem with a Multi-step Model (2019, methodology): “The interplay between the two errors leads to what is referred to as the compounding-error problem [Asadi et al. , 2018b].”
Observational Overfitting in Reinforcement Learning (2019, background): “In this framework, there is a spectrum of analysis, ranging from almost purely theoretical analysis (Wang et al., 2019; Asadi et al., 2018) to full empirical results on diverse environments (Zhang et al.”
Learning to Predict Without Looking Ahead: World Models Without Forward Prediction (2019, background): “Errors in the world model compound, and cause issues when used for control [3, 61].”
Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future (2019, background): “This is referred to as the compounding error phenomenon (Talvi-tie, 2014; Asadi et al., 2018; Weber et al., 2017).”