Efficient Exploration via State Marginal Matching

By Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric P. Xing, Sergey Levine, Ruslan Salakhutdinov. 2019

289 citations recorded by the source corpus.

Exploration is critical to a reinforcement learning agent's performance in its given environment. Prior exploration methods are often based on using heuristic auxiliary predictions to guide policy behavior, lacking a mathematically-grounded objective with clear properties. In contrast, we recast exploration as a problem of State Marginal Matching (SMM), where we aim to learn a policy for which the state marginal distribution matches a given target state distribution. The target distribution is a uniform distribution in most cases, but can incorporate prior knowledge if available. In effect, SMM amortizes the cost of learning to explore in a given environment. The SMM objective can be viewed as a two-player, zero-sum game between a state density model and a parametric policy, an idea that we use to build an algorithm for optimizing the SMM objective. Using this formalism, we further demonstrate that prior work approximately maximizes the SMM objective, offering an explanation for the success of these methods. On both simulated and real-world tasks, we demonstrate that agents that directly optimize the SMM objective explore faster and adapt more quickly to new tasks as compared to prior exploration methods.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, background): “Yet another approach with apparent similarity to KL-control and DPG is state marginal matching (SMM) (Hazan et al., 2018; Lee et al., 2019).”
Continual Learning of Control Primitives: Skill Discovery via Reset-Games (2020, background): “Adversarial games have been proposed in RL with the goal of improving exploration [42, 27], learning goal conditioned policies [36] and generating an automated curriculum [45, 41, 2].”
A Divergence Minimization Perspective on Imitation Learning Methods (2019, methodology): “[34] also present a unifying probabilistic perspective on IL; however, their empirical experiments solely focus on grid world domains, while our work provides comparative results on high-dimensional continuous control environments and also evaluates the effectiveness of IL algorithms for state margi”
BeBold: Exploration Beyond the Boundary of Explored Regions (2020, background): “Other criteria exist, e.g., diversity (Eysenbach et al., 2018), feature control (Jaderberg et al., 2016; Dilokthanakul et al., 2019) or the KL divergence between current distribution over states and a target distribution of states (Lee et al., 2019).”
Unsupervised Meta-Learning for Reinforcement Learning (2018, background): “Using the calculus of variations (for more details refer to Appendix C in Lee et al. (2019)), the exploration policy for the optimal meta-learner, π ∗ , satisﬁes: Thus, when the goal sampling distribution p ( s g ) is known, the optimal learning procedure is obtained by ﬁnding π ∗ satisfying Eq.”
If MaxEnt RL is the Answer, What is the Question? (2019, background): “[26] Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov.”
Understanding the origin of information-seeking exploration in probabilistic objectives for control (2021, methodology): “However, this has not yet been the case, and probabilistic methods have largely contented themselves with purely random, entropy maximizing exploration (Haarnoja, Zhou, Abbeel, & Levine, 2018; Haarnoja, Zhou, Hartikainen, et al., 2018; Lee et al., 2019; K. Rawlik, Toussaint, & Vijayakumar, 2010).”
Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills (2020, methodology): “Given that SMM relies on a curiosity-like bonus (Lee et al., 2019), we attribute this failure to well-known issues of these methods such as derailment and detachment (Ecoffet et al., 2019).”
Action and Perception as Divergence Minimization (2020, methodology): “Divergence minimization relates the above methods and motivates an additional entropy regularizer for inputs (Todorov, 2008; Lee et al., 2019b; Xin et al., 2020).”
Grounding Language in Play (2020, background): “…sort of state space coverage autonomously is a long-standing problem (Thrun, 1992) motivating work on tractable heuristics (Kolter and Ng, 2009), intrinsic motivation (Oudeyer et al., 2008; Schmid-huber, 2010), and more recent information-theoretic approaches (Pong et al., 2019; Lee et al., 2019).”