DiCE: The Infinitely Differentiable Monte-Carlo Estimator

By Jakob N. Foerster, Gregory Farquhar, Maruan Al-Shedivat, Tim Rocktaschel, Eric P. Xing, Shimon Whiteson. 2018

107 citations recorded by the source corpus.

The score function estimator is widely used for estimating gradients of stochastic objectives in Stochastic Computation Graphs (SCG), eg. in reinforcement learning and meta-learning. While deriving the first-order gradient estimators by differentiating a surrogate loss (SL) objective is computationally and conceptually simple, using the same approach for higher-order gradients is more challenging. Firstly, analytically deriving and implementing such estimators is laborious and not compliant with automatic differentiation. Secondly, repeatedly applying SL to construct new objectives for each order gradient involves increasingly cumbersome graph manipulations. Lastly, to match the first-order gradient under differentiation, SL treats part of the cost as a fixed sample, which we show leads to missing and wrong terms for higher-order gradient estimators. To address all these shortcomings in a unified way, we introduce DiCE, which provides a single objective that can be differentiated repeatedly, generating correct gradient estimators of any order in SCGs. Unlike SL, DiCE relies on automatic differentiation for performing the requisite graph manipulations. We verify the correctness of DiCE both through a proof and through numerical evaluation of the DiCE gradient estimates. We also use DiCE to propose and evaluate a novel approach for multi-agent learning. Our code is available at this https URL

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “…and Faust, 2021b; Kirsch, Flennerhag, van Hasselt, Friesen, Oh, and Chen, 2022; Wan, Peng, and Gangwani, 2022; Melo, 2022; Nam, Sun, Pertsch, Hwang, and Lim, 2022) and multi-agent RL (Foerster et al., 2018a,1; Kim et al., 2021a; Al-Shedivat, Bansal, Burda, Sutskever, Mordatch, and Abbeel, 2017).”
Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments (2017, background): “A general solution for such issues is developed in [4].”
ProMP: Proximal Meta-Policy Search (2018, methodology): “As discussed by Foerster et al. (2018), and shown in Appendix B.3, the score function surrogate objective approach is ill suited for calculating higher order derivatives via automatic differentiation toolboxes.”
When MAML Can Adapt Fast and How to Assist When It Cannot (2019, background): “…attempted to explain the generalization ability of GBML (Guiroy et al., 2019; Nichol et al., 2018), the bias induced by restricting the number of adaptation steps (Wu et al., 2018), or the effect of higher-order terms in the meta-gradient estimation (Foerster et al., 2018; Rothfuss et al., 2019).”
Multi-Step Model-Agnostic Meta-Learning: Convergence and Improved Algorithms (2020, background): “Many follow-up studies (Grant et al., 2018; Finn et al., 2019; Jerfel et al., 2018; Finn and Levine, 2017; Finn et al., 2018; Mi et al., 2019; Liu et al., 2019; Rothfuss et al., 2018; Foerster et al., 2018; Fallah et al., 2019; Collins et al., 2020; Fallah et al., 2020) have extended MAML from diffe”
Stable Opponent Shaping in Differentiable Games (2018, background): “Foerster et al. (2018) assume that opponents are naive learners, namely ∆ θ 2 = − α 2 ∇ 2 L 2 .”
Monte Carlo Gradient Estimation in Machine Learning (2019, methodology): “And this framework has been further expanded by incorporating methods that include variance reductions within the computational graph, and to allow for higher-order gradients to be computed more easily and with lower variance (Foerster et al., 2018; Mao et al., 2018; Parmas et al., 2018; Weber et al”
VIREL: A Variational Inference Framework for Reinforcement Learning (2018, background): “where a [·] is the stopgrad operator, which sets the gradient of its operand to zero, a [·] = ·, ∇a [·] = 0 [16].”
Deep Multi Agent Reinforcement Learning for Autonomous Driving (2020, background): “Recent research work [11, 13] has focused on learning decentralized MARL policies by sharing the gradient information between agents during training.”
Influencing Long-Term Behavior in Multiagent Reinforcement Learning (2022, unknown): “While those frameworks show improved adaptation performance over methods that neglect the learning of other agents (Lowe et al., 2017; Foerster et al., 2018b; Iqbal & Sha, 2019), they suffer from myopic evaluation : an agent only considers a few anticipated updates to the policies of other agents…”