Linearly-solvable Markov decision problems

By E. Todorov. 2006

548 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “Furthermore, even if RL with KL penalties (Todorov, 2006a; Kappen et al., 2012; Jaques et al., 2017; 2019), the method used to fine-tune a LM in RLHF, is defined only in terms of reward maximization, it has also been shown to be equivalent to minimizing the reverse KL divergence KL(πθ||p) of πθ to…”
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, background): “KL-control (Todorov, 2007; Kappen et al., 2012), was leveraged by Jaques et al.”
Human-centric Dialog Training via Offline Reinforcement Learning (2020, methodology): “The approach we propose is based on KL-control, a branch of stochastic optimal control (SOC) (Sten-gel, 1986) where the Kullback-Leibler (KL) divergence from some distribution is used to regularize an RL policy (Abdolmaleki et al., 2018; Kappen et al., 2012; Rawlik et al., 2012; Todorov, 2007).”
Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, methodology): “Todorov (2006) formulated a class of reinforcement learning problems termed “linearly-solvable” MDPs (LMDPs).”
Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-control (2016, background): “There is also a connection between this work and Maximum Entropy Inverse RL (Ziebart et al., 2008), which can be seen as KL control with a ﬂat, improper prior.”
Bridging the Gap Between Value and Policy Based Reinforcement Learning (2017, methodology): “Even though entropy regularized policy optimization is a well studied topic in RL [46, 39, 40, 47, 5, 4, 6, 7]–in fact, one that has been attracting renewed interest from concurrent work [25, 11]–we contribute new observations to this study that are essential for the methods we propose: first, we id”
Reinforcement Learning with Deep Energy-Based Policies (2017, methodology): “1 Optimization problems of this type have been explored in a number of prior works (Kappen, 2005; Todorov, 2007; Ziebart et al., 2008), which are covered in more detail in Section 4.”
Taming the Noise in Reinforcement Learning via Soft Updates (2015, neutral): “Information considerations have received attention in recent years in various machine learning settings, with the free energy F π and similar quantities used as a design principle for policies in known MDPs [10, 29, 30].”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “Of course, model-based RL methods present their own set of challenges: while some MDPs are easy to model accurately, others can be exceedingly difficult.”
AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, background): “…in the literature, including exploration (de Farias and Van Roy, 2000; Haarnoja et al., 2017, 2018), smoothing (Dai et al., 2018b), avoiding premature convergence (Nachum et al., 2017a), ensuring tractability (Todorov, 2006), and mitigating observation noise (Rubin et al., 2012; Fox et al., 2016).”
Deep Imitative Models for Flexible Inference, Planning, and Control (2018, methodology): “Given costs c(si|φ), this can be treated by including an optimality variable C in G, where p(C = 1|s1:T , φ) ∝ ∏T t=1 exp−c(st|φ) (Todorov, 2007; Levine, 2018).”
Boosting the Actor with Dual Critic (2017, methodology): “Their algorithm is based on the duality of entropy-regularized Bellman equation (Todorov, 2007; Rubin et al., 2012; Fox et al., 2015; Haarnoja et al., 2017; Nachum et al., 2017), rather than the exact Bellman optimality equation used in our work.”
Smoothed Dual Embedding Control (2017, methodology): “tropy-regularized MDP has been proposed for balancing exploration and exploitation (Haarnoja et al., 2017), taming the noises in observations (Rubin et al., 2012; Fox et al., 2015), and tractability (Todorov, 2007). Specically, Fox et al. (2015) proposed soft Q-learning which extended the Q-learning”
MAVEN: Multi-Agent Variational Exploration (2019, methodology): “[18] use normalising flows [41] to learn hierarchical latent space policies using max entropy RL [49, 53, 9], which is related to MI maximisation but ignores the variational posterior over latent space behaviours.”
Soft Actor-Critic Algorithms and Applications (2018, background): “Optimization problems of this type have been explored in a number of prior works (Kappen, 2005; Todorov, 2007; Ziebart et al., 2008).”
Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization (2016, background): “Several works have proposed to relax this requirement, for example by learning a value function instead of a cost (Todorov, 2006), solving an approximate local con-trol problem (Levine & Koltun, 2012; Dragan & Srinivasa, 2012), generating a discrete graph of states (Byravan et al., 2015; Monfort et…”
Reward Augmented Maximum Likelihood for Neural Structured Prediction (2016, background): “There is some relationship to the work presented here and work on relative entropy policy search [27], and policy optimization via expctation maximization [43] and KLdivergence [16, 39], however none of these bridge the gap between the two directions of the KLdivergence, nor do they consider any sup”
Dynamic policy programming (2010, methodology): “Following Todorov (2006), we state Proposition 1: Proposition 1 Let η be a positive constant, then for all x ∈ X the optimal value function V ∗ ¯ π ( x ) and for all ( x, a ) ∈ Z the optimal policy ¯ π ∗ ( a | x ) , respectively, satisfy: Proof See Appendix A.”