General duality between optimal control and estimation

By E. Todorov. 2008

353 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Networks and Communications, Control and Systems Engineering, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Control and Systems Engineering

In Robotics & Control

Citations in the corpus

Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review (2018, background): “…connection between probabilistic inference and control has been explored in the literature under a variety of names, including the Kalman duality (Todorov, 2008), maximum entropy reinforcement learning (Ziebart, 2010), KLdivergence control (Kappen et al., 2012; Kappen, 2011), and stochastic…”
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, methodology): “This framework has been used in many contexts, from inverse reinforcement learning (Ziebart et al., 2008) to optimal control (Todorov, 2008; Toussaint, 2009; Rawlik et al., 2012).”
Reinforcement Learning with Deep Energy-Based Policies (2017, methodology): “…← θ , ¯ φ ← φ . end if end for case of linear-quadratic systems, the mean of the maximum entropy policy is exactly the optimal deterministic policy (Todorov, 2008), which has been exploited to construct practical path planning methods based on iterative linearization and probabilistic inference…”
Maximum a Posteriori Policy Optimisation (2018, methodology): “Among these are methods that perform trajectory optimisation using either linearised dynamics (Todorov, 2008; Toussaint, 2009; Levine & Koltun, 2013) or general dynamics as in path integral control (Kappen, 2005; Theodorou et al., 2010).”
Reinforcement Learning from Imperfect Demonstrations (2018, background): “Maximum entropy reinforcement learning has been explored in a number of prior works (Todorov, 2008; Toussaint, 2009; Ziebart et al., 2008), including several recent works that extend it into a deep reinforcement learning setting (Nachum et al., 2017; Haarnoja et al., 2017b; Schulman et al., 2017).”
Soft Actor-Critic Algorithms and Applications (2018, methodology): “This framework has been used in many contexts, from inverse reinforcement learning (Ziebart et al., 2008) to optimal control (Todorov, 2008; Toussaint, 2009; Rawlik et al., 2012).”
Optimal control as a graphical model inference problem (2009, neutral): “12 for KL control problems was ﬁrst established in [9].”
Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition (2018, background): “Duality between control and inference in the case of linear dynamical systems has been studied in Kalman (1960); Todorov (2008).”
Efficient computation of optimal actions (2009, background): “This duality is wellknown for linear systems (4) and was recently generalized (8, 10) to nonlinear continuous systems of the form Eqs.”
Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning (2021, background): “Especially in the continuous control domain, interpreting RL as probabilistic inference [60, 61] has yielded many kinds of algorithms with strong empirical performances [34, 50, 51, 12, 29, 21, 16, 24].”
Stochastic Activation Actor Critic Methods (2019, neutral): “In order to do so, the return (or related RL objectives) is defined as a factor in a graphical model and probabilistic inference is applied to determine a sequence of actions optimizing this objective [45].”
Learning to Walk via Deep Reinforcement Learning (2018, methodology): “This framework has been used in many contexts, from inverse reinforcement learning [46] to optimal control [43, 44, 36].”
Action and Perception as Divergence Minimization (2020, methodology): “This term is analogous to the objective in KL control (Todorov, 2008; Kappen et al., 2009), except that the inputs now depend upon latent variables via the policy.”
Modeling the Interaction between Agents in Cooperative Multi-Agent Reinforcement Learning (2021, background): “3 Maximum Entropy RL Maximum Entropy Learning is an active research area in recent years and it has been applied inmany aspects [33, 43].”