Compositionality of optimal control laws

By E. Todorov. 2009

139 citations recorded by the source corpus.

Subfields: Mathematical Optimization, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Mathematical Optimization

In Robotics & Control

Citations in the corpus

Reinforcement Learning with Deep Energy-Based Policies (2017, background): “In the context of linearly solvable MDPs, several prior works have shown that policies trained for different tasks can be composed to create new optimal policies (Da Silva et al., 2009; Todorov, 2009).”
Multi-task Deep Reinforcement Learning with PopArt (2018, background): “Multi-task learning, as considered in this paper, where we get to execute, in parallel, the policies learned for each task, has potential additional beneﬁts, including deep exploration (Osband et al. 2016), and policy composition (Mankowitz et al. 2018; Todorov 2009).”
Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control (2018, background): “Since the log-sum-exp function approximates mean + variance for small κ > 0 [18, 19], this procedure encourages the agent to additionally explore parts of the state space where the disagreement between the function approximators is large.”
Open-Ended Learning Leads to Generally Capable Agents (2021, background): “From a formal standpoint the goal space of XLand forms a Boolean algebra and thus can benefit from exploitation of this structure (Nangue Tasse et al., 2020; Todorov, 2009; Van Niekerk et al., 2019).”
Black-Box Policy Search with Probabilistic Programs (2015, background): “The normalization constant Zλ is the expected value of the exponentiated reward exp(βR(τ)), which is known as the desirability in the context of optimal control [Kappen, 2005, Todorov, 2009].”
Modular Hopping and Running via Parallel Composition (2017, background): “Additionally, very little is known about composing the solutions of optimization problems in order to re-use prior libraries of solutions Todorov (2009).”
Modular Lifelong Reinforcement Learning via Neural Composition (2022, background): “One additional form of composition in RL uses full task policies as the components, and combines policies together when solving a new task (e.g., by summing their value functions; Todorov, 2009; Barreto et al., 2018; Haarnoja et al., 2018; Van Niekerk et al., 2019; Nangue Tasse et al., 2020).”
Composable Deep Reinforcement Learning for Robotic Manipulation (2018, background): “A related idea has been discussed by Todorov [36], who considers”
Hierarchy Through Composition with Multitask LMDPs (2017, background): “As shown in Todorov (2009b), the Bellman equation in this setting reduces to (I −MiPTi )zi =MiPTb zb (4) where Mi = diag(qi) and, because boundary states are absorbing, zb = qb.”
Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning (2021, background): “Popular approaches to composing skills involve combining them in the space of value functions (Todorov, 2009; Hunt et al., 2019; da Silva et al., 2009; Ziebart, 2010; Haarnoja et al., 2018), and more recently, using pseudo-rewards (Barreto et al.”
The Option Keyboard: Combining Skills in Reinforcement Learning (2021, unknown): “Although we are not aware of any other algorithm that explicitly attempts to combine skills in the space of cumulants, there are methods that do so in the space of value functions [29, 6, 13, 16].”
How to Reuse and Compose Knowledge for a Lifetime of Tasks: A Survey on Continual Learning and Functional Composition (2022, background): “Todorov (2009) showed that the linear composition of value functions is optimal in the case of linearly solvable Markov decision processes (MDPs).”