PAC-inspired Option Discovery in Lifelong Reinforcement Learning

By Emma Brunskill, Lihong Li. 2014

141 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

A Deep Hierarchical Approach to Lifelong Learning in Minecraft (2016, background): “For example, Ammar et al. (2014) and Ammar et al. (2015) transfer knowledge via a latent basis whereas Brunskill and Li (2014) perform batch optimization across all encountered tasks.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “…2020; Klissarov and Precup, 2021), work on discovering options in a multi-task context has shown some promising potential both theoretically (Brunskill and Li, 2014) and empirically (Achiam, Edwards, Amodei, and Abbeel, 2018; Riemer, Liu, and Tesauro, 2018; Igl, Gambardella, He, Nardelli,…”
Near-Optimal Representation Learning for Hierarchical Reinforcement Learning (2018, methodology): “…information relates closely with maximum likelihood learning of energy-based models, and our bounds effectively correspond to bounds based on model-based predictive errors, a basic family of bounds in representation learning in MDPs (Sorg & Singh, 2009; Brunskill & Li, 2014; Abel et al., 2017).”
Unicorn: Continual Learning with a Universal, Off-policy Agent (2018, background): “Brunskill and Li [8] derive sample complexity bounds for option discovery in a lifelong learning setting.”
Lifelong Hyper-Policy Optimization with Multiple Importance Sampling Regularization (2021, background): “Still moving towards a more realistic setting, another modiﬁcation is the lifelong interaction with the environment (Silver, Yang, and Li 2013; Brunskill and Li 2014).”
Eigenoption Discovery through the Deep Successor Representation (2017, methodology): “In the reinforcement learning framework, options are a well-known formalization of the notion of actions extended in time; and they have been shown to speed up learning and planning when appropriately deﬁned ( e.g. , Brunskill & Li, 2014; Guo et al., 2017; Solway et al., 2014).”
Iterative Hierarchical Optimization for Misspecified Problems (IHOMP) (2016, background): “Learning useful options has been a topic of intense research [McGovern and Barto, 2001, Moerman, 2009, Konidaris and Barto, 2009, Brunskill and Li, 2014, Hauskrecht et al., 1998].”
Context-Aware Policy Reuse (2018, background): “Source policies in [8] are restricted to be learned in a PAC-learning way to obtain a ε-optimal option library.”
Lifelong Federated Reinforcement Learning: A Learning Architecture for Navigation in Cloud Robotic Systems (2019, unknown): “In contrast, Brunskill and Li [20] suggested a temporal transfer”
Portable Option Discovery for Automated Learning Transfer in Object-Oriented Markov Decision Processes (2015, background): “Brunskill and Li [2014] provide the first theoretical analysis of sample complexity for option discovery in MDPs, and propose a PACinspired algorithm capable of working in analogous domains with a performance approaching that of handcrafted options.”
Using Options and Covariance Testing for Long Horizon Off-Policy Policy Evaluation (2017, background): “The value of policy π is the expected return when running that policy: Temporal abstraction can reduce the computational complexity of planning and online learning [2, 9, 10, 14].”
Value Preserving State-Action Abstractions (2020, background): “Brunskill and Li (2014) study the problem of learning options that improve sample efficiency in lifelong RL.”
Safe Reinforcement Learning (2015, unknown): “There are many PAC RL algorithms and lower bounds on the sample complexity of an arbitrary RL algorithm (Auer et al., 2010, Brunskill and Li, 2014, Guo and Brunskill, 2015, Strehl and Littman, 2005, 2008, Strehl et al., 2009, 2006, Szita and Szepesvári, 2010).”
Towards Safe Policy Improvement for Non-Stationary MDPs (2020, unknown): “While some works for lifelong-reinforcement learning [10, 1, 12, 13] or meta-reinforcement learning [3, 67] do aim to address the problem of non-stationarity, they do not provide any safety guarantees.”
State Abstractions for Lifelong Reinforcement Learning (2018, background): “Of growing interest in the RL literature is lifelong learning, in which an agent must interact with and solve many tasks over the course of a lifetime, as in Brunskill & Li (2014); Isele et al. (2016); Walsh et al. (2006) and Wilson et al. (2007).”
Model Primitive Hierarchical Lifelong Reinforcement Learning (2019, methodology): “Adopting the framework from Brunskill and Li [4], in lifelong RL, the agent receives S,A, initial state distribution ρ0 ∈ Π(S), horizon H , discount factor γ , and an unknown distribution over reward-transition function pairs, D.”
Toward Good Abstractions for Lifelong Learning (2017, background): “The core principle is to augment the agent’s model of the world with a collection of high level actions that probe more deeply into the search space, resulting in targeted exploration, deep planning, or transfer [18, 6, 5, 4].”