Generalizing Skills with Semi-Supervised Reinforcement Learning

By Chelsea Finn, Tianhe Yu, Justin Fu, Pieter Abbeel, Sergey Levine. 2016

71 citations recorded by the source corpus.

Deep reinforcement learning (RL) can acquire complex behaviors from low-level inputs, such as images. However, real-world applications of such methods require generalizing to the vast variability of the real world. Deep networks are known to achieve remarkable generalization when provided with massive amounts of labeled data, but can we provide this breadth of experience to an RL agent, such as a robot? The robot might continuously learn as it explores the world around it, even while deployed. However, this learning requires access to a reward function, which is often hard to measure in real-world domains, where the reward could depend on, for example, unknown positions of objects or the emotional state of the user. Conversely, it is often quite practical to provide the agent with reward functions in a limited set of situations, such as when a human supervisor is present or in a controlled setting. Can we make use of this limited supervision, and still benefit from the breadth of experience an agent might collect on its own? In this paper, we formalize this problem as semisupervised reinforcement learning, where the reward function can only be evaluated in a set of "labeled" MDPs, and the agent must generalize its behavior to the wide range of states it might encounter in a set of "unlabeled" MDPs, by using experience from both settings. Our proposed method infers the task objective in the unlabeled MDPs through an algorithm that resembles inverse RL, using the agent's own prior experience in the labeled MDPs as a kind of demonstration of optimal behavior. We evaluate our method on challenging tasks that require control directly from images, and show that our approach can improve the generalization of a learned deep neural network policy by using experience for which no reward function is available. We also show that our method outperforms direct supervised learning of the reward.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Deep Reinforcement Learning from Human Preferences (2017, background): “, 2017), semi-supervised skill generalization (Finn et al., 2017), and bootstrapping RL from demonstrations (Silver et al.”
DARLA: Improving Zero-Shot Transfer in Reinforcement Learning (2017, background): “…source and target domain representations by utilising observation and reward signals from both domains (Tzeng et al., 2016; Daftry et al., 2016; Parisotto et al., 2015; Guez et al., 2012; Talvitie & Singh, 2007; Niekum et al., 2013; Gupta et al., 2017; Finn et al., 2017; Rajendran et al., 2017).”
Taskonomy: Disentangling Task Transfer Learning (2018, methodology): “as employed in reinforcement learning [19, 29, 26], optimization [2, 79, 46], or certain architectural mechanisms [25, 28, 84, 62].”
Building Generalizable Agents with a Realistic and Rich 3D Environment (2018, background): “Finn et al. (2017b) study the generalization skills of an agent across similar task conﬁgurations while rewards are provided only for a subset of tasks.”
Learning to Generalize: Meta-Learning for Domain Generalization (2017, neutral): “…in its operating environment (MDP) – without being allowed any rewarded ( ≈ supervised domain adaptation (Finn, Abbeel, and Levine 2017; Ammar et al. 2014)), or un-rewarded ( ≈ unsupervised domain adaptation (Finn et al. 2017; Ammar et al. 2015b)) trials in the target environment for adaptation.”
Learning Robust Rewards with Adversarial Inverse Reinforcement Learning (2017, background): “There are many scenarios where IRL may be preferred over direct imitation learning, such as re-optimizing a reward in novel environments (Finn et al., 2017) or to infer an agent’s intentions, but IRL methods have not been shown to scale to the same complexity of tasks as direct imitation learning.”
Benchmark Environments for Multitask Learning in Continuous Domains (2017, methodology): “…environment with an added image state space of the environment; jumping over a wall using a model similar to the OpenAI Half-Cheetah environment (Finn et al., 2016); varying the gravity of various standard OpenAI Gym benchmark environments (Reacher, Hopper, Humanoid, HalfCheetah) and…”
Learning to Learn: Meta-Critic Networks for Sample Efficient Learning (2017, background): “…advantages: it allows application to RL, which is not non-trivial for existing weight-synthesis methods oriented at SL; it allows learning from unlabelled data / unrewarded interactions with the environment [10]; and it does not constrain the meta and target networks to have matching architectures.”
PERCEPTUAL GOAL SPECIFICATIONS FOR REINFORCEMENT LEARNING (2017, background): “[19] described an approach that allowed generalizing skills learned from “labeled” MDPs with known reward functions to similar “unlabeled” MDPs with unspecified reward functions.”
Decoupling Dynamics and Reward for Transfer Learning (2018, background): “Finn et al. (2016) is also a semi-supervised learning method that incorporates unlabeled data, but infer labels for better generalization as opposed to transfer to different reward functions.”
Deep Reinforcement Learning: An Overview (2017, background): “• semi-supervised RL, e.g., Finn et al. (2017) • deep symbolic RL, Garnelo et al. (2016) • intrinsic motivation, e.g., Stadie et al. (2015); Kulkarni et al. (2016); Oudeyer et al. (2016) • hyperparameter learning, e.g. Andrychowicz et al. (2016) • information extraction, e.g., Narasimhan et al.…”
Deep Reinforcement Learning (2018, background): “et al.(2018),Merel et al.(2017),Stadie et al. (2017),Su et al.(2016),Wang et al.(2017), and,Zheng et al.(2018b). We discuss robotics with imitation learning in Section16.2, includingDuan et al.(2017);Finn et al. (2017c);Yu et al.(2018);Finn et al.(2016b).Hu et al.(2018d) leverage IRL techniques to l”
DQN-TAMER: Human-in-the-Loop Reinforcement Learning with Intractable Feedback (2018, background): “Robots with such a mechanism would be capable of lifelong learning [21], [22] after deployment in the real world.”