MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale

By Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, Karol Hausman. 2021

331 citations recorded by the source corpus.

General-purpose robotic systems must master a large repertoire of diverse skills to be useful in a range of daily tasks. While reinforcement learning provides a powerful framework for acquiring individual behaviors, the time needed to acquire each skill makes the prospect of a generalist robot trained with RL daunting. In this paper, we study how a large-scale collective robotic learning system can acquire a repertoire of behaviors simultaneously, sharing exploration, experience, and representations across tasks. In this framework new tasks can be continuously instantiated from previously learned tasks improving overall performance and capabilities of the system. To instantiate this system, we develop a scalable and intuitive framework for specifying new tasks through user-provided examples of desired outcomes, devise a multi-robot collective learning system for data collection that simultaneously collects experience for multiple tasks, and develop a scalable and generalizable multi-task deep reinforcement learning method, which we call MT-Opt. We demonstrate how MT-Opt can learn a wide range of skills, including semantic picking (i.e., picking an object from a particular category), placing into various fixtures (e.g., placing a food item onto a plate), covering, aligning, and rearranging. We train and evaluate our system on a set of 12 real-world tasks with data collected from 7 robots, and demonstrate the performance of our system both in terms of its ability to generalize to structurally similar new tasks, and acquire distinct new tasks more quickly by leveraging past experience. We recommend viewing the videos at https://karolhausman.github.io/mt-opt/

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Mechanical Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Mechanical Engineering

In Reinforcement Learning

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, methodology): “Given Qπ(s, a) with action a and state s, value v(s) = maxaQπ(s, a) is found through optimization via the cross entropy method, similar to MT-Opt.”
Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “Engineered success detectors can operate on ground-truth state in simulation, while learned success detectors can be trained on real examples of successes and failures in the real world [84, 85, 86, 87, 88].”
CLIPort: What and Where Pathways for Robotic Manipulation (2021, background): “Recently, a number of end-to-end frameworks have been proposed for vision-based manipulation [2, 3, 4, 5].”
BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning (2022, background): “This includes works that generalize skills to novel objects [31, 32, 33, 34, 35], to novel environments [36], from simulation to reality [37, 38, 39, 40, 41], and to new manipulation skills and objects [42, 43, 44, 45].”
Skill-based Meta-Reinforcement Learning (2022, background): “In recent years, deep reinforcement learning methods have achieved impressive results in robot learning (Gu et al., 2017; Andrychowicz et al., 2020; Kalashnikov et al., 2021).”
Text2Motion: from natural language instructions to feasible plans (2023, background): “…we distill their returns into stochastic policies π through the maximum-entropy update [74]: Instead of evaluating the policies on D v , which contains states for which no feasible action exists, the policies are synchronously evaluated in an environment that exhibits only feasible states.”
Efficiently Identifying Task Groupings for Multi-Task Learning (2021, background): “Robotic arms are asked to concurrently learn how to pick, place, cover, align, and rearrange various objects to improve learning efficiency [25], and online movie recommendation systems model multiple engagement metrics to facilitate low-latency personalized recommendations [13].”
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0 (2023, unknown): “…we directly train a policy on X-embodiment data, without any mechanisms to reduce the embodiment gap, and observe positive transfer by leveraging that data. spanning grasping [60–71], pushing interactions [23, 72–74], sets of objects and models [75–85], and teleoperated demonstrations [8, 86–95].”
DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset (2024, unknown): “Broadly, these can be categorized into datasets collected autonomously via scripted and semi-random behaviors or learned agents [3, 8, 20, 26, 27, 32, 40], and datasets collected via human teleoperation [1, 2, 13, 14, 24, 36, 50, 59].”
What Matters in Language Conditioned Robotic Imitation Learning Over Unstructured Data (2022, methodology): “we analyze the influence of using the 7-DoF proprioceptive information as input for both the plan encodings and conditioning the policy, as many works report improved performance from it [1], [4], [7].”
Temporal Difference Learning for Model Predictive Control (2022, methodology): “Kalashnikov et al. (2018); Shao et al. (2020); Kalashnikov et al. (2021) also learn Q θ , but replace or augment π θ with model-free CEM.”
Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets (2021, methodology): “Most existing robot datasets, such as MIME [16], DAML [10], RoboTurk [17], [18], and many others [19], [20], [21], [22], [23], [24], [5] only feature a single domain, making them difﬁcult to use for boosting the generalization in other domains.”
RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots (2024, methodology): “It has also been used to learn from large offline robot manipulation datasets [19, 3, 11, 23, 22].”
PI-QT-Opt: Predictive Information Improves Multi-Task Robotic Reinforcement Learning at Scale (2022, methodology): “Some approaches to real-world robot learning for manipulation utilize RL with real robot data collection or simulation with domain randomization [15, 29, 12, 13, 38, 39, 40, 41, 42, 43], while other approaches focus on imitation learning from expert demonstrations [27, 44, 45, 46, 47, 48].”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “VLAs encode instructions using the token embeddings of their LLMs and employ various strategies to align vision and language embeddings, including approaches like BLIP-2 [7] and LLaVA [8].”
Instruction-driven history-aware policies for robotic manipulations (2022, background): “Different multi-task approaches are explored by discovering which tasks should be trained together [15, 34], determining shared features across tasks [35, 36], meta-learning [37, 38, 39], goal-conditioned learning [40, 41], or inverse reinforcement learning [42].”
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation (2021, methodology): “While many approaches have been taken to task-specification, including task IDs [54, 55], robot and human demonstrations [56, 57, 58], and meta-learning from rewards [59], a common approach is goalconditioned learning [60, 61, 2, 1], where an agent learns to reach particular goal states or distribut”
CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks (2021, background): “Moreover, multitask learning approaches traditionally assume that tasks are specified to the agent at test time via mechanisms such as goal images [1] and one-hot skill selectors [2], [3] that are not practical for non-expert users to instruct robots in everyday real-world settings.”