QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation

By Dmitry Kalashnikov, A. Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke and 1 more. 2018

1,735 citations recorded by the source corpus.

In this paper, we study the problem of learning vision-based dynamic manipulation skills using a scalable reinforcement learning approach. We study this problem in the context of grasping, a longstanding challenge in robotic manipulation. In contrast to static learning behaviors that choose a grasp point and then execute the desired grasp, our method enables closed-loop vision-based control, whereby the robot continuously updates its grasp strategy based on the most recent observations to optimize long-horizon grasp success. To that end, we introduce QT-Opt, a scalable self-supervised vision-based reinforcement learning framework that can leverage over 580k real-world grasp attempts to train a deep neural network Q-function with over 1.2M parameters to perform closed-loop, real-world grasping that generalizes to 96% grasp success on unseen objects. Aside from attaining a very high success rate, our method exhibits behaviors that are quite distinct from more standard grasping systems: using only RGB vision-based perception from an over-the-shoulder camera, our method automatically learns regrasping strategies, probes objects to find the most effective grasps, learns to reposition objects and perform other non-prehensile pre-grasp manipulations, and responds dynamically to disturbances and perturbations.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Augmented Language Models: a Survey (2023, methodology): “…with an environment, and it has been used for applications such as playing games (Mnih et al., 2015; Silver et al., 2016; Vinyals et al., 2019; Team et al., 2021; Bakhtin et al., 2022) or controlling robots (Gu et al., 2017; Kalashnikov et al., 2018; Akkaya et al., 2019; Lee et al., 2020).”
PaLM-E: An Embodied Multimodal Language Model (2023, positive): “PaLM-E outperforms PaLI (zero-shot), as well as thresholding on value functions trained with QT-OPT (Tab.”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, methodology): “While the reinforcement learning algorithms have been an active area of research for decades, the introduction of effective high-capacity function approximators – deep neural networks – into reinforcement learning, along with effective algorithms for training them, has allowed reinforcement learning”
Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “Q-learning methods train the Q-function by iteratively applying the Bellman optimality operator B ∗ Q ( s , a ) = r ( s , a ) + γ E s (cid:48) ∼ P ( s (cid:48) | s , a ) [max a (cid:48) Q ( s (cid:48) , a (cid:48) )] , and use exact or an approximate maximization scheme, such as CEM [26] to recover ”
Solving Rubik's Cube with a Robot Hand (2019, neutral): “Alternative approaches include using many physical robots simultaneously, in order to be able to collect sufficient experience [36, 57, 49] or leveraging a model-based learning algorithms, which generally possess much more favorable sample complexity characteristics [121].”
D4RL: Datasets for Deep Data-Driven Reinforcement Learning (2020, background): “Moreover, significant efforts have been made to incorporate large-scale datasets into off-policy RL [17, 26, 9], but these works generally use large numbers of robots to collect online interaction during training.”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, background): “References [1] Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, and Sergey Levine.”
Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning (2020, background): “This is especially evident in robotics applications (see e.g. Riedmiller et al. 2018; Haarnoja et al. 2018b; Kalashnikov et al. 2018 for recent examples learning on robots).”
A Framework for Data-Driven Robotics (2019, background): “Among the RL for robotics literature, QT-Opt [15] is the closest approach to ours.”
Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, background): “Recent progress in mastering the games of Go [50] and Poker [43, 10] and advances in robotic control [36, 44, 30] present additional supporting evidence for the enormous potential of deep RL.”
Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction (2019, background): “, [19, 30, 24, 10]) have demonstrated sample-efficient performance on complex tasks in robotics [24] and simulated environments [37].”
Diagnosing Bottlenecks in Deep Q-learning Algorithms (2019, methodology): “In recent years, such methods have been applied to great effect in domains such as playing video games from raw pixels (Mnih et al., 2015) and continuous control in robotics (Kalashnikov et al., 2018).”
Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience (2018, background): “While there have been recent attempts on learning by collecting large scale data directly on real robots [1, 2, 3, 4], such an approach still remains challenging as collecting real world data is prohibitively laborious and expensive.”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, background): “Recent years have seen reinforcement learning (RL) successfully applied to a number of robotics tasks such as in-hand manipulation [1], grasping [2] and door opening [3].”
Towards Characterizing Divergence in Deep Q-Learning (2019, methodology): “DQL algorithms have been applied fruitfully in video games (Mnih et al., 2015), robotics (Kalashnikov et al., 2018; Haarnoja et al., 2018a), and user interactions on social media (Gauci et al., 2018).”
Implicit Behavioral Cloning (2021, background): “) Additionally, critic-only Q-learning [50] similarly defines the action mapping implicitly, but using reinforcement rather than supervised learning.”
EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL (2020, methodology): “QT-OPT (Kalashnikov et al., 2018b) makes use of two iterations of the Cross-Entropy Method (Rubinstein & Kroese, 2013), while CAQL (Ryu et al., 2019) uses Mixed-Integer Programming to find the exact maximizing action while also introducing faster approximate alternatives.”
Benchmarks for Deep Off-Policy Evaluation (2021, background): “…& Joachims, 2015; Jiang & Li, 2015; Wang et al., 2017; Yang et al., 2020) and practical progress (Gilotte et al., 2018; Nie et al., 2019; Kalashnikov et al., 2018) on OPE algorithms has been made in a range of different domains, there are few broadly accepted evaluation tasks that…”
COMBO: Conservative Offline Model-Based Policy Optimization (2021, background): “It has found applications in domains including robotic manipulation [25, 38, 48, 54], NLP [21, 22] and healthcare [52, 62].”