Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresight

By Annie Xie, Frederik D. Ebert, Sergey Levine, Chelsea Finn. 2019

95 citations recorded by the source corpus.

Machine learning techniques have enabled robots to learn narrow, yet complex tasks and also perform broad, yet simple skills with a wide variety of objects. However, learning a model that can both perform complex tasks and generalize to previously unseen objects and goals remains a significant challenge. We study this challenge in the context of "improvisational" tool use: a robot is presented with novel objects and a user-specified goal (e.g., sweep some clutter into the dustpan), and must figure out, using only raw image observations, how to accomplish the goal using the available objects as tools. We approach this problem by training a model with both a visual and physical understanding of multi-object interactions, and develop a sampling-based optimizer that can leverage these interactions to accomplish tasks. We do so by combining diverse demonstration data with self-supervised interaction data, aiming to leverage the interaction data to build generalizable models and the demonstration data to guide the model-based RL planner to solve complex tasks. Our experiments show that our approach can solve a variety of complex tool use tasks from raw pixel inputs, outperforming both imitation learning and self-supervised learning individually. Furthermore, we show that the robot can perceive and use novel objects as tools, including objects that are not conventional tools, while also choosing dynamically to use or not use tools depending on whether or not they are required.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

RoboNet: Large-Scale Multi-Robot Learning (2019, background): “The actions are sampled randomly and need to provide sufficient exploration of the state space, which has been explored in prior work [7, 46].”
Emergent Tool Use From Multi-Agent Autocurricula (2019, background): “For example, in [54] a real-world robot uses familiar and novel tools to move objects using model-based RL and imitation learning.”
MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale (2021, background): “…and composite skills has been addressed through hierarchical reinforcement learning with options [75, 7, 14], combining multiple sub-tasks [6, 16, 49, 26, 69, 89], reusing samples between tasks [39], relabeling experience in hindsight [3], introducing demonstrations [58, 81, 29, 41, 70, 66, 67].”
The Tools Challenge: Rapid Trial-and-Error Learning in Physical Problem Solving (2019, background): “Recent work has also investigated combining imitation learning with model learning and policy updates to learn how to use objects as tools from pixel inputs alone (A. Xie et al., 2019), as well as how to use speciﬁc tools to accomplish new tasks with self-supervised objectives (Fang et al., 2018).”
Deep Visual Reasoning: Learning to Predict Action Sequences for Task and Motion Planning from an Initial Scene Image (2020, methodology): “To enable planning with raw sensory input, there are several works that learn a compact representation and its dynamics in sensor space to then apply planning or reinforcement learning (RL) in the learned latent space [3, 50, 17, 22, 47, 15, 29, 38].”
A Review on Deep Learning Techniques for Video Prediction (2020, background): “In the same domain, video prediction has facilitated decision making in vision-based robotic control [6] and motion planning [7], [8] and has provided accurate world models, of high-dimensional environments, to model-based Reinforcement Learning (RL) approaches.”
Parrot: Data-Driven Behavioral Priors for Reinforcement Learning (2020, background): “Other works learn these distributions from data (Xie et al., 2019; Rhinehart et al., 2020) and utilize them for planning at test time to optimize a user-provided cost function.”
COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning (2020, background): “…in a new scenario, and use prior interaction datasets to achieve this ability via model-free RL. Visual foresight [29] and its followups [30, 9, 31, 32] also address temporally extended tasks with large datasets by learning video prediction models, but with signiﬁcantly shorter time horizons…”
Generalization to New Actions in Reinforcement Learning (2020, background): “…been used to transfer policies to new environments (Cobbe et al., 2018; Nichol et al., 2018; Packer et al., 2018), agent morphologies (Wang et al., 2018; Sanchez-Gonzalez et al., 2018; Pathak et al., 2019), and visual inputs for manipulation of unseen tools (Fang et al., 2018; Xie et al., 2019).”
How to train your robot with deep reinforcement learning: lessons we have learned (2021, methodology): “, as a proposal distribution), including demonstrations into the model training dataset can enable a robot to perform complex behaviors, such as using tools (Figure 7), which would be extremely difficult to discover automatically (Xie et al., 2019).”
KETO: Learning Keypoint Representations for Tool Manipulation (2019, methodology): “Manipulation of tool objects can be performed through learning and planning as shown in [16], [52], [18], [57],”
Transformers for One-Shot Visual Imitation (2020, background): “There are myriad ways to introduce goal conditioning, including with the robot’s state [24], ”goal” images of the ﬁnal state [25, 26, 27], natural language [28], and video or images of humans [29, 30].”