Robobarista: Object Part Based Transfer of Manipulation Trajectories from Crowd-Sourcing in 3D Pointclouds

By Jaeyong Sung, Seok Hyun Jin, Ashutosh Saxena. 2015

71 citations recorded by the source corpus.

There is a large variety of objects and appliances in human environments, such as stoves, coffee dispensers, juice extractors, and so on. It is challenging for a roboticist to program a robot for each of these object types and for each of their instantiations. In this work, we present a novel approach to manipulation planning based on the idea that many household objects share similarly-operated object parts. We formulate the manipulation planning as a structured prediction problem and design a deep learning model that can handle large noise in the manipulation demonstrations and learns features from three different modalities: point-clouds, language and trajectory. In order to collect a large number of manipulation demonstrations for different objects, we developed a new crowd-sourcing platform called Robobarista. We test our model on our dataset consisting of 116 objects with 249 parts along with 250 language instructions, for which there are 1225 crowd-sourced manipulation demonstrations. We further show that our robot can even manipulate objects it has never seen before.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, 3D Vision & Pose, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In 3D Vision & Pose

In Robotics & Control

Citations in the corpus

ROBOTURK: A Crowdsourcing Platform for Robotic Skill Learning through Imitation (2018, background): “Alternatively, teleoperation-based techniques for collecting task demonstrations have been employed for over two decades [23, 24, 25].”
Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-to-End Learning from Demonstration (2017, methodology): “In recent years, LfD was successfully used in the field of robotics for applications as varied as autonomous helicopter maneuvers [1], playing table tennis [2], object manipulation [3], making coffee [4], grasping novel objects [5], carrot grating [6], and eggplant cutting [7].”
Unsupervised Perceptual Rewards for Imitation Learning (2016, background): “Deep learning has been used in a range of applications within robotic perception and learning [29, 23, 6].”
Learning Dexterous Grasping with Object-Centric Visual Affordances (2020, background): “, Rubik’s cube [1]) or by incorporating explicit human demonstrations [46], [12], [39], [15], [57], [47], [38].”
Environment-Driven Lexicon Induction for High-Level Instructions (2015, unknown): “Other works in this area have considered mapping text to a variety of manipulation actions (Sung et al., 2015).”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “Neural networks are also highly effective at combining data from multiple sensor modalities or information sources [119, 233, 236, 370, 369, 240, 173].”
From Language to Goals: Inverse Reinforcement Learning for Vision-Based Instruction Following (2019, background): “Several works adopt a behavioral cloning approach, where the model is trained using supervised learning with language-action sequences pairs (Anderson et al., 2018; Mei et al., 2016; Sung et al., 2015).”
PixL2R: Guiding Reinforcement Learning Using Natural Language by Mapping Pixels to Rewards (2020, methodology): “tructions directly to an action sequence to be executed. Tellex et al. [9] dynamically instantiate a graphical model given a language command, from which a plan for the agent is inferred. Sung et al. [10] learn a neural network to predict relatedness between htrajectory, languageipairs and htrajecto”