Neural Task Graphs: Generalizing to Unseen Tasks From a Single Video Demonstration

By De-An Huang, Suraj Nair, Danfei Xu, Yuke Zhu, Animesh Garg, Li Fei-Fei, Silvio Savarese, Juan Carlos Niebles. 2018

151 citations recorded by the source corpus.

Our goal is to generate a policy to complete an unseen task given just a single video demonstration of the task in a given domain. We hypothesize that to successfully generalize to unseen complex tasks from a single video demonstration, it is necessary to explicitly incorporate the compositional structure of the tasks into the model. To this end, we propose Neural Task Graph (NTG) Networks, which use conjugate task graph as the intermediate representation to modularize both the video demonstration and the derived policy. We empirically show NTG achieves inter-task generalization on two complex tasks: Block Stacking in BulletPhysics and Object Collection in AI2-THOR. NTG improves data efficiency with visual input as well as achieve strong generalization without the need for dense hierarchical supervision. We further show that similar performance trends hold when applied to real-world data. We show that NTG can effectively predict task structure on the JIGSAWS surgical dataset and generalize to unseen tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, methodology): “Machine learning has recently been applied to enable abstract task specification, allow general primitives, or relax constraints [67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78].”
Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations (2020, methodology): “The task can be represented as a full video [41], a sequence of image keyframes [16], or a trajectory in state space [10], while the training objective can”
BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning (2022, background): “Multiple prior imitation learning works have achieved different forms of generalization, includ56 ing one-shot generalization to novel objects [1, 2, 3, 4, 18], to novel object configurations [19], 57 and to novel goal configurations [6, 7, 20], as well as zero-shot generalization to new objects [5]”
Discovering Motor Programs by Recomposing Demonstrations (2020, background): “, 2018), or as task graphs (Huang et al., 2019).”
Learning About Objects by Learning to Interact with Them (2020, background): “Our agent does not interact with the physical world, but instead, is instantiated within AI2THOR [30], a virtual near photo-realistic environment with a physics engine, that is often used to study embodied agents [27, 20, 28, 65, 22].”
Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence (2020, background): “While machine learning enables us to push the state-of-the-art in many fields such as natural language processing [116,123,163,164] and computer vision [131,165–167], certain applications involve sensitive data that demands responsible treatment.”
Beyond imitation: Zero-shot task transfer on robots by learning concepts as cognitive programs (2018, background): “Recent work (3) has sought to build an intermediate representation from a demonstration that would allow generalization to more settings but still relied on having a demonstration available for each variation of a task, in addition to having a large annotated background set for the family of variati”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “Other recent approaches have relied on observations only, learning to segment videos of multi-step tasks into composable primitives [127, 433, 165, 424].”
Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Tasks (2021, background): “Other works have also explored learning hierarchical policies on top of hand-designed skills [70, 71, 72].”
Procedure Planning in Instructional Videos (2019, background): “Similar ideas have been leveraged in the conjugate task graph approaches [9, 11] to improve the planning efficiency.”
Deep reinforcement learning with relational inductive biases (2018, background): “, 2018), and structured behavior representation (Huang et al., 2018), so that the structured internal representations and patterns of reasoning can translate into more structured behaviors.”
Learning Robot Skills with Temporal Variational Inference (2020, methodology): “But even using demonstrations, approaches such as Xu et al. (2018); Huang et al. (2019) require heavy supervision such as segmentation of demonstration data.”
A Behavioral Approach to Visual Navigation with Graph Localization Networks (2019, background): “This idea can be related to motor control through primitives in manipulation [12, 21, 51], but our goal is not to solve new tasks given demonstrations of an ordered set of actions.”
Learning to Synthesize Programs as Interpretable and Generalizable Policies (2021, background): “[36] De-An Huang, Suraj Nair, Danfei Xu, Yuke Zhu, Animesh Garg, Li Fei-Fei, Silvio Savarese, and Juan Carlos Niebles.”
Synthesizing Environment-Aware Activities via Activity Sketches (2019, background): “[9] represents such goals as a graph, with nodes being actions and edges being precondition states, while we propose program representations.”
Parrot: Data-Driven Behavioral Priors for Reinforcement Learning (2020, background): “, 2020) and meta-imitation methods (Duan et al., 2017; Finn et al., 2017c; Huang et al., 2018; James et al., 2018; Paine et al., 2018; Yu et al., 2018; Huang et al., 2019; Zhou et al., 2020) also seek to speed up learning for new tasks by leveraging experience from previously seen tasks.”