Beyond imitation: Zero-shot task transfer on robots by learning concepts as cognitive programs

By Miguel L'azaro-Gredilla, Dianhuan Lin, J. Swaroop Guntupalli, Dileep George. 2018

84 citations recorded by the source corpus.

Programs on a visual cognitive computer learned underlying concepts and generalized well to new settings without explicit demonstrations. Humans can infer concepts from image pairs and apply those in the physical world in a completely different setting, enabling tasks like IKEA assembly from diagrams. If robots could represent and infer high-level concepts, then it would notably improve their ability to understand our intent and to transfer tasks between different environments. To that end, we introduce a computational framework that replicates aspects of human concept learning. Concepts are represented as programs on a computer architecture consisting of a visual perception system, working memory, and action controller. The instruction set of this cognitive computer has commands for parsing a visual scene, directing gaze and attention, imagining new objects, manipulating the contents of a visual working memory, and controlling arm movement. Inferring a concept corresponds to inducing a program that can transform the input to the output. Some concepts require the use of imagination and recursion. Previously learned concepts simplify the learning of subsequent, more elaborate concepts and create a hierarchy of abstractions. We demonstrate how a robot can use these abstractions to interpret novel concepts presented to it as schematic images and then apply those concepts in very different situations. By bringing cognitive science ideas on mental imagery, perceptual symbols, embodied cognition, and deictic mechanisms into the realm of machine learning, our work brings us closer to the goal of building robots that have interpretable representations and common sense.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

COBRA: Data-Efficient Model-Based RL through Unsupervised Object Discovery and Curiosity-Driven Exploration (2019, methodology): “However, most existing approaches require hand-crafting object representations or their relations (Cobo et al., 2013; Diuk et al., 2008; Garnelo et al., 2016; Lázaro-Gredilla et al., 2019).”
DreamCoder: bootstrapping inductive program synthesis with wake-sleep library learning (2021, methodology): “All such systems predict a distribution over programs conditioned on a specification, ranging from unigram distributions [34], to bigram-like ones [29, 30], to neural autoregressive models [12].”
Leveraging Language to Learn Program Abstractions and Search Heuristics (2021, background): “…et al., 2016; Devlin et al., 2017), program libraries, or both jointly from data (Shin et al., 2019; Dumancić & Cropper; Ellis et al., 2021; 2020; Lázaro-Gredilla et al., 2019), but even the current best learning approaches can be computationally inefficient (often requiring upwards of…”
Learning to Synthesize Programs as Interpretable and Generalizable Policies (2021, background): “[20] Miguel Lázaro-Gredilla, Dianhuan Lin, J Swaroop Guntupalli, and Dileep George.”