Programmable Agents

By Misha Denil, Sergio Gomez Colmenarejo, Serkan Cabi, David Saxton, Nando de Freitas. 2017

43 citations recorded by the source corpus.

We build deep RL agents that execute declarative programs expressed in formal language. The agents learn to ground the terms in this language in their environment, and can generalize their behavior at test time to execute new programs that refer to objects that were not referenced during training. The agents develop disentangled interpretable representations that allow them to generalize to a wide variety of zero-shot semantic tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Code Generation & Program Synthesis

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Code Generation & Program Synthesis

Citations in the corpus

Graph Attention Networks (2017, methodology): “Similarly, our proposed attention model can be connected to the works by Duan et al. (2017) and Denil et al. (2017), which use a neighborhood attention operation to compute attention coefﬁcients between different objects in an environment.”
Learning with Latent Language (2017, neutral): “Structured language-like annotations have been used to improve learning of generalizable structured policies (Oh et al., 2017; Andreas et al., 2017; Denil et al., 2017).”
Neural Logic Machines (2019, methodology): “Examples include OO-MDP (Diuk et al., 2008; Kansky et al., 2017), symbolic models for learning in interactive domains (Pasula et al., 2007), structured task deﬁnition by object-oriented instructions (Denil et al., 2017), and structured policy learning (Garnelo et al., 2016).”
Embodied Question Answering (2017, background): “The form and structure of these goal speciﬁca-tions range from declarative programs [26], to simple tem-plated commands [27,28], to free-form natural language instructions [29,30].”
Relational inductive biases, deep learning, and graph networks (2018, methodology): “…hybrids of structure-based methods and deep learning (e.g., Reed and De Freitas, 2016; Garnelo et al., 2016; Ritchie et al., 2016; Wu et al., 2017; Denil et al., 2017; Hudson and Manning, 2018), we see great promise in synthesizing new techniques by drawing on the full AI toolkit and marrying the…”
Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions (2018, background): “…of neural networks with the compositionality of symbolic programs in modeling physics (Battaglia et al., 2016; Chang et al., 2016), playing games (Denil et al., 2017; Kansky et al., 2017), learning algorithms (Bošnjak et al., 2017; Cai et al., 2017; Li et al., 2016; Reed & De Freitas, 2015),…”
Relational Deep Reinforcement Learning (2018, background): “This mechanism has parallels with graph neural networks and, more generally, message passing computations [27, 28, 29, 12, 30].”
Deformable ConvNets V2: More Deformable, Better Results (2018, background): “Modules are ﬁrst proposed in natural language processing [14, 15, 4, 36] and physical system modeling [3, 38, 23, 35, 10, 32].”
LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation (2021, methodology): “Finally, research has framed the task-learning problem as one of program synthesis [37] or program-guided learning [38], [39], [40], [41].”
Composable Planning with Attributes (2018, background): “The Programmable Agents of (Denil et al., 2017) put the notions of objects and attributes (as in relational MDP) into an end-to-end differentiable neural architecture.”
Learning to Follow Language Instructions with Adversarial Reward Induction (2018, background): “[14, 20, 11, 6, 9, 31] consider instruction-following in 2D or 3D environments and reward the agent for arriving at the correct location or object.”
Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotation (2021, background): “Many prior works have studied this problem in the context of instruction following, where an agent aims to complete a task specified by formal language/programs [13, 14, 15, 16, 17, 18] or natural language [10, 11, 19, 20].”
LTL2Action: Generalizing LTL Instructions for Multi-Task RL (2021, background): “Examples of such languages include policy sketches (Andreas et al., 2017), task graphs (Sohn et al., 2018), procedural programs (Sun et al., 2019), declarative programs (Denil et al., 2017), reward machines (Toro Icarte et al., 2020), and temporal logic (Leon et al., 2020).”
Learning Compositional Neural Programs with Recursive Tree Search and Planning (2019, background): “The user specifying to the agent what to do, and not how to do it is reminiscent of programmable agents Denil et al. [2017] and declarative vs imperative programming.”
Continual and Multi-task Reinforcement Learning With Shared Episodic Memory (2019, methodology): “As a baseline we used a batch multi-task A2C ( Multitask-A2C ) with weight sharing and task parametrization (Chaplot et al., 2017; Denil et al., 2017; Florensa et al., 2017; Dosovitskiy & Koltun, 2016).”
How Attentive are Graph Attention Networks? (2021, background): “The GAT formulation of Veličković et al. (2018) rose as the most popular framework for attentional GNNs, thanks to its simplicity, generality, and applicability beyond reinforcement learning (Denil et al., 2017; Duan et al., 2017).”
Compositional Imitation Learning: Explaining and executing one task at a time (2018, background): “An important beneﬁt of compositional sub-sequence representations is combinatorial generalization to never-before-seen conjunctions [6].”
Explainability in Deep Reinforcement Learning (2020, background): “Indeed, the explanation provided by the graph in bold arrows is: "Because it is more desirable to do action build_supply_depot ( A s ) to have more Supply Depots as the goal is to have more Destroyed Units ( D u ) and Destroyed Buildings ( D b )" .”