Intra-agent speech permits zero-shot task acquisition

By Chen Yan, Federico Carnevale, Petko Georgiev, Adam Santoro, Aurelia Guy, Alistair Muldal, Chia-Chun Hung, Josh Abramson, Timothy P. Lillicrap, Greg Wayne. 2022

10 citations recorded by the source corpus.

Human language learners are exposed to a trickle of informative, context-sensitive language, but a flood of raw sensory data. Through both social language use and internal processes of rehearsal and practice, language learners are able to build high-level, semantic representations that explain their perceptions. Here, we take inspiration from such processes of"inner speech"in humans (Vygotsky, 1934) to better understand the role of intra-agent speech in embodied behavior. First, we formally pose intra-agent speech as a semi-supervised problem and develop two algorithms that enable visually grounded captioning with little labeled language data. We then experimentally compute scaling curves over different amounts of labeled data and compare the data efficiency against a supervised learning baseline. Finally, we incorporate intra-agent speech into an embodied, mobile manipulator agent operating in a 3D virtual world, and show that with as few as 150 additional image captions, intra-agent speech endows the agent with the ability to manipulate and answer questions about a new object without any related task-directed experience (zero-shot). Taken together, our experiments suggest that modelling intra-agent speech is effective in enabling embodied agents to learn new tasks efficiently and without direct interaction experience.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Language and Linguistics, Reinforcement Learning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Chelsea Finn
3. Andy Zeng
4. Brian Ichter
5. Karol Hausman
6. Pierre Sermanet
7. Pieter Abbeel
8. Igor Mordatch

In Computer Vision and Pattern Recognition

1. Sergey Levine
2. Luc Van Gool
3. Li Fei-Fei
4. Trevor Darrell
5. Jitendra Malik
6. Dacheng Tao
7. Xiaogang Wang
8. Andrew Zisserman

In Language and Linguistics

1. Sergey Levine

In Reinforcement Learning

1. Sergey Levine
2. Karol Hausman
3. Chelsea Finn
4. Pierre Sermanet
5. Brian Ichter
6. Pieter Abbeel
7. Igor Mordatch
8. Jonathan Tompson

In Multimodal Learning

1. Sergey Levine
2. Igor Mordatch
3. Jonathan Tompson
4. Brian Ichter
5. Pierre Sermanet
6. Andy Zeng

Citations in the corpus

• Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, background): “Various works have investigated strategies for the challenging problem of fusing vision, language, and control [60, 61, 62, 63, 64, 65, 66].”