Do As I Can, Not As I Say: Grounding Language in Robotic Affordances
By Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog and 33 more. 2022
3,376 citations recorded by the source corpus.
Large language models can encode a wealth of semantic knowledge about the world. Such knowledge could be extremely useful to robots aiming to act upon high-level, temporally extended instructions expressed in natural language. However, a significant weakness of language models is that they lack real-world experience, which makes it difficult to leverage them for decision making within a given embodiment. For example, asking a language model to describe how to clean a spill might result in a reasonable narrative, but it may not be applicable to a particular agent, such as a robot, that needs to perform this task in a particular environment. We propose to provide real-world grounding by means of pretrained skills, which are used to constrain the model to propose natural language actions that are both feasible and contextually appropriate. The robot can act as the language model's"hands and eyes,"while the language model supplies high-level semantic knowledge about the task. We show how low-level skills can be combined with large language models so that the language model provides high-level knowledge about the procedures for performing complex and temporally-extended instructions, while value functions associated with these skills provide the grounding necessary to connect this knowledge to a particular physical environment. We evaluate our method on a number of real-world robotic tasks, where we show the need for real-world grounding and that this approach is capable of completing long-horizon, abstract, natural language instructions on a mobile manipulator. The project's website and the video can be found at https://say-can.github.io/.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, LLM Agents & Tool Use, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Mohit Bansal
- Jianfeng Gao
- Yoshua Bengio
- Quoc V. Le
- Percy Liang
- Luke S. Zettlemoyer
- Chelsea Finn
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Jitendra Malik
- Xiaogang Wang
- Ming-Hsuan Yang
- Andrew Zisserman
- Li Fei-Fei
In LLM Agents & Tool Use
- Dieter Fox
- Lucas Manuelli
- Mohit Shridhar
- Igor Mordatch
- Brian Ichter
- Pieter Abbeel
- Andy Zeng
- Song-Chun Zhu
In Reinforcement Learning
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Ofir Nachum
- Shixiang Shane Gu
- Aviral Kumar
- Igor Mordatch
- Nicolas Manfred Otto Heess
Citations in the corpus
- Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, methodology): “Finally, the SayCan dataset (Ahn et al., 2022) involves mapping a natural language instruction to a sequence of robot actions from a discrete set.”
- ReAct: Synergizing Reasoning and Acting in Language Models (2022, neutral): “In SayCan, LLMs were prompted to directly predict possible actions a robot can take, which is then reranked by an affordance model grounded on the visual environments for final prediction.”
- Emergent Abilities of Large Language Models (2022, neutral): “The TruthfulQA benchmark (Lin et al., 2021) showed that GPT-3 models were more likely to mimic human falsehoods as they got larger, though Rae et al. (2021) later showed on a multiple-choice version that scaling Gopher to 280B enabled emergent performance substantially better than random.”
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “…on top of a general-domain instruction-tuned model, and collect a custom instruction-tuning mixture for their use, e.g., with multiple modalities (Ahn et al., 2022; Huang et al., 2022; Xu et al., 2022) or special domains such as science and medicine (Lewkowycz et al., 2022; Singhal et al., 2022).”
- Language Models (Mostly) Know What They Know (2022, methodology): “Language models finetuned to perform non-language tasks [Ahn et al., 2022, Dinh et al., 2022] might provide an interesting test-bed for honesty in the future.”
- Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, methodology): “Similar to ours are recent task planning approaches that leverage pre-trained autoregressive LLMs to decompose abstract, high-level instructions into a sequence of low-level steps executable by an agent [20, 21] in a zero-shot manner.”
- Adversarial Training for High-Stakes Reliability (2022, background): “Advances in deep learning have led to increasingly powerful AI systems, for example in sequential decision making [1, 2, 3, 4], robotics [5, 6], and language modeling and text-based reasoning [7, 8, 9, 10, 11].”
- MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action (2023, background): “Nevertheless, LLMs for reasoning [17,31] and LLMs for action [1, 15, 22] , when used independently, fail to solve complex tasks that require breaking down the problem via reasoning and solving sub-steps via planned actions.”
- PaLM-E: An Embodied Multimodal Language Model (2023, methodology): “…semantic similarity between an LLM’s generation and an eligible set of instructions (Huang et al., 2022b), incorporating affordance functions (Ahn et al., 2022), visual feedback (Huang et al., 2022c), generating world models (Nottingham et al., 2023; Zellers et al., 2021a), planning over…”
- Language Models are Multilingual Chain-of-Thought Reasoners (2022, background): “Ahn et al. (2022) apply chain-of-thought prompting in robotics scenarios, including a multilingual setting.”
- Inductive biases for deep learning of higher-level cognition (2020, neutral): “with efficient transfer learning) (Ravi and Larochelle, 2016; Wang et al., 2016; Finn et al., 2017; Cabi et al., 2019; Jang et al., 2022; Reed et al., 2022; Ahn et al., 2022; Brown et al., 2020; Alayrac et al., 2022; Borgeaud et al., 2022; Chowdhery et al., 2022; Sanh et al., 2021; Lu et al., 2022; ”