TidyBot: Personalized Robot Assistance with Large Language Models

By Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, Thomas A. Funkhouser. IEEE/RJS International Conference on Intelligent RObots and Systems, 2023

451 citations recorded by the source corpus.

For a robot to personalize physical assistance effectively, it must learn user preferences that can be generally reapplied to future scenarios. In this work, we investigate personalization of household cleanup with robots that can tidy up rooms by picking up objects and putting them away. A key challenge is determining the proper place to put each object, as people's preferences can vary greatly depending on personal taste or cultural background. For instance, one person may prefer storing shirts in the drawer, while another may prefer them on the shelf. We aim to build systems that can learn such preferences from just a handful of examples via prior interactions with a particular person. We show that robots can combine language-based planning and perception with the few-shot summarization capabilities of large language models (LLMs) to infer generalized user preferences that are broadly applicable to future interactions. This approach enables fast adaptation and achieves 91.2% accuracy on unseen objects in our benchmark dataset. We also demonstrate our approach on a real-world mobile manipulator called TidyBot, which successfully puts away 85.0% of objects in real-world test scenarios.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/e7a4e987dc250ac6a016ee2011bc7a552cfa8e8a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control (2023, background): “Other works have incorporated pre-trained language models, often either as an instruction encoder [63, 64, 43, 41, 44, 1, 65] or for high-level planning [66, 19, 17, 67, 68, 69].”
ExpeL: LLM Agents Are Experiential Learners (2023, background): “LLM agents have been instantiated in many areas such as robotics (Ha, Florence, and Song 2023; Brohan et al. 2023; Mu et al. 2023; Mirchandani et al. 2023; Wu et al. 2023), natural sciences (Bran et al.”
BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset (2023, background): “The recent advent of large language models (LLMs) [1, 2, 3, 4, 5] promises vast transformative potential across multiple sectors, from healthcare [6, 7, 8] and education [9, 10, 11] to robotics [12, 13, 14] and commerce [15].”
A Comprehensive Overview of Large Language Models (2023, unknown): “The work in [187] focuses on personalizing robot household cleanup tasks.”
Text2Motion: from natural language instructions to feasible plans (2023, methodology): “Several recent works [3– 6] capitalize on their ability to perform task planning for robot systems without needing to manually specify symbolic planning domains.”
A survey on large language model based autonomous agents (2023, unknown): “ChemCrow [75], Boiko et al. [115], Grossmann et al. [122] Natural Science Education ChemCrow [75], CodeHelp [120], Boiko et al. [115], MathA-gent [117], Drori et al. [118] Engineering”
Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation (2024, background): “In order to tackle the exploration problem in high-dimensional state and action spaces of mobile manipulation tasks, prior works use predefined skill primitives [86, 91, 92], reinforcement learning with decomposed action spaces [38, 48, 58, 94, 101], or whole-body control objectives [36, 42, 99].”
RT-1: Robotics Transformer for Real-World Control at Scale (2022, methodology): “via reinforcement-learning [32, 26, 67, 37, 65, 20, 18, 11, 69], and recently through foundation models [3, 38, 66].”
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners (2023, background): “Large language models have shown a wide range of capabilities: reasoning [19, 20], logic [21, 22], math [23], physics [24, 25], high-level planning [1, 26, 27, 28, 29, 30] with language feedback [31, 2], and writing robot code [5, 32, 33].”