Building Cooperative Embodied Agents Modularly with Large Language Models

By Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Josh Tenenbaum, Tianmin Shu, Chuang Gan. International Conference on Learning Representations, 2023

315 citations recorded by the source corpus.

In this work, we address challenging multi-agent cooperation problems with decentralized control, raw sensory observations, costly communication, and multi-objective tasks instantiated in various embodied environments. While previous research either presupposes a cost-free communication channel or relies on a centralized controller with shared observations, we harness the commonsense knowledge, reasoning ability, language comprehension, and text generation prowess of LLMs and seamlessly incorporate them into a cognitive-inspired modular framework that integrates with perception, memory, and execution. Thus building a Cooperative Embodied Language Agent CoELA, who can plan, communicate, and cooperate with others to accomplish long-horizon tasks efficiently. Our experiments on C-WAH and TDW-MAT demonstrate that CoELA driven by GPT-4 can surpass strong planning-based methods and exhibit emergent effective communication. Though current Open LMs like LLAMA-2 still underperform, we fine-tune a CoELA with data collected with our agents and show how they can achieve promising performance. We also conducted a user study for human-agent interaction and discovered that CoELA communicating in natural language can earn more trust and cooperate more effectively with humans. Our research underscores the potential of LLMs for future research in multi-agent cooperation. Videos can be found on the project website https://vis-www.cs.umass.edu/Co-LLM-Agents/.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/587352c3b95c90de6d37f061c8e117f42be0b575

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

MetaGPT: Meta Programming for Multi-Agent Collaborative Framework (2023, background): “Zhang et al. [8] built a framework for collaboration between agents that enables verbal communication, collaboration, and improved efficiency.”
AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors in Agents (2023, unknown): “To facilitate effective decision-making, previous research has investigated the impact of different communication structures among agents (Chan et al., 2023; Zhang et al., 2023b; Wu et al., 2023).”
Large Language Model based Multi-Agents: A Survey of Progress and Challenges (2024, methodology): “Different from Agent4Rec work, [ Zhang et al. , 2023e ] treats both users and items as agents, optimizing them collectively to reflect and adjust to real-world interaction disparities.”
The rise and potential of large language model based agents: a survey (2025, unknown): “Recent research on simulated societies has followed two primary lines, namely, exploring the boundaries of the collective intelligence capabilities of LLM-based agents [90, 115, 327, 330, 332] and using them to accelerate discoveries in the social sciences [16,438,439].”
Empowering Biomedical Discovery with AI Agents (2024, unknown): “An instance of this is Generative agents [62], which create interactive environments where agents mimic human behavior and interact using natural language.”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “By integrating ReCon++ with LLaMA, ShapeLLM improves embodied interaction performance on their newly proposed 3D benchmark, 3D MM-Vet.”
A survey on large language model based autonomous agents (2023, unknown): “Common metrics include success rate [156, 151, 125, 90], reward/score [156, 151, 99], coverage [161], and accuracy [113, 1, 61].”