Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models

By Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Jianfeng Gao. 2023

483 citations recorded by the source corpus.

Large language models (LLMs) have achieved remarkable progress in solving various natural language processing tasks due to emergent reasoning abilities. However, LLMs have inherent limitations as they are incapable of accessing up-to-date information (stored on the Web or in task-specific knowledge bases), using external tools, and performing precise mathematical and logical reasoning. In this paper, we present Chameleon, an AI system that mitigates these limitations by augmenting LLMs with plug-and-play modules for compositional reasoning. Chameleon synthesizes programs by composing various tools (e.g., LLMs, off-the-shelf vision models, web search engines, Python functions, and heuristic-based modules) for accomplishing complex reasoning tasks. At the heart of Chameleon is an LLM-based planner that assembles a sequence of tools to execute to generate the final response. We showcase the effectiveness of Chameleon on two multi-modal knowledge-intensive reasoning tasks: ScienceQA and TabMWP. Chameleon, powered by GPT-4, achieves an 86.54% overall accuracy on ScienceQA, improving the best published few-shot result by 11.37%. On TabMWP, GPT-4-powered Chameleon improves the accuracy by 17.0%, lifting the state of the art to 98.78%. Our analysis also shows that the GPT-4-powered planner exhibits more consistent and rational tool selection via inferring potential constraints from instructions, compared to a ChatGPT-powered planner. The project is available at https://chameleon-llm.github.io.

Subfields: Artificial Intelligence, Information Systems, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

A Comprehensive Overview of Large Language Models (2023, unknown): “The initial work on alignment, such as InstructGPT [76] aligns GPT-3 using a 3-step approach, instruction-tuning, reward modeling, and fine-tuning with reinforcement learning (RL).”
VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use (2023, background): “One approach focuses on leveraging existing pretrained task-specific tools such as image captioners [15], object detectors [61] and text-to-image generators [62] by either creating multimodal prompt interfaces [42, 63] or by executing LLM-generated programs [64, 65, 66].”
Not What You've Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection (2023, unknown): “Recently, numerous methods have been proposed to augment LLMs with tools and APIs [28, 48, 53, 70] by letting the LLM itself infer which API to call and its arguments.”
TheoremQA: A Theorem-driven Question Answering dataset (2023, methodology): “Since current LLMs don’t support such multimodal inputs, we propose a solution: to employ captions in a manner similar to Chameleon(Lu et al., 2023a).”
Large Language Models as Tool Makers (2023, methodology): “Various studies (Liu et al., 2023; Parisi et al., 2022; Schick et al., 2023; Shen et al., 2023; Lu et al., 2023; Paranjape et al., 2023; Liang et al., 2023) have demonstrated that supplementing LLMs with tools, such as calculators, search engines, translation systems, calendars, or even API calls…”
LayoutGPT: Compositional Visual Planning and Generation with Large Language Models (2023, background): “With the strong generalization ability of contemporary LLMs, recent work attempts to adapt the power of LLMs on multimodal tasks [31, 55].”
A Survey on Multimodal Large Language Models for Autonomous Driving (2023, methodology): “…of multimodal data [38, 52, 101, 102, 196]; Multimodal Chain of Thought, which enables the model to maintain a chain of thought across different modalities [43,54,146,206]; and LLM-Aided Visual Reasoning (LAVR), which uses LLMs to aid in visual reasoning tasks [52,101,154,178,188,196, 205, 211].”
V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs (2023, methodology): “LLM-tool-using systems or LLM-based agents treat the LLM as a black box and give them access to some vision expert systems to perform certain vision-language tasks through reasoning [14, 31, 53, 54, 56, 62].”
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, background): “Our analysis highlights that hallucination is a major source of errors in the generative foundation models (Lu et al., 2023c; Ji et al., 2023).”
MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2023, unknown): “…et al., 2022; Touvron et al., 2023; OpenAI, 2023) have exhibited notable generalization abilities across a wide range of tasks and applications (Lu et al., 2023; Zhou et al., 2023; Park et al., 2023), with these capabilities stemming from their extensive training on vast comprehensive corpora…”
A Survey of Deep Learning for Mathematical Reasoning (2022, methodology): “A more recent work, Chameleon (Lu et al., 2023), integrates different tools to enhance the abilities of LLMs for compositional reasoning.”
ToolQA: A Dataset for LLM Question Answering with External Tools (2023, methodology): “Several additional works [56, 32, 57] expand the scope of tool utilization by incorporating different types of basic tools ( e.g. calculator, calendar, machine translation) to solve complex reasoning tasks.”
Tool learning with large language models: a survey (2024, unknown): “In earlier studies on tool learning [17, 69, 89], the primary paradigm is tool learning with one-step task solving: upon receiving a user question, LLMs would analyze the requests of user to understand the user intent and immediately plan all the sub-tasks needed to solve the problem.”