FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance

By Lingjiao Chen, Matei A. Zaharia, James Y. Zou. Trans. Mach. Learn. Res., 2023

768 citations recorded by the source corpus.

There is a rapidly growing number of large language models (LLMs) that users can query for a fee. We review the cost associated with querying popular LLM APIs, e.g. GPT-4, ChatGPT, J1-Jumbo, and find that these models have heterogeneous pricing structures, with fees that can differ by two orders of magnitude. In particular, using LLMs on large collections of queries and text can be expensive. Motivated by this, we outline and discuss three types of strategies that users can exploit to reduce the inference cost associated with using LLMs: 1) prompt adaptation, 2) LLM approximation, and 3) LLM cascade. As an example, we propose FrugalGPT, a simple yet flexible instantiation of LLM cascade which learns which combinations of LLMs to use for different queries in order to reduce cost and improve accuracy. Our experiments show that FrugalGPT can match the performance of the best individual LLM (e.g. GPT-4) with up to 98% cost reduction or improve the accuracy over GPT-4 by 4% with the same cost. The ideas and findings presented here lay a foundation for using LLMs sustainably and efficiently.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/585f8b9725f5f5e5495c3508d39f70d1c053e190

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing (2024, methodology): “The advent of LLMs has led to a paradigm shift in the study of natural language processing (NLP), computer vision, information retrieval, and other domains(Menghani, 2023; Chen et al., 2023; Jiang et al., 2023).”
Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, methodology): “For example, it costs $30 for 10M tokens using OpenAI’s GPT-4, while only $ 0.2 for GPT-J hosted by Textsyth [207].”
RouteLLM: Learning to Route LLMs with Preference Data (2024, neutral): “Frugal-GPT [9] employs an LLM cascade, sequentially querying LLMs until a reliable response is found.”
LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset (2023, methodology): “We encourage the entire community to explore these topics with this dataset, including building model selection and request caching algorithms (Chen et al., 2023; Zhu et al., 2023), training better models with RLHF and RLAIF (Bai et al., 2022b), data selection and curation algorithms (Xu et al.,…”
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (2023, background): “…this, observe that any LM call seeking to implement a particular signature needs to specify parameters that include: (1) the specific LM to call (Chen et al., 2023), (2) the prompt instructions (Yang et al., 2023) and the string prefix of each signature field and, most importantly, (3) the…”
Mixture-of-Agents Enhances Large Language Model Capabilities (2024, methodology): “Additionally, FrugalGPT (Chen et al., 2023b) proposed reducing the cost of using LLMs by employing different models in a cascading manner.”
Large Language Models as Tool Makers (2023, methodology): “The ability of LLMs to self-debug has been demonstrated effectively in recent research (Madaan et al., 2023; Chen et al., 2023c; Lu et al., 2023; Kim et al., 2023).”
HybridInfer: Thermal-Aware Reinforcement-Learning Tier Routing for On-Device, Edge, and Cloud LLM Inference (2026, unknown): “FrugalGPT [1] cascades from cheaper to stronger models and stops when an answer looks adequate.”