The False Promise of Imitating Proprietary LLMs

By Arnav Gudibande, Eric Wallace, Charles Burton Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, Dawn Song. 2023

280 citations recorded by the source corpus.

An emerging method to cheaply improve a weaker language model is to finetune it on outputs from a stronger model, such as a proprietary system like ChatGPT (e.g., Alpaca, Self-Instruct, and others). This approach looks to cheaply imitate the proprietary model's capabilities using a weaker open-source model. In this work, we critically analyze this approach. We first finetune a series of LMs that imitate ChatGPT using varying base model sizes (1.5B--13B), data sources, and imitation data amounts (0.3M--150M tokens). We then evaluate the models using crowd raters and canonical NLP benchmarks. Initially, we were surprised by the output quality of our imitation models -- they appear far better at following instructions, and crowd workers rate their outputs as competitive with ChatGPT. However, when conducting more targeted automatic evaluations, we find that imitation models close little to none of the gap from the base LM to ChatGPT on tasks that are not heavily supported in the imitation data. We show that these performance discrepancies may slip past human raters because imitation models are adept at mimicking ChatGPT's style but not its factuality. Overall, we conclude that model imitation is a false promise: there exists a substantial capabilities gap between open and closed LMs that, with current methods, can only be bridged using an unwieldy amount of imitation data or by using more capable base LMs. In turn, we argue that the highest leverage action for improving open-source models is to tackle the difficult challenge of developing better base LMs, rather than taking the shortcut of imitating proprietary systems.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Information Systems, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Information Systems

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, background): “These models rely on intricate tuning techniques to align with human preferences (Gudibande et al., 2023), a process that is still being explored and refined within the open-source community.”
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “This approach has been tried in our earlier blog post [7] and other concurrent or follow-up work [5, 25, 12, 11, 43, 15].”
Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions (2023, background): “Although there are still many discussions regarding how competitive these finetuned instruction models can become (see Gudibande et al. (2023) for a pessimistic view and see Zhou et al. (2023) for a more optimistic one), the community has picked up these models relatively quickly and many…”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, background): “Numerous studies have noted that the apparent scientific style of language used by ChatGPT can mislead human observers regarding the reliability of its outputs [357], [358].”
Towards Understanding Sycophancy in Language Models (2023, unknown): “Human evaluators are imperfect (Saunders et al., 2022; Gudibande et al., 2023), make mistakes e.g., due to limited time (Chmielewski & Kucker, 2020) or cognitive biases (Pandey et al., 2022), and sometimes have diverse, contradictory preferences (Bakker et al., 2022).”
The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (2023, background): “Moreover, Gudibande et al. (2023) demonstrate that aligning open-source LLMs by imitating proprietary LLMs (e.g., ChatGPT) may not always yield desirable results, emphasizing the importance of a strong pre-trained base LLM for producing factual content.”
Instruction Tuning for Large Language Models: A Survey (2023, unknown): “…(2) there has been an increasing concern that IT only improves on tasks that are heavily supported in the IT training dataset [44]; and (3) there has been an intense criticism that IT only captures surface-level patterns and styles (e.g., the output format) rather than comprehending and…”
Zephyr: Direct Distillation of LM Alignment (2023, background): “Distillation has proven to be an effective tool for improving open models on a range of different tasks (Chiang et al., 2023); however, it does not reach the performance of the teacher models (Gudibande et al., 2023).”
Orca: Progressive Learning from Complex Explanation Traces of GPT-4 (2023, background): “[12], where the authors assert that “model imitation is a false promise” since “broadly matching ChatGPT using purely imitation would require (1) a concerted effort to collect enormous imitation datasets and (2) far more diverse and higher quality imitation data than is currently available.”
How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources (2023, methodology): “Concurrent to this work, Gudibande et al. [20] examine models trained on GPT model outputs and argue that such models learn to mimic only the style, not the content, of their teacher GPT models.”
Explainability for Large Language Models: A Survey (2023, background): “Another recent study [Gudibande et al. 2023] showed that imitation can successfully improve the style, persona, and ability of the language model to follow instructions, but does not improve language models on more complex dimensions such as factuality, coding, and problem solving.”
Evaluating Large Language Models at Evaluating Instruction Following (2023, background): “Furthermore, unlike attributes that can be easily acquired through imitation learning, such as engaging tones (Gudibande et al., 2023), even the strongest LLMs today struggle with following instructions (Wu et al., 2023b; Li et al., 2023c).”
Style Over Substance: Evaluation Biases for Large Language Models (2023, unknown): “As pointed out by Gudibande et al. (2023) and Zheng et al. (2023) that there is a gap between users’ perception and standardized evaluation suites, recent LLM studies often incorporate human evaluation for a more nuanced understanding of model performance (Wang et al., 2022a; Chiang et al., 2023).”