Orca: Progressive Learning from Complex Explanation Traces of GPT-4

By Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, Ahmed Hassan Awadallah. arXiv.org, 2023

421 citations recorded by the source corpus.

Recent research has focused on enhancing the capability of smaller models through imitation learning, drawing on the outputs generated by large foundation models (LFMs). A number of issues impact the quality of these models, ranging from limited imitation signals from shallow LFM outputs; small scale homogeneous training data; and most notably a lack of rigorous evaluation resulting in overestimating the small model's capability as they tend to learn to imitate the style, but not the reasoning process of LFMs. To address these challenges, we develop Orca (We are working with our legal team to publicly release a diff of the model weights in accordance with LLaMA's release policy to be published at https://aka.ms/orca-lm), a 13-billion parameter model that learns to imitate the reasoning process of LFMs. Orca learns from rich signals from GPT-4 including explanation traces; step-by-step thought processes; and other complex instructions, guided by teacher assistance from ChatGPT. To promote this progressive learning, we tap into large-scale and diverse imitation data with judicious sampling and selection. Orca surpasses conventional state-of-the-art instruction-tuned models such as Vicuna-13B by more than 100% in complex zero-shot reasoning benchmarks like Big-Bench Hard (BBH) and 42% on AGIEval. Moreover, Orca reaches parity with ChatGPT on the BBH benchmark and shows competitive performance (4 pts gap with optimized system message) in professional and academic examinations like the SAT, LSAT, GRE, and GMAT, both in zero-shot settings without CoT; while trailing behind GPT-4. Our research indicates that learning from step-by-step explanations, whether these are generated by humans or more advanced AI models, is a promising direction to improve model capabilities and skills.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/0244aeb7c6927e2fb0c2e668687e160a00737dbe

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Aligning Large Multimodal Models with Factually Augmented RLHF (2023, result): “…the absence of a negative feedback mechanism preventing the model from identifying hallucinations in its output, and the potential limitations of our Fact-RLHF reward model, especially when compared against the high-capacity oracle models in previous successful studies (Touvron et al., 2023b).”
Instruction Tuning for Large Language Models: A Survey (2023, neutral): “This section details our approach to searching, selecting, and analyzing the relevant literature.”
MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (2023, neutral): “• Instruction Tuning: We include Orca-Platypus (Mukherjee et al., 2023), Vicuna-1.”
LESS: Selecting Influential Data for Targeted Instruction Tuning (2024, background): “…Sanh et al., 2022; Wei et al., 2022b; Longpre et al., 2023), and (2) open-ended instruction following datasets, broadly covering a wide range of topics (Taori et al., 2023; Conover et al., 2023; K ¨ opf et al., 2023; Xu et al., 2023; Mukherjee et al., 2023; Zhou et al., 2023; Ding et al., 2023).”
Making Large Language Models Better Reasoners with Alignment (2023, positive): “Recent works (Chung et al., 2022; Hsieh et al., 2023; Mukherjee et al., 2023) find that training LLMs using data with a chain of thought (COT) reasoning process is a very effective method to improve the reasoning ability of LLMs.”
Aligning Large Language Models with Human: A Survey (2023, methodology): “, 2022b), Orca (Mukherjee et al., 2023), Lion (Jiang et al.”
SeaLLMs - Large Language Models for Southeast Asia (2023, background): “However, that may not even be feasible for low-resource non-Latin languages because of the unfavorable tokenization of ChatGPT as explained in §2.2.”
Improving Text Embeddings with Large Language Models (2023, methodology): “Unnatural Instructions [15] is a synthetic instruction following dataset by prompting existing LLMs. Orca [28] and Phi [14] propose to train better small language models by using high-quality synthetic data from GPT-3.”
CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation (2024, positive): “Data augmentation has been shown to mitigate these issues for pre-training data generation (Maini et al., 2024).”
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (2023, background): “The distillation is still done in a supervised way, and other kind of enhanced prompts could be considered (Li et al., 2022; Mukherjee et al., 2023).”
ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs (2023, background): “A recent line of work has explored student-teacher frameworks with the goal of distilling reasoning capabilities from a stronger teacher to a weaker student (Magister et al., 2023; Fu et al., 2023; Ho et al., 2023; Saha et al., 2023; Mukherjee et al., 2023).”
Augmenting large language models with chemistry tools (2023, neutral): “1 .”
Explainability for Large Language Models: A Survey (2023, background): “One recent study proposes explanation tuning, an approach that trains smaller language models using detailed step-by-step explanations from more advanced models as a form of supervision (Mukherjee et al., 2023).”
A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “…GPT series models to generate prompt-response data pairs and then train the student LLMs by supervised fine-tuning (Wang et al., 2022a; Taori et al., 2023; Chiang et al., 2023; Wu et al., 2023c; Xu et al., 2023a; Mukherjee et al., 2023; Mitra et al., 2023; Luo et al., 2023c; Peng et al., 2023a).”
UltraFeedback: Boosting Language Models with High-quality Feedback (2023, positive): “For FLAN (Longpre et al., 2023), we adopt a stratified sampling strategy (Mukherjee et al., 2023), randomly picking 3k instructions from the “CoT” subset and sampling 10 instructions per task for the other three subsets, while excluding those with overly long instructions.”
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, neutral): “Presently, Llama assumes a pivotal role as the foundational model for supervised fine-tuning, ushering in the emergence of several extremely remarkable models, including Alpaca [22], Vicuna [23], Guanaco [56], WizardLM [24], and Orca [57], RFT [16] etc.”
Mobile Game Evaluation Method Based on Data Mining of Affective Time Series (2025, unknown): “As such, it allows for exceptions to the separation criterion: The prime example is the tutorial, which typically should overlap with most of the other GAIMs in order to provide the player with a comprehensive introduction to all the elements of the gameplay.”