Chain-of-Thought Hub: A Continuous Effort to Measure Large Language Models' Reasoning Performance

By Yao Fu, Litu Ou, Mingyu Chen, Yuhao Wan, Hao-Chun Peng, Tushar Khot. 2023

134 citations recorded by the source corpus.

As large language models (LLMs) are continuously being developed, their evaluation becomes increasingly important yet challenging. This work proposes Chain-of-Thought Hub, an open-source evaluation suite on the multi-step reasoning capabilities of large language models. We are interested in this setting for two reasons: (1) from the behavior of GPT and PaLM model family, we observe that complex reasoning is likely to be a key differentiator between weaker and stronger LLMs; (2) we envisage large language models to become the next-generation computational platform and foster an ecosystem of LLM-based new applications, this naturally requires the foundation models to perform complex tasks that often involve the composition of linguistic and logical operations. Our approach is to compile a suite of challenging reasoning benchmarks to track the progress of LLMs. Our current results show that: (1) model scale clearly correlates with reasoning capabilities; (2) As of May 2023, Claude-v1.3 and PaLM-2 are the only two models that are comparable with GPT-4, while open-sourced models still lag behind; (3) LLaMA-65B performs closely to code-davinci-002, indicating that with successful further development such as reinforcement learning from human feedback (RLHF), it has great potential to be close to GPT-3.5-Turbo. Our results also suggest that for the open-source efforts to catch up, the community may focus more on building better base models and exploring RLHF.

Subfields: Prompting & Reasoning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Prompting & Reasoning

In Evaluation & Benchmarks

Citations in the corpus

MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (2024, methodology): “This CoT reasoning, adapted from Chain-of-Thought Hub [16], incorporates essential reasoning steps from the original MMLU dataset.”
Scaling Relationship on Learning Mathematical Reasoning with Large Language Models (2023, methodology): “There is also prior research that distilled the emergent reasoning ability of LLMs to small language models (Fu et al., 2023b; Shridhar et al., 2023).”
Fine-Grained Human Feedback Gives Better Rewards for Language Model Training (2023, background): “State-of-the-art AI is built on pre-trained language models that are then trained through interaction with humans [29, 28, 9], with a combination of supervised learning and reinforcement learning.”
SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (2023, background): “Recent benchmarks focus on assessing problem-solving skills of LLMs, particularly in scientific and mathematical domains [31, 11, 30, 49, 33, 8, 14, 16].”
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, unknown): “Figure 4 illustrates that the evolution process does not yield higher similarity scores.”
Connecting Large Language Models with Evolutionary Algorithms Yields Powerful Prompt Optimizers (2024, unknown): “(Kojima et al., 2022) with 3-shot Chain-of-Thought demonstrations (following Fu et al. (2023)) on the test set.”
Zephyr: Direct Distillation of LM Alignment (2023, methodology): “The HuggingFace Open LLM leaderbaord (Beeching et al., 2023), the Chain-of-Thought Hub (Fu et al., 2023), ChatEval (Sedoc et al., 2019), and FastEval (fas, 2023) are examples of other tools for evaluating chatty models.”