Instruction-Following Evaluation for Large Language Models

By Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, Le Hou. 2023

995 citations recorded by the source corpus.

One core capability of Large Language Models (LLMs) is to follow natural language instructions. However, the evaluation of such abilities is not standardized: Human evaluations are expensive, slow, and not objectively reproducible, while LLM-based auto-evaluation is potentially biased or limited by the ability of the evaluator LLM. To overcome these issues, we introduce Instruction-Following Eval (IFEval) for large language models. IFEval is a straightforward and easy-to-reproduce evaluation benchmark. It focuses on a set of"verifiable instructions"such as"write in more than 400 words"and"mention the keyword of AI at least 3 times". We identified 25 types of those verifiable instructions and constructed around 500 prompts, with each prompt containing one or more verifiable instructions. We show evaluation results of two widely available LLMs on the market. Our code and data can be found at https://github.com/google-research/google-research/tree/master/instruction_following_eval

Subfields: Artificial Intelligence, Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Evaluation & Benchmarks

Citations in the corpus

From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline (2024, background): “Many have considered task-based evaluation such as IFEval [53], SWE-Bench [22], or AgentBench [28].”
DeepSeek LLM: Scaling Open-Source Language Models with Longtermism (2024, methodology): “Instruction Following Evaluation: On Nov 15th, 2023, Google released an instruction following the evaluation dataset (Zhou et al., 2023).”
LiveBench: A Challenging, Contamination-Free LLM Benchmark (2024, methodology): “Table 4: The list of 25 instructions used in [68], and the 16 that are both ‘real-world’ and automatically verifiable, which we used in LiveBench .”
Evaluation and mitigation of the limitations of large language models in clinical decision-making (2024, neutral): “To mitigate the risk of physicians recognizing the pattern of four distinct target pathologies, a further five patients presenting with gastritis, a urinary tract infection, esophageal reflux and a hernia were included.”
ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools (2024, methodology): “We adhere to the methodologies outlined by [62] to calculate prompt-level and instruction-level accuracy in both strict mode and loose mode .”
Small Language Models are the Future of Agentic AI (2025, positive): “…(an indicator of basic understanding), tool calling and code generation (both indicators of the ability to correctly communicate across the model → tool/code interface; see Figure 1; [74, 75]), and instruction following (ability to correctly respond back across the code ← model interface; [80]).”
DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (2024, methodology): “We also conduct an instruction-following evaluation (IFEval) (Zhou et al., 2023) for DeepSeek-V2 Chat (SFT), using prompt-level loose accuracy as the metric.”
Phi-4 Technical Report (2024, neutral): “We also consider MMLU-pro [WMZ + 24], HumanEval+ [LXWZ23], ArenaHard [CZS + 24], and IFEval [ZLM + 23], for which we use an internal framework and prompting and extraction.”
Instruction Tuning for Large Language Models: A Survey (2023, neutral): “Finally, the immense computational cost of training these models presents a major barrier, limiting accessibility and slowing the pace of academic research in this domain.”
InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, positive): “MMLU-Pro [61], GAOKAO [177], IFEval [185]; (4) Agentic Tasks : SGP-Bench [102], ScreenSpot [16], ScreenSpot-v2 [150], OSWorld-G [156], VSI-Bench [161], ERQA [121], SpaCE-10 [38].”
SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild (2025, neutral): “IFEVAL measures instruction-following capability, MMLU assesses the model’s mastery of general knowledge, and GPQA-Diamond is a challenging benchmark that tests domain-specific expertise in chemistry, physics, and biology.”
SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks (2023, positive): “More specifically, as is common in the adversarial robustness literature, we now seek a surrogate for SmoothLLM that is differentiable and amenable to GCG attacks.”
Qwen3-VL Technical Report (2025, positive): “(1) Knowledge: MMLU-Pro (Wang et al., 2024f), MMLU-Redux (Gema et al., 2024), GPQA (Rein et al., 2023), Su-perGPQA (Team, 2025), (2) Reasoning: AIME-25 (AIME, 2025), HMMT-25 (HMMT, 2025), LiveBench (2024-11-25) (White et al., 2024), (3) Code: LiveCodeBench v6 (Jain et al., 2024), CFEval, OJBench (Wa”