Evaluating Large Language Models

By February Friday. 2022

Subfields: Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Evaluation & Benchmarks

Citations in the corpus

An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA (2021, methodology): “Among the k answer predictions, we select the one with the highest sum of log-probability t log p LM ( y t ) as the ﬁnal answer (Chen et al. 2021 examples and distributing them into k prompts, we combine the two methods and obtain the gains from both approaches.”
Using Large Language Models to Enhance Programming Error Messages (2022, methodology): “GPT-3 also powers several other tools such as Codex which is essentially a GPT-3 model that has been fine-tuned with more than 50 million repositories representing the majority of Python code available on GitHub totalling 159GB of source code [13].”
Auditing large language models: a three-layered approach (2023, background): “LLMs can act as intermediary assets that are adapted for specific tasks, such as diagnosing medical conditions [51], generating code [52], [53] and translating languages [54].”
Baldur: Whole-Proof Generation and Repair with Large Language Models (2023, background): “ProofNet [4] introduces a dataset and benchmark suite for auto-formalization in Lean, based on undergraduate mathematics, and shows preliminary promising results autoformalizing proofs on that benchmark using Codex [14] with few-shot learning.”
Inverse Scaling: When Bigger Isn't Better (2023, background): “In other words, the gap in the correctness of generated code between subtle-bug prompts and no-bug prompts grows with model size, as shown in Figure 12 of Chen et al. (2021).”
ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health (2023, methodology): “LLMs facilitate the implementation of advanced code generation capabilities for statistical and data analytics, two large-scale AI-powered code generation tools have recently come into the spotlight: OpenAI Codex, a GPT language model fine-tuned on publicly available code from GitHub, (33) and DeepM”
Demystifying GPT Self-Repair for Code Generation (2023, methodology): “Using this instead of the traditional pass@k [5, 17] metric (which weighs pass rate against the number of trials), we are able to accurately compare performance gained through self-repair against any additional work done by the model when generating the feedback and carrying out the repair.”
Orca: A Distributed Serving System for Transformer-Based Generative Models (2022, background): “Language generation tasks are becoming increasingly paramount to many types of applications, such as chatbot [9, 52], summarization [41,45,54], code generation [13], and caption generation [65,66].”
Jigsaw: Large Language Models meet Program Synthesis (2021, methodology): “In particular, Codex [11], that is part of GitHub Copilot, generates Python code from natural language descriptions.”