Large Language Models Are Not Robust Multiple Choice Selectors
By Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, Minlie Huang. 2023
481 citations recorded by the source corpus.
Multiple choice questions (MCQs) serve as a common yet important task format in the evaluation of large language models (LLMs). This work shows that modern LLMs are vulnerable to option position changes in MCQs due to their inherent"selection bias", namely, they prefer to select specific option IDs as answers (like"Option A"). Through extensive empirical analyses with 20 LLMs on three benchmarks, we pinpoint that this behavioral bias primarily stems from LLMs' token bias, where the model a priori assigns more probabilistic mass to specific option ID tokens (e.g., A/B/C/D) when predicting answers from the option IDs. To mitigate selection bias, we propose a label-free, inference-time debiasing method, called PriDe, which separates the model's prior bias for option IDs from the overall prediction distribution. PriDe first estimates the prior by permutating option contents on a small number of test samples, and then applies the estimated prior to debias the remaining samples. We demonstrate that it achieves interpretable and transferable debiasing with high computational efficiency. We hope this work can draw broader research attention to the bias and robustness of modern LLMs.
Subfields: Artificial Intelligence, Information Systems, Large Language Models, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Hannaneh Hajishirzi
- Yi Tay
- Yejin Choi
- Quoc V. Le
In Information Systems
- Tushar Khot
- Xueguang Ma
- Hoifung Poon
- Yi Tay
- Omar Khattab
- Keshav Santhanam
- Marie-Anne Lachaux
- Donald Metzler
In Large Language Models
- Yi Tay
- Donald Metzler
- Mostafa Dehghani
- Colin Raffel
- Noam Shazeer
- Tushar Khot
- Luke S. Zettlemoyer
- Neil Houlsby
In Safety, Robustness & Interpretability
- Dan Hendrycks
- Steven Basart
- Colin Raffel
- Neil Houlsby
- Michael Matena
- Dimitris Tsipras
- Nikita Nangia
- Yi Tay
Citations in the corpus
- MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (2024, background): “Besides the saturation issue, the performance on MMLU is also known to be highly sensitive to the prompt and scoring function [47, 3], which causes significant order changes in the leaderboard.”
- Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine (2023, background): “By reordering or shuffling components of a few-shot prompt, ensembling techniques can also address the order sensitivity commonly found with foundation models [26, 39], thus improving robustness.”
- When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards (2024, methodology): “Leaderboards based on multiple choice questions (MCQ) for evaluation (Wang et al., 2018, 2019; Nie et al., 2019; Zhong et al., 2023; Hendrycks et al., 2020) present both convenience and significant limitations (Pezeshkpour and Hr-uschka, 2023; Zheng et al., 2023).”