Position Bias in Multiple-Choice Questions

By N. Blunch. 1984

Subfields: Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

1. Zhanghao Wu
2. Eric P. Xing
3. Hoifung Poon
4. Zhuohan Li
5. Ion Stoica
6. Joseph E. Gonzalez
7. Sheng Zhang

Citations in the corpus

• Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “This bias is not unique to our context and has been seen in human decision-making [3, 29] and other ML domains [18, 35].”
• JudgeLM: Fine-tuned Large Language Models are Scalable Judges (2023, unknown): “Position bias means that the LLM judges prefer answers in a certain position and it widely exists in natural language processing tasks (Ko et al., 2020; Wang et al., 2018) and decision-making of humans (Blunch, 1984; Raghubir & Valenzuela, 2006).”
• Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine (2023, background): “While less severe than other foundation models, GPT-4 can exhibit a propensity to favor certain options in multiple choice answers over others (regardless of the option content), i.e., the model can show position bias [1, 16, 40].”