Training for calibration.

By S. Lichtenstein, Baruch Fischhoff. 1980

Subfields: Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

1. Ethan Perez
2. Sam Bowman
3. A. P. Mcgraw
4. Peter Mcgraw
5. Dawn Drain
6. Ilana Ritov
7. Stanislav Fort
8. Tom B. Brown

Citations in the corpus

• Measuring Progress on Scalable Oversight for Large Language Models (2022, background): “We expect that explicit calibration training on the task would have improved these results for human participants and human–model teams (Lichtenstein and Fischhoff, 1980), which could potentially improve raw accuracy as well by better weighting votes across participants.”
• The Role of Teacher Epistemic Cognition, Epistemic Beliefs, and Calibration in Instruction (2008, result): “Thus, it is recommended that teachers engage in reflection and receive feedback in order to become better aware of their epistemic beliefs and better trained in how to calibrate them to different situations, a suggestion compatible with previous findings (Lichtenstein and Fischhoff 1980).”
• Improving metacognition in the classroom through instruction, training, and feedback (2016, background): “Additionally, when feedback was paired with training, calibration improved (Huff and Nietfeld 2009; Lichtenstein and Fischhoff 1980; Renner and Renner 2001).”
• Exploring the Relationship between Calibration and Self-Regulated Learning (2000, methodology): “Measures of under- and overconfidence are commonly used to evaluate calibration levels (Bjorkman, 1992; Lichtenstein and Fischhoff, 1980 ).”
• The Ethical Mirage: A Temporal Explanation as to Why We Are Not as Ethical as We Think We Are (2010, background): “Feedback has also been identified as being important in the debiasing process (Lichtenstein and Fischhoff, 1980).”
• The ‘heuristics and biases’ bias in expert elicitation (2007, methodology): “There are also some criticisms that some scoring rules are sensitive to changes in resolution (say, usually using a one-digit response but occasionally using a two- or three-digit response) and also to the number of questions asked (Lichtenstein and Fischhoff, 1980).”
• The affective costs of overconfidence (2004, methodology): “Yet some procedures have been effective, such as training and feedback (Lichtenstein & Fischhoff, 1980), asking respondents to discuss their responses with peers (Arkes, Christensen, Lai, & Blumer, 1987), and asking respondents to consider contrary evidence (Koriat, Lichtenstein, & Fischhoff, 1980).”
• The recalcitrance of overconfidence and its contribution to decision aid neglect (2005, background): “…attempts to reduce overconfidence have been made in various task settings, and with varying degrees of theoretical motivation (e.g. Arkes, Christensen, Lai, & Blumer, 1987; Griffin & Buehler, 1999; Koriat, Lichtenstein, & Fischhoff, 1980; Lichtenstein & Fischhoff, 1980; Sieck & Yates, 2001).”