Absolute identification by relative judgment.

By Neil Stewart, Gordon D. A. Brown, N. Chater. 2005

398 citations recorded by the source corpus.

Subfields: Cognitive Neuroscience, Experimental and Cognitive Psychology, Cognitive Science & Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Experimental and Cognitive Psychology

In Cognitive Science & Psychology

Citations in the corpus

Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “…more labor intensive than A/B tests, as it requires longer and higher-cognitive involvement from the annotators, and it relies on absolute identification rather than relative discrimination, even though absolute identification is not reliable in humans (Stewart, Brown, and Chater, 2005).”
A conceptual analysis of five measures of metacognitive monitoring (2009, background): “My purpose is to address the ongoing debate about how to measure metacognitive judgments (Allwood et al. 2005; Dunn 2004; Juslin et al. 1996; Keren 1991; Koriat et al. 2002; Nelson 1996; Prins et al. 2006; Stewart et al. 2005; Wallsten 1996; Wright 1996).”
The futility of attempting to codify academic achievement standards (2014, background): “Assessors in general find it difficult to hold standards constant in their heads in any sort of absolute way that allows them to be applied on demand (Stewart et al. 2005).”
Deep Learning the City: Quantifying Urban Perception at a Global Scale (2016, background): “Studies have shown that gathering relative comparisons is a more efficient and accurate way of obtaining human rankings as compared to obtaining numerical scores from each user [45,46].”
A Review of Human Evaluation for Style Transfer (2021, background): “(2019) point to evidence that relative judgments can be more reliable than absolute judgments (Stewart et al., 2005), as part of their work on designing automatic metrics for ST evaluation.”
Evaluating Style Transfer for Text (2019, background): “Existing work from other fields have, however, shown that asking human raters to evaluate two relative comparisons provides more accurate scores than asking them to provide a numerical score for a single observation (Stewart et al., 2005; Bijmolt and Wedel, 1995).”
A temporal ratio model of memory. (2007, background): “The same tendency is evident in absolute identification of both temporal and nontemporal stimuli (e.g., G. D. A. Brown et al., 2005; Stewart et al., 2005).”
Human Evaluation of Conversations is an Open Problem: comparing the sensitivity of various methods for evaluating dialogue agents (2022, background): “Likert scoring relies on absolute identification rather than relative discrimination, which is less reliable in humans (Stewart et al., 2005), leading to different biases per annotator (Kulikov et al., 2019).”
The ordinal nature of emotions (2017, background): “theories, including relative judgment models [10] suggesting that experience with stimuli gradually creates our internal context, and discussions of anchors [11], against which we rank any forthcoming stimulus or perceived experience.”
Estimation from Pairwise Comparisons: Sharp Minimax Bounds with Topology Dependence (2015, background): “Ordinal measurements are also recognized to be easier or faster for humans to make (Barnett, 2003; Stewart et al., 2005), allowing for more evaluations with the same amount of time, eﬀort and cost.”