Usability evaluation considered harmful (some of the time)

By Saul Greenberg, William Buxton. International Conference on Human Factors in Computing Systems, 2008

687 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/ad3fe16611312d4e58f7a7f90a21a7230f5ec6f6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Human-Computer Interaction

Citations in the corpus

Evaluating Information Visualizations (2008, background): “Greenberg and Buxton [27] discuss this problem in terms of interaction sketches, encouraging caution when thinking about conducting usability testing on new ideas and new interface sketches in order to avoid interfering with the development of new ideas.”
A practical guide to controlled experiments of software engineering tools with human participants (2013, background): “…as HCI have even argued that the expectation of experimental evaluation before publishing a new tool can be a hindrance to innovation, disincentivizing researchers from gathering richer and more useful experiences with a technology through the use of qualitative methods (Greenberg and Buxton 2008).”
Structuring dimensions for collaborative systems evaluation (2012, background): “Even when evaluations are done, many of them seem to be performed in an ad hoc way, depending on the researchers’ interests or the practical adequateness for a specific setting [Inkpen et al. 2004; Greenberg and Buxton 2008].”
Design Study Methodology: Reflections from the Trenches and the Stacks (2012, methodology): “The phenomenon that what a target user says they do in retrospect is only an incomplete match with their actual activities is well-known in psychology [23] and HCI [60]; most users do not accurately introspect about their own data analysis and visualization needs, making these needs invisible to the”
Evaluation Strategies for HCI Toolkit Research (2018, background): “utility [31]: while a toolkit may be usable, it may not be useful.”
A Systematic Review on the Practice of Evaluating Visualization (2013, methodology): “Rather than blindly assuming and expecting certain evaluation methods, we argue for a more thorough consideration of research questions/goals and the right evaluation approach for this question/goal, echoing previous calls [23, 47, 50].”
Empirical Studies in Information Visualization: Seven Scenarios (2012, neutral): “Its main goal is to perfect the design by spotting major flaws and deficiencies in existing prototypes [24].”
From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI (2022, background): “In the context of usability evaluation in the HCI community, Greenberg and Buxton [84] argue that there is a risk of executing user studies in an early design phase, since this can quash creative ideas or promote poor ideas.”
Local Standards for Sample Size at CHI (2016, background): “This rationale, which we refer to as the sample size fallacy, has not been empirically studied in the CHI community (though it has been described; see [18]).”