Measuring usability: are effectiveness, efficiency, and satisfaction really correlated?

By Erik Frøkjær, M. Hertzum, Kasper Hornbæk. International Conference on Human Factors in Computing Systems, 2000

765 citations recorded by the source corpus.

Usability comprises the aspects effectiveness, efficiency, and satisfaction. The correlations between these aspects are not well understood for complex tasks. We present data from an experiment where 87 subjects solved 20 information retrieval tasks concerning programming problems. The correlation between efficiency, as indicated by task completion time, and effectiveness, as indicated by quality of solution, was negligible. Generally, the correlations among the usability aspects depend in a complex way on the application domain, the user's experience, and the use context. Going through three years of CHI Proceedings, we find that 11 out of 19 experimental studies involving complex tasks account for only one or two aspects of usability. When these studies make claims concerning overall usability, they rely on risky assumptions about correlations between usability aspects. Unless domain specific studies suggest otherwise, effectiveness, efficiency, and satisfaction should be considered independent aspect of usability and all be included in usability testing.

Subfields: Information Systems and Management, Social Psychology, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/e9f6a7cbdcdfa274c2c59b8a7c82717344f80e39

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems and Management

In Social Psychology

In Human-Computer Interaction

Citations in the corpus

Towards a Science of Human-AI Decision Making: A Survey of Empirical Studies (2021, methodology): “of efficiency are not seen in the papers we reviewed, although self-reported task efficiency is a common metric used in usability testing [46].”
An empirical study of the effects of interactivity on web user attitude (2003, background): “Apparently, the three contributing factors proposed in Ducoffe’s attitude model are closely related to the three key constructs of system usability: satisfaction, effectiveness and efficiency (Frokjaer et al., 2000).”
Eye Tracking in Human-Computer Interaction and Usability Research (2003, background): “Various measures of usability are necessary to gather the whole picture (see for example Frokjaer, Hertzum & Hornbaek, 2000 ).”
Current practice in measuring usability: Challenges to usability studies and research (2006, methodology): “Outside the studies reviewed, papers discuss correlations between different aspects of usability (Frøkjær et al., 2000; Hassenzahl et al., 2000).”
A method to standardize usability metrics into a single score (2005, background): “Frøkjær et al [12] earlier has made the case for including all aspects (effectiveness, efficiency and satisfaction) when measuring the usability of a system since it was found that these aspects did not always correlate in the data they reviewed.”