Optimal number of response categories in rating scales: reliability, validity, discriminating power, and respondent preferences.

By Carolyn C. Preston, Andrew M. Colman. Acta Psychologica, 2000

2,015 citations recorded by the source corpus.

Using a self-administered questionnaire, 149 respondents rated service elements associated with a recently visited store or restaurant on scales that differed only in the number of response categories (ranging from 2 to 11) and on a 101-point scale presented in a different format. On several indices of reliability, validity, and discriminating power, the two-point, three-point, and four-point scales performed relatively poorly, and indices were significantly higher for scales with more response categories, up to about 7. Internal consistency did not differ significantly between scales, but test-retest reliability tended to decrease for scales with more than 10 response categories. Respondent preferences were highest for the 10-point scale, closely followed by the seven-point and nine-point scales. Implications for research and practice are discussed.

Subfields: Applied Psychology, Information Systems and Management, Organizational Behavior and Human Resource Management

View Paper: https://www.semanticscholar.org/paper/de1353f6aca782bb603e5190ae0ec8ded4ba5451

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Psychology

In Information Systems and Management

In Organizational Behavior and Human Resource Management

Citations in the corpus

Five-Point Likert Items: t test versus Mann-Whitney-Wilcoxon (2010, background): “It is known that item reliability is affected by the number of points (depending on the distribution) and various recommendations exist ranging from 2–3 points (Matell & Jacoby, 1971) to 7–10 points (Preston & Colman, 2000).”
Can Charisma Be Taught? Tests of Two Interventions (2011, background): “Note that as demonstrated by Preston and Colman (2000), increasing scale points from a 5-point scale to a 9-point scale does not perturb reliability or criterion validity (see also Aiken, 1987; Lissitz & Green, 1975).”
Exploring Consumer Behavior in Virtual Reality Tourism Using an Extended Stimulus-Organism-Response Model (2018, background): “Because 7-point Likert-type scales offer high reliability and discriminant validity according to Cicchetti, Showalter, and Tyrer (1985) and Preston and Colman (2000), a total of 27 items were appraised on a 7-point Likert-type scale ranging from (1) strongly disagree to (7) strongly agree.”
Development and preliminary testing of the new five-level version of EQ-5D (EQ-5D-5L) (2011, background): “The choice of a five-level descriptive system is also supported by substantial psychometric literature [16–18].”
Quick, Simple and Reliable: Forced Binary Survey Questions (2011, background): “This contradiction may be due to the difference in questionnaires; our questionnaire used the same answer format for all questions within one version of the questionnaire, whereas Preston and Colman (2000) used different answer formats for different questions in their survey.”
Response Styles in Survey Research: A Literature Review of Antecedents, Consequences, and Remedies (2013, methodology): “Response scales have also been examined with many other criteria— for example, reliability, information recovery, distribution of scale means, and ease of use (see Preston & Colman, 2000; Weng, 2004)—resulting in similar recommendations with respect to the optimal number of response categories.”
Middle Response Functioning in Likert-responses to Personality Items (2008, neutral): “Preston and Colman (2000) investigated the reliability, validity, and discriminating power of the number of response categories.”
Impact of the Number of Response Categories and Anchor Labels on Coefficient Alpha and Test-Retest Reliability (2004, background): “For example, Matell and Jacoby (1971) suggested 2 or 3 response categories, Johnson et al. (1982) recommended a 3-point format, Ko (1994) and Oaster (1989) recommended 6- and 7-point scale designs, and Preston and Colman (2000) recommended 7 to 10 points.”
Likert Scale: Explored and Explained (2015, background): “Since the advent of Likert scale in 1932, there have been debates among the users about its best possible usability in term of reliability and validity of number of points on the scale [17-20].”
Five-Point Likert Items: t test versus Mann-Whitney-Wilcoxon ( Addendum added October 2012 ) (2010, background): “It is known that item reliability is affected by the number of points (depending on the distribution) and various recommendations exist ranging from 2–3 points (Matell & Jacoby, 1971) to 7–10 points (Preston & Colman, 2000).”
Best practices for the human evaluation of automatically generated text (2019, background): “Most of the experimental literature’s ﬁndings found that 7-point scales maximise reliability, validity and discriminative power (for instance, Miller, 1956; Green and Rao, 1970; Jones, 1968; Cicchetti et al., 1985; Lissitz and Green, 1975; Preston and Colman, 2000).”
Do data characteristics change according to the number of scale points used (2007, methodology): “One method is based on a formula used by Preston and Colman (2000).”