Measure Twice, Cut down Error: A Process for Enhancing the Validity of Survey Scales

By Hunter Gehlbach, Maureen E. Brinkworth. 2011

260 citations recorded by the source corpus.

For years psychologists across many subfields have undertaken the formidable challenge of designing survey scales to assess attitudes, opinions, and behaviors. Correspondingly, scholars have written much to guide researchers in this undertaking. Yet, many new scales violate established best practices in survey design, suggesting the need for a new approach to designing surveys. This article presents 6 steps to facilitate the construction of questionnaire scales. Unlike previous processes, this one front loads input from other academics and potential respondents in the item-development and revision phase with the goal of achieving credibility across both populations. Specifically, the article describes how (a) a literature review and (b) focus group–interview data can be (c) synthesized into a comprehensive list to facilitate (d) the development of items. Next, survey designers can subject the items to (e) an expert review and (f) cognitive pretesting before executing a pilot test.

Subfields: Applied Psychology, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/d5f223434b1c1f099d3ba99b2e2f48045afc3b33

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Psychology

In Sociology and Political Science

Citations in the corpus

Measurement Matters: Assessing Personal Qualities Other Than Cognitive Ability for Educational Purposes (2015, background): “…that use construct-specific verbal labels to avoid acquiescence bias, use the optimal number of response options, balance bipolar choices, and so on (Gehlbach & Brinkworth, 2011; Krosnick, 1999; Schuman & Presser, 1981; Schwarz & Oyserman, 2001)—and many fewer of them to solve this latter problem…”
Measuring cost: The forgotten component of expectancy-value theory (2015, background): “Gehlbach and Brinkworth (2011) discuss an in-depth process of review where the construct of interest is connected to all other relevant theories and ideas.”
Experimental Evidence on Teachers’ Racial Bias in Student Evaluation: The Role of Grading Scales (2020, methodology): “The 4-point personal narrative rubric was taken directly from an actual writing rubric, while the choice of a 7-point grade-level scale was made to align with recommendations for developing bipolar response scales (Gehlbach & Brinkworth, 2011).”
Leadership as social identity management: Introducing the Identity Leadership Inventory (ILI) to assess and validate a four-dimensional model (2014, methodology): “Given that we placed particular emphasis on developing a clear theoretical foundation and followed commonly best practices in generating our items (Gehlbach & Brinkworth, 2011), we settled for 20 items that we sought to test with the aim of reducing these to no more than four items for each of the…”
Using qualitative methods to develop a survey measure of math and science engagement (2016, methodology): “As suggested in Gelhbach and Brinkworth (2011), in the next step, we compared responses from our focus groups and interviews with students and teachers about the meaning of engagement and disengagement in math and science against the results from the literature review to determine points of…”