Standards for Internet-based experimenting.

By Ulf-Dietrich Reips. Experimental Psychology, 2002

934 citations recorded by the source corpus.

This article summarizes expertise gleaned from the first years of Internet-based experimental research and presents recommendations on: (1) ideal circumstances for conducting a study on the Internet; (2) what precautions have to be undertaken in Web experimental design; (3) which techniques have proven useful in Web experimenting; (4) which frequent errors and misconceptions need to be avoided; and (5) what should be reported. Procedures and solutions for typical challenges in Web experimenting are discussed. Topics covered include randomization, recruitment of samples, generalizability, dropout, experimental control, identity checks, multiple submissions, configuration errors, control of motivational confounding, and pre-testing. Several techniques are explained, including "warm-up," "high hurdle," password methods, "multiple site entry," randomization, and the use of incentives. The article concludes by proposing sixteen standards for Internet-based experimenting.

Subfields: Applied Psychology, Sociology and Political Science

View Paper: https://www.semanticscholar.org/paper/59b4fd1cd96e9ef7d1fbfd612530f2f4f870167f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Applied Psychology

In Sociology and Political Science

Citations in the corpus

Web-based versus lab-based studies: A response to Kendall (2008) (2008, neutral): “On the contrary, proper experiment design in Internet-based experimenting has been much advocated as crucial in optimizing the benefits and avoiding some problems in this type of research (e.g., Birnbaum, 2007; Birnbaum & Reips, 2005; Reips, 2000, 2002a, 2002b, 2007 ).”
Dissecting work commitment: the role of Machiavellianism (2011, methodology): “The entire study was run via the internet and in German language, adhering as closely as possible to existing standards for web-based data collection ( Reips, 2002”
Temporal distance and ease of retrieval. (2007, methodology): “The implementation of the online experiment conformed to common standards recommended for online research (Birnbaum, 2004; Reips, 2000, 2002).”
Emotion regulation strategy mediates both positive and negative relationships between music uses and well-being (2014, methodology): “Participants’ response data and timings were screened for non-serious responding, as per the guidelines recommended for webbased experiments (Reips, 2002).”
Mindfulness-based stress reduction (MBSR) enhances distress tolerance and resilience through changes in mindfulness (2016, methodology): “Data were collected online following Internet testing guidelines by Reips (2002) to increase data quality.”
Tracing the Path from Personality — via Cooperativeness — to Conservation (2013, background): “The hypotheses sketched earlier were tested in a web-based study that was constructed in adherence to common standards for Internet data collection (Reips, 2002).”
Conducting behavioral research on Amazon’s Mechanical Turk (2011, neutral): “There are many advantages to online experimentation, many of which are detailed in prior work (Reips, 2000, 2002).”
Development of measures of online privacy concern and protection for use on the Internet (2007, background): “However, it remains important to ensure that psychological measures used online really are valid tests of the constructs they purport to address (e.g., Buchanan & Smith, 1999; Buchanan, Johnson, & Goldberg, 2005; Buchanan et al., 2005; Reips, 2000, 2002).”
Web-based experiments for the study of collective social dynamics in cultural markets (2009, positive): “These experiments will pose specific challenges above and beyond the challenges of Web-based experiments in general (Kraut et al., 2004; Nosek, Banaji, & Greenwald, 2002a; Reips, 2002; Skitka & Sargis, 2006).”