To Control or Not to Control: The question of whether experimental designs are appropriate for evaluating teaching innovations in higher education

By David Kember. 2003

104 citations recorded by the source corpus.

It is becoming increasingly common for academics to engage in research into their own teaching, usually by introducing some form of innovation into one of their courses. When doing so, they are often inclined by disciplinary tradition, or urged by senior colleagues, to use experimental designs in order that any observed outcomes can be attributed to the innovation itself. This article points out the problems with experimental designs for naturalistic studies of innovative teaching in higher education. A genuine control is impossible. Practical difficulties in separating groups often result in contamination of designs. Educational issues are complex with many variables involved. Therefore, experimental designs with limited numbers of cells result in over-simplification because they deal with only a few of the relevant factors. Particular types of innovation are not precisely reproducible so generalisation can be misleading. As an alternative, triangulation across multi-method evaluations from several sources is recommended, with the aim of establishing evidence beyond reasonable doubt. Comparison and synthesis across related projects appears to be a promising way to derive recommendations for ways to formulate important aspects of innovations.

Subfields: Education, Management of Technology and Innovation

View Paper: https://www.semanticscholar.org/paper/090c9909cbe6f0dcb80414dd4f2faa6c886daf9d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Education

In Management of Technology and Innovation

Citations in the corpus

The flipped classroom and cooperative learning: Evidence from a randomised experiment (2016, background): “The use of experiments to evaluate teaching interventions has limitations, with several potential pitfalls that might threat the value of an experiment (Kember, 2003).”
Involving Students in Assessment (2004, background): “Use of a control group is possible, though Kember (2003) argued that, in naturalistic studies, genuine control is impossible.”
Unravelling peer assessment: Methodological, functional, and conceptual developments (2010, background): “It should be noted that ecologically valid research settings complicate the inclusion of a genuine control group, and that a mixed-method approach or triangulation of multiple and diverse studies is proposed (Kember, 2003).”
Building effectiveness in teaching through targeted evaluation and response: connecting evaluation to teaching improvement in higher education (2008, background): “…for institutional student evaluation of teaching systems, are not tailored to the evaluation of particular innovations or practices, or the theoretical principles underpinning these, and so are of no use for evaluating specific experiments and developments in teaching practice (see Kember 2003).”
A comparative study of peer and teacher feedback and of various peer feedback forms in a secondary school writing curriculum (2010, background): “What are the features of a ‘zero treatment’ (Kember, 2003 ) or the ‘normal, traditional’ way of doing things in case of peer assessment research? Is teacher feedback present in the ‘zero treatment’ learning environment and is peer feedback considered supplementary or substitional to teacher feedbac”
The impact of cooperative learning on student engagement: Results from an intervention (2013, methodology): “‘treatment’ are in close proximity with each other, the use of the control group design has been shown to be very problematic for a number of reasons, and the pre–post design with multiple sources of data has been proposed as an alternative (Kember, 2003).”
‘Feed‐forward’: improving students' use of tutors' comments (2007, background): “The results are not conclusive, as to date they include only a small amount of data on one dimension (Kember, 2003).”
Changes in professional vision in the context of practice (2013, background): “although our study is embedded in a naturalistic field setting, which has a high degree of ecological validity (Kember 2003), we see the need for more research investigating the effects of diverse instructional settings in teacher-education programs.”