Inferential Statistics in Computing Education Research: A Methodological Review

By Kate Sanders, Judithe Sheard, Brett A. Becker, Anna Eckerdal, Sally Hamouda, Simon. 2019

35 citations recorded by the source corpus.

The goal of most computing education research is to effect positive change in how computing is taught and learned. Statistical techniques are one important tool for achieving this goal. In this paper we report on an analysis of ICER papers that use inferential statistics. We present the most commonly used techniques; an overview of the techniques the ICER community has used over its first 14 years of papers, grouped according to the purpose of the technique; and a detailed analysis of three of the most commonly used techniques (t-test, chi-squared test, and Mann-Whitney-Wilcoxon). We identify common flaws in reporting and give examples of papers where statistics are reported well. In sum, the paper draws a picture of the use of inferential statistics by the ICER community. This picture is intended to help orient researchers who are new to the use of statistics in computing education research and to encourage reflection by the ICER community on how it uses statistics and how it can improve that use.

Subfields: Computer Science Applications, Information Systems, Statistics and Probability, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Science Applications

In Information Systems

In Statistics and Probability

In Evaluation & Benchmarks

Citations in the corpus

Comparing Code Explanations Created by Students and Large Language Models (2023, methodology): “Following the guidelines of [46] and the broader discussion in [37], we use p values as only one source of evidence and outline supporting statistics including two effect sizes – Rank-Biserial (RBC) Correlation [19] and Common-Language Effect Size (CLES) [30] – when presenting the results of the stu”