AXIS: Generating Explanations at Scale with Learnersourcing and Machine Learning

By Joseph Jay Williams, Juho Kim, Anna N. Rafferty, Samuel G. Maldonado, Krzysztof Z Gajos, Walter S. Lasecki, Neil Heffernan III. 2016

177 citations recorded by the source corpus.

While explanations may help people learn by providing information about why an answer is correct, many problems on online platforms lack high-quality explanations. This paper presents AXIS (Adaptive eXplanation Improvement System), a system for obtaining explanations. AXIS asks learners to generate, revise, and evaluate explanations as they solve a problem, and then uses machine learning to dynamically determine which explanation to present to a future learner, based on previous learners' collective input. Results from a case study deployment and a randomized experiment demonstrate that AXIS elicits and identifies explanations that learners find helpful. Providing explanations from AXIS also objectively enhanced learning, when compared to the default practice where learners solved problems and received answers without explanations. The rated quality and learning benefit of AXIS explanations did not differ from explanations generated by an experienced instructor.

Subfields: Artificial Intelligence, Computer Science Applications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

Citations in the corpus

Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI (2019, neutral): “Doshi-Velez, 2017; Selvaraju et al., 2016; Sheh, 2017; Symeonidis, Nanopoulos, and Manolopoulos, 2009; Tapaswi et al., 2016; Williams et al., 2016).”
WILDS: A Benchmark of in-the-Wild Distribution Shifts (2020, neutral): “…et al., 2014; Taghipour and Ng, 2016), estimating student knowledge and ability (Desmarais and Baker, 2012; Wu et al., 2020), identifying students who need assistance (Ahadi et al., 2015), or automatically generating explanations for student submissions (Williams et al., 2016; Wu et al., 2019).”
'It's Reducing a Human Being to a Percentage': Perceptions of Justice in Algorithmic Decisions (2018, background): “Contexts studied include aiding medical experts in diagnosis, increasing test performance of users of personalised learning environments, or enabling network security analysts to correctly identify incidents (e.g. [34, 83, 21]).”
Towards A Rigorous Science of Interpretable Machine Learning (2017, methodology): “…a visualization for correcting segmentations from microscopy data would be evaluated via user studies on segmentation on the target image task [Suissa-Peleg et al., 2016]; a homework-hint system is evaluated on whether the student achieves better post-test performance [Williams et al., 2016].”
Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models (2022, background): “Given that code walkthroughs can be mentally demanding, being presented with an explanation of one’s own code may reduce the cognitive demands associated with debugging [55] and evidence from other educational domains suggests that being presented with explanations produced by others can improve lea”
A Roadmap for a Rigorous Science of Interpretability (2017, methodology): “…a visualization for correcting segmentations from microscopy data would be evaluated via user studies on segmentation on the target image task [Suissa-Peleg et al., 2016]; a homework-hint system is evaluated on whether the student achieves better post-test performance [Williams et al., 2016].”
Benchmarking and survey of explanation methods for black box models (2021, background): “– Application-grounded evaluation methods require human experts to validate the specific task under analysis (Williams et al. 2016; Suissa-Peleg et al. 2016).”
Post-hoc Interpretability for Neural NLP: A Survey (2021, background): “For example, does the explanations result in higher survival-rates in a medical setting, a higher-grades in a homework-hint system, or a better model in a label-correction setting [37, 132].”
Evaluating the Quality of Learning Resources: A Learnersourcing Approach (2021, methodology): “Successful examples of learnersourced artifacts created by students include traces of videos watched by students [2], annotating videos for future students [1], creating open-ended artifact, such as solutions [3], or explanations [4], [5], and curriculum design [6].”