A Comparison of the Quality of Data-Driven Programming Hint Generation Algorithms

Autor/inn/en	Price, Thomas W.; Dong, Yihuan; Zhi, Rui; Paaßen, Benjamin; Lytle, Nicholas; Cateté, Veronica; Barnes, Tiffany
Titel	A Comparison of the Quality of Data-Driven Programming Hint Generation Algorithms
Quelle	In: International Journal of Artificial Intelligence in Education, 29 (2019) 3, S.368-395 (28 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Price, Thomas W.)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1560-4292
DOI	10.1007/s40593-019-00177-z
Schlagwörter	Comparative Analysis; Programming Languages; Data Analysis; Evaluation Methods; Specialists; Computer Software; Benchmarking + Suchen Sie Ihr Suchwort? Auswertung
Abstract	In the domain of programming, a growing number of algorithms automatically generate data-driven, next-step hints that suggest how students should edit their code to resolve errors and make progress. While these hints have the potential to improve learning if done well, few evaluations have directly assessed or compared the quality of different hint generation approaches. In this work, we present the QualityScore procedure, a novel method for automatically evaluating and comparing the quality of next-step programming hints using expert ratings. We first demonstrate that the automated QualityScore ratings agree with experts' manual ratings. We then use the QualityScore procedure to compare the quality of six data-driven, next-step hint generation algorithms using two distinct programming datasets in two different programming languages. Our results show that there are large and significant differences between the quality of the six algorithms and that these differences are relatively consistent across datasets and problems. We also identify situations where the six algorithms struggle to produce high-quality hints, and we suggest ways that future work might address these challenges. We make our methods and data publicly available and encourage researchers to use the QualityScore procedure to evaluate additional algorithms and benchmark them against our results. (As Provided).
Anmerkungen	Springer. Available from: Springer Nature. 233 Spring Street, New York, NY 10013. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-348-4505; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01