Student Log-Data from a Randomized Evaluation of Educational Technology: A Causal Case Study

Autor/inn/en	Sales, Adam C.; Pane, John F.
Titel	Student Log-Data from a Randomized Evaluation of Educational Technology: A Causal Case Study
Quelle	In: Journal of Research on Educational Effectiveness, 14 (2021) 1, S.241-269 (29 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Sales, Adam C.) ORCID (Pane, John F.)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1934-5747
DOI	10.1080/19345747.2020.1823538
Schlagwörter	Educational Technology; Use Studies; Randomized Controlled Trials; Mathematics Curriculum; Curriculum Evaluation; Algebra; Intelligent Tutoring Systems; Cues; Problem Solving; Statistical Analysis; Causal Models + Suchen Sie Ihr Suchwort? Unterrichtsmedien; Benutzerschulung; Curriculum; Evaluation; Curriculumevaluation; Lehrplan; Rahmenplan; Evaluierung; Intelligentes Tutorsystem; Stichwort; Problemlösen; Statistische Analyse; Kausalanalyse
Abstract	Randomized evaluations of educational technology produce log data as a bi-product: highly granular data on student and teacher usage. These datasets could shed light on causal mechanisms, effect heterogeneity, or optimal use. However, there are methodological challenges: implementation is not randomized and is only defined for the treatment group, and log datasets have a complex structure. This article discusses three approaches to help surmount these issues. One approach uses data from the treatment group to estimate the effect of usage on outcomes in an observational study. Another, causal mediation analysis, estimates the role of usage in driving the overall effect. Finally, principal stratification estimates overall effects for groups of students with the same "potential" usage. We analyze hint data from an evaluation of the Cognitive Tutor Algebra I curriculum using these three approaches, with possibly conflicting results: the observational study and mediation analysis suggest that hints reduce posttest scores, while principal stratification finds that treatment effects may be correlated with higher rates of hint requests. We discuss these mixed conclusions and give broader methodological recommendations. (As Provided).
Anmerkungen	Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01