When Does Disengagement Correlate with Performance in Spoken Dialog Computer Tutoring?

Autor/inn/en	Forbes-Riley, Kate; Litman, Diane
Titel	When Does Disengagement Correlate with Performance in Spoken Dialog Computer Tutoring?
Quelle	In: International Journal of Artificial Intelligence in Education, 22 (2013) 1-2, S.39-58 (20 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1560-4292
DOI	10.3233/JAI-130028
Schlagwörter	Correlation; Learner Engagement; Oral Language; Computer Assisted Instruction; Computational Linguistics; Evaluators; Prediction; Student Attitudes; Learning Processes; Intelligent Tutoring Systems; Physics; College Students; Natural Language Processing; Science Instruction; Dialogs (Language); Pennsylvania (Pittsburgh) + Suchen Sie Ihr Suchwort? Korrelation; Oral interpretation; Mündlicher Sprachgebrauch; Computer based training; Computerunterstützter Unterricht; Linguistics; Computerlinguistik; Vorhersage; Schülerverhalten; Learning process; Lernprozess; Intelligentes Tutorsystem; Physik; Collegestudent; Natürliche Sprache; Teaching of science; Science education; Natural sciences Lessons; Naturwissenschaftlicher Unterricht; Dialog; Dialogs; Dialogue; Dialogues
Abstract	In this paper we investigate how student disengagement relates to two performance metrics in a spoken dialog computer tutoring corpus, both when disengagement is measured through manual annotation by a trained human judge, and also when disengagement is measured through automatic annotation by the system based on a machine learning model. First, we investigate whether manually labeled overall disengagement and six different disengagement types are predictive of learning and user satisfaction in the corpus. Our results show that although students' percentage of overall disengaged turns negatively correlates both with the amount they learn and their user satisfaction, the individual types of disengagement correlate differently: some negatively correlate with learning and user satisfaction, while others don't correlate with either metric at all. Moreover, these relationships change somewhat depending on student prerequisite knowledge level. Furthermore, using multiple disengagement types to predict learning improves predictive power. Overall, these manual label-based results suggest that although adapting to disengagement should improve both student learning and user satisfaction in computer tutoring, maximizing performance requires the system to detect and respond differently based on disengagement type. Next, we present an approach to automatically detecting and responding to user disengagement types based on their differing correlations with correctness. Investigation of our machine learning model of user disengagement shows that its automatic labels negatively correlate with both performance metrics in the same way as the manual labels. The similarity of the correlations across the manual and automatic labels suggests that the automatic labels are a reasonable substitute for the manual labels. Moreover, the significant negative correlations themselves suggest that redesigning ITSPOKE to automatically detect and respond to disengagement has the potential to remediate disengagement and thereby improve performance, even in the presence of noise introduced by the automatic detection process. (As Provided).
Anmerkungen	IOS Press. Nieuwe Hemweg 6B, Amsterdam, 1013 BG, The Netherlands. Tel: +31-20-688-3355; Fax: +31-20-687-0039; e-mail: info@iospress.nl; Web site: http://www.iospress.nl
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01