Coding Energy Knowledge in Constructed Responses with Explainable NLP Models

Autor/inn/en	Gombert, Sebastian; Di Mitri, Daniele; Karademir, Onur; Kubsch, Marcus; Kolbe, Hannah; Tautz, Simon; Grimm, Adrian; Bohm, Isabell; Neumann, Knut; Drachsler, Hendrik
Titel	Coding Energy Knowledge in Constructed Responses with Explainable NLP Models
Quelle	In: Journal of Computer Assisted Learning, 39 (2023) 3, S.767-786 (20 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Gombert, Sebastian) ORCID (Di Mitri, Daniele) ORCID (Karademir, Onur) ORCID (Kubsch, Marcus) ORCID (Grimm, Adrian) ORCID (Bohm, Isabell) ORCID (Neumann, Knut) ORCID (Drachsler, Hendrik)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0266-4909
DOI	10.1111/jcal.12767
Schlagwörter	Coding; Energy; Scientific Concepts; Formative Evaluation; Student Evaluation; Test Items; Science Tests; Natural Language Processing; Elementary School Students; Secondary School Students; Foreign Countries; Artificial Intelligence; Models; Germany + Suchen Sie Ihr Suchwort? Codierung; Programmierung; Energie; Schulnote; Studentische Bewertung; Test content; Testaufgabe; Natürliche Sprache; Sekundarschüler; Ausland; Künstliche Intelligenz; Analogiemodell; Deutschland
Abstract	Background: Formative assessments are needed to enable monitoring how student knowledge develops throughout a unit. Constructed response items which require learners to formulate their own free-text responses are well suited for testing their active knowledge. However, assessing such constructed responses in an automated fashion is a complex task and requires the application of natural language processing methodology. In this article, we implement and evaluate multiple machine learning models for coding energy knowledge in free-text responses of German K-12 students to items in formative science assessments which were conducted during synchronous online learning sessions. Dataset: The dataset we collected for this purpose consists of German constructed responses from 38 different items dealing with aspects of energy such as manifestation and transformation. The units and items were implemented with the help of project-based pedagogy and evidence-centered design, and the responses were coded for seven core ideas concerning the manifestation and transformation of energy. The data was collected from students in seventh, eighth and ninth grade. Methodology: We train various transformer- and feature-based models and compare their ability to recognize the respective ideas in students' writing. Moreover, as domain knowledge and its development can be formally modeled through knowledge networks, we evaluate how well the detection of the ideas within responses translated into accurate co-occurrence-based knowledge networks. Finally, in terms of the descriptive accuracy of our models, we inspect what features played a role for which prediction outcome and if the models pick up on undesired shortcuts. In addition to this, we analyze how much the models match human coders in what evidence within responses they consider important for their coding decisions. Results: A model based on a modified GBERT-large can achieve the overall most promising results, although descriptive accuracy varies much more than predictive accuracy for the different ideas assessed. For reasons of comparability, we also evaluate the same machine learning architecture using the SciEntsBank 3-Way benchmark with an English RoBERTa-large model, where it achieves state-of-the-art results in two out of three evaluation categories. (As Provided).
Anmerkungen	Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01