Who's the Best Detective? Large Language Models vs. Traditional Machine Learning in Detecting Incoherent Fourth Grade Math Answers

Autor/inn/en	Urrutia, Felipe; Araya, Roberto
Titel	Who's the Best Detective? Large Language Models vs. Traditional Machine Learning in Detecting Incoherent Fourth Grade Math Answers
Quelle	In: Journal of Educational Computing Research, 61 (2024) 8, S.187-218 (32 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Urrutia, Felipe) ORCID (Araya, Roberto)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0735-6331
DOI	10.1177/07356331231191174
Schlagwörter	Elementary School Students; Grade 4; Elementary School Mathematics; Mathematics Tests; Test Items; Test Wiseness; Automation; Artificial Intelligence; Computer Assisted Testing; Identification; Accuracy; Spelling; Natural Language Processing + Suchen Sie Ihr Suchwort? School year 04; 4. Schuljahr; Schuljahr 04; Elementare Mathematik; Schulmathematik; Test content; Testaufgabe; Künstliche Intelligenz; Identifikation; Identifizierung; Schreibweise; Natürliche Sprache
Abstract	Written answers to open-ended questions can have a higher long-term effect on learning than multiple-choice questions. However, it is critical that teachers immediately review the answers, and ask to redo those that are incoherent. This can be a difficult task and can be time-consuming for teachers. A possible solution is to automate the detection of incoherent answers. One option is to automate the review with Large Language Models (LLM). They have a powerful discursive ability that can be used to explain decisions. In this paper, we analyze the responses of fourth graders in mathematics using three LLMs: GPT-3, BLOOM, and YOU. We used them with zero, one, two, three and four shots. We compared their performance with the results of various classifiers trained with Machine Learning (ML). We found that LLMs perform worse than MLs in detecting incoherent answers. The difficulty seems to reside in recursive questions that contain both questions and answers, and in responses from students with typical fourth-grader misspellings. Upon closer examination, we have found that the ChatGPT model faces the same challenges. (As Provided).
Anmerkungen	SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01