Exploring Rater Accuracy Using Unfolding Models Combined with Topic Models: Incorporating Supervised Latent Dirichlet Allocation

Autor/inn/en	Wheeler, Jordan M.; Engelhard, George; Wang, Jue
Titel	Exploring Rater Accuracy Using Unfolding Models Combined with Topic Models: Incorporating Supervised Latent Dirichlet Allocation
Quelle	In: Measurement: Interdisciplinary Research and Perspectives, 20 (2022) 1, S.34-46 (13 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Wheeler, Jordan M.) ORCID (Engelhard, George) ORCID (Wang, Jue)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1536-6367
DOI	10.1080/15366367.2021.1915094
Schlagwörter	Accuracy; Scoring; Statistical Analysis; Models; Writing Tests; Essay Tests + Suchen Sie Ihr Suchwort? Bewertung; Statistische Analyse; Analogiemodell; Writing test; Schreibtest; Schriftlicher Sprachgebrauch
Abstract	Objectively scoring constructed-response items on educational assessments has long been a challenge due to the use of human raters. Even well-trained raters using a rubric can inaccurately assess essays. Unfolding models measure rater's scoring accuracy by capturing the discrepancy between criterion and operational ratings by placing essays on an unfolding continuum with an ideal-point location. Essay unfolding locations indicate how difficult it is for raters to score an essay accurately. This study aims to explore a substantive interpretation of the unfolding scale based on a supervised Latent Dirichlet Allocation (sLDA) model. We investigate the relationship between latent topics extracted using sLDA and unfolding locations with a sample of essays (n = 100) obtained from an integrated writing assessment. Results show that (a) three latent topics moderately explain (r[superscript 2] = 0.561) essay locations defined by the unfolding scale and (b) failing to use and/or cite the source articles led to essays that are difficult-to-score accurately. (As Provided).
Anmerkungen	Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01