Statistically Comparing the Performance of Multiple Automated Raters across Multiple Items

Autor/inn/en	Kieftenbeld, Vincent; Boyer, Michelle
Titel	Statistically Comparing the Performance of Multiple Automated Raters across Multiple Items
Quelle	In: Applied Measurement in Education, 30 (2017) 2, S.117-128 (12 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0895-7347
DOI	10.1080/08957347.2017.1283316
Schlagwörter	Automation; Scoring; Comparative Analysis; Test Items; Statistical Analysis; Differences; Interrater Reliability; Essay Tests; Test Format; Competition + Suchen Sie Ihr Suchwort? Bewertung; Test content; Testaufgabe; Statistische Analyse; Unterscheiden; Interrater-Reliabilität; Schriftlicher Sprachgebrauch; Testentwicklung; Wettkampf
Abstract	Automated scoring systems are typically evaluated by comparing the performance of a single automated rater item-by-item to human raters. This presents a challenge when the performance of multiple raters needs to be compared across multiple items. Rankings could depend on specifics of the ranking procedure; observed differences could be due to random sampling of items and/or responses in the validation sets. Any statistical hypothesis test of the differences in rankings needs to be appropriate for use with rater statistics and adjust for multiple comparisons. This study considered different statistical methods to evaluate differences in performance across multiple raters and items. These methods are illustrated leveraging data from the 2012 Automated Scoring Assessment Prize competitions. Using average rankings to test for significant differences in performance between automated and human raters, findings show that most automated raters did not perform statistically significantly different from human-to-human inter-rater agreement for essays but they did perform differently on short-answer items. Differences in average rankings between most automated raters were not statistically significant, even when their observed performance differed substantially. (As Provided).
Anmerkungen	Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01