Impact of Categorization and Scaling on Classification Agreement and Prediction Accuracy Statistics. Research Report. ETS RR-21-26

Autor/inn/en	Wang, Wei; Dorans, Neil J.
Titel	Impact of Categorization and Scaling on Classification Agreement and Prediction Accuracy Statistics. Research Report. ETS RR-21-26
Quelle	In: ETS Research Report Series, (2021), (20 Seiten)Infoseite zur Zeitschrift PDF als Volltext kostenfreie Datei Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	2330-8516
Schlagwörter	Classification; Scaling; Prediction; Accuracy; Statistics; Measurement; Correlation; Scoring; Computer Assisted Testing; Automation; Essays + Suchen Sie Ihr Suchwort? Classification system; Klassifikation; Klassifikationssystem; Scale construction; Skalenkonstruktion; Vorhersage; Statistik; Messverfahren; Korrelation; Bewertung; Essay; Aufsatzunterricht
Abstract	Agreement statistics and measures of prediction accuracy are often used to assess the quality of two measures of a construct. Agreement statistics are appropriate for measures that are supposed to be interchangeable, whereas prediction accuracy statistics are appropriate for situations where one variable is the target and the other variables are predictors. Using bivariate normality assumptions, we analytically examine the impact of categorization of a continuous variable and mean/sigma scaling on different measures of agreement and different measures of prediction accuracy. We vary the degree of relationship (squared correlation) between two continuous measures of a construct and the degree to which these measures are reduced to fewer and fewer categories (categorization). The main findings include that (a) categorization influences all the statistics investigated, (b) the correlation between the continuous variables affects the values of the statistics, and (c) scaling a prediction of a target variable to have the same mean and variability as the target increases agreement (according to Cohen's kappa and quadratic weighted kappa) but does so at the expense of prediction accuracy. The implications of these results for scoring of essays by humans or machines are also discussed. (As Provided).
Anmerkungen	Educational Testing Service. Rosedale Road, MS19-R Princeton, NJ 08541. Tel: 609-921-9000; Fax: 609-734-5410; e-mail: RDweb@ets.org; Web site: https://www.ets.org/research/policy_research_reports/ets
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01