Detecting Rater Centrality Effects in Performance Assessments: A Model-Based Comparison of Centrality Indices

Autor/inn/en	Jin, Kuan-Yu; Eckes, Thomas
Titel	Detecting Rater Centrality Effects in Performance Assessments: A Model-Based Comparison of Centrality Indices
Quelle	In: Measurement: Interdisciplinary Research and Perspectives, 20 (2022) 4, S.228-247 (20 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Jin, Kuan-Yu) ORCID (Eckes, Thomas)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1536-6367
DOI	10.1080/15366367.2021.1972654
Schlagwörter	Performance Based Assessment; Evaluators; Scoring; Sample Size; Evaluation Criteria; Statistics; Correlation; Rating Scales; Difficulty Level; Language Tests; German + Suchen Sie Ihr Suchwort? Leistungsermittlung; Bewertung; Statistik; Korrelation; Rating-Skala; Schwierigkeitsgrad; Language test; Sprachtest; Deutscher
Abstract	Recent research on rater effects in performance assessments has increasingly focused on rater centrality, the tendency to assign scores clustering around the rating scale's middle categories. In the present paper, we adopted Jin and Wang's (2018) extended facets modeling approach and constructed a centrality continuum, ranging from raters exhibiting strong central tendencies to raters exhibiting strong tendencies in the opposite direction (extremity). In two simulation studies, we examined three model-based centrality detection indices (rater infit statistics, residual-expected correlations, and rater threshold SDs) as well as the raw-score SD in terms of their efficiency of reconstructing the true rater centrality rank ordering. Findings confirmed the superiority of the residual-expected correlation, rater threshold SD, and raw-score SD statistics, particularly when the examinee sample size was large and the number of scoring criteria was high. By contrast, the infit statistic results were much less consistent and, under conditions of large differences between criterion difficulties, suggested erroneous conclusions about raters' central tendencies. Analyzing real rating data from a large-scale speaking performance assessment confirmed that infit statistics are unsuitable for identifying raters' central tendencies. The discussion focuses on detecting centrality effects under different facets models and the indices' implications for rater monitoring and fair performance assessment. (As Provided).
Anmerkungen	Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01