Literaturnachweis - Detailanzeige
Autor/in | Wind, Stefanie A. |
---|---|
Titel | A Sequential Approach to Detecting Differential Rater Functioning in Sparse Rater-Mediated Assessment Networks |
Quelle | In: Language Testing, 40 (2023) 2, S.209-226 (18 Seiten)Infoseite zur Zeitschrift
PDF als Volltext |
Zusatzinformation | ORCID (Wind, Stefanie A.) |
Sprache | englisch |
Dokumenttyp | gedruckt; online; Zeitschriftenaufsatz |
ISSN | 0265-5322 |
DOI | 10.1177/02655322221092388 |
Schlagwörter | Evaluators; Decision Making; Student Characteristics; Performance Based Assessment; Academic Achievement; Item Analysis; Accuracy; Writing Evaluation; Simulation; Sequential Approach; Language Tests; English; Language Arts; High School Students; Multiple Choice Tests; Writing Tests; Case Studies; Bias Decision-making; Entscheidungsfindung; Leistungsermittlung; Schulleistung; Itemanalyse; Simulation program; Simulationsprogramm; Schrittfolge; Language test; Sprachtest; English language; Englisch; Sprachkultur; High school; High schools; Student; Students; Oberschule; Schüler; Schülerin; Studentin; Multiple choice examinations; Multiple-choice tests, Multiple-choice examinations; Multiple-Choice-Verfahren; Writing test; Schreibtest; Case study; Fallstudie; Case Study |
Abstract | Researchers frequently evaluate rater judgments in performance assessments for evidence of differential rater functioning (DRF), which occurs when rater severity is systematically related to construct-irrelevant student characteristics after controlling for student achievement levels. However, researchers have observed that methods for detecting DRF may be limited in sparse rating designs, where it is not possible for every rater to score every student. In these designs, there is limited information with which to detect DRF. Sparse designs can also exacerbate the impact of artificial DRF, which occurs when raters are inaccurately flagged for DRF due to statistical artifacts. In this study, a sequential method is adapted from previous research on differential item functioning (DIF) that allows researchers to detect DRF more accurately and distinguish between true and artificial DRF. Analyses of data from a rater-mediated writing assessment and a simulation study demonstrate that the sequential approach results in different conclusions about which raters exhibit DRF. Moreover, the simulation study results suggest that the sequential procedure results in improved accuracy in DRF detection across a variety of rating design conditions. Practical implications for language testing research are discussed. (As Provided). |
Anmerkungen | SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com |
Erfasst von | ERIC (Education Resources Information Center), Washington, DC |
Update | 2024/1/01 |