Determining the Scoring Validity of a Co-Constructed CEFR-Based Rating Scale

Autor/inn/en	Deygers, Bart; Van Gorp, Koen
Titel	Determining the Scoring Validity of a Co-Constructed CEFR-Based Rating Scale
Quelle	In: Language Testing, 32 (2015) 4, S.521-541 (21 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0265-5322
DOI	10.1177/0265532215575626
Schlagwörter	Rating Scales; Scoring; Validity; Interrater Reliability; Novices; Data Analysis; Error of Measurement; Research Problems; Item Response Theory; Componential Analysis; Focus Groups; Coding; Foreign Countries; Language Tests; Indo European Languages; Netherlands + Suchen Sie Ihr Suchwort? Rating-Skala; Bewertung; Gültigkeit; Interrater-Reliabilität; Auswertung; Messfehler; Forschungskritik; Item-Response-Theorie; Codierung; Programmierung; Ausland; Language test; Sprachtest; Indoeuropäisch; Niederlande
Abstract	Considering scoring validity as encompassing both reliable rating scale use and valid descriptor interpretation, this study reports on the validation of a CEFR-based scale that was co-constructed and used by novice raters. The research questions this paper wishes to answer are (a) whether it is possible to construct a CEFR-based rating scale with novice raters that yields reliable ratings and (b) allows for a uniform interpretation of the descriptors. Additionally, this study focuses on the question whether co-constructing a rating scale with novice raters helps to stimulate a shared interpretation of the descriptors over time. For this study, six novice raters employed a CEFR-based scale that had been co-constructed by themselves and 14 peers to rate 200 spoken and written performances in a missing data design. The quantitative data were analysed using item response theory, classical test theory and principal component analysis. The focus group data, collected after the rating process, were transcribed and coded using both a priori and inductive coding. The results indicate that novice raters can reliably use the CEFR-based rating scale, but that the interpretations of the descriptors, in spite of training and co-construction, are not as homogeneous as the inter-rater reliability would suggest. (As Provided).
Anmerkungen	SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: http://sagepub.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01