Raters' Perceptions of Rating Scales Criteria and Its Effect on the Process and Outcome of Their Rating

Autor/inn/en	Heidari, Nasim; Ghanbari, Nasim; Abbasi, Abbas
Titel	Raters' Perceptions of Rating Scales Criteria and Its Effect on the Process and Outcome of Their Rating
Quelle	In: Language Testing in Asia, 12 (2022), Artikel 20 (19 Seiten) PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Ghanbari, Nasim)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
DOI	10.1186/s40468-022-00168-3
Schlagwörter	Evaluators; Rating Scales; Language Tests; English (Second Language); Second Language Learning; Essays; Writing Evaluation; Second Language Instruction; Profiles; Protocol Analysis; Scoring; Holistic Approach; Decision Making; Training; Native Language; Attitudes; Evaluation Criteria; International English Language Testing System + Suchen Sie Ihr Suchwort? Rating-Skala; Language test; Sprachtest; English as second language; English; Second Language; Englisch als Zweitsprache; Zweitsprachenerwerb; Essay; Aufsatzunterricht; Fremdsprachenunterricht; Charakterisierung; Profilanalyse; Bewertung; Holistischer Ansatz; Decision-making; Entscheidungsfindung; Ausbildung; Attitude; Einstellung; Verhalten; Language tests; Englisch
Abstract	It is widely believed that human rating performance is influenced by an array of different factors. Among these, rater-related variables such as experience, language background, perceptions, and attitudes have been mentioned. One of the important rater-related factors is the way the raters interact with the rating scales. In particular, how raters perceive the components of the scales to further plan their scoring seems important. For this aim, the present study investigated the raters' perceptions of the rating scales and their subsequent rating behaviors for two analytic and holistic rating scales. Hence, nine highly experienced raters were asked to verbalize their thoughts while rating student essays using IELTS holistic scale and the analytic scale of ESL Composition Profile. Upon analyzing the think-aloud protocols, four themes emerged. The findings showed that when rating holistically, the raters either referred to the holistic scale components to validate their ratings (validation) or had a pre-evaluation reading to rate in a more reliable way (dominancy). In analytic rating, on the other hand, the raters used a pre-evaluation scale reading in order to keep the components and their criteria to memory to evaluate the text more accurately (dominancy) or regularly moved between the text and the scale components to assign a score (oscillation). Furthermore, the results of a Wilcoxon signed-rank test showed that when using the holistic and analytic rating scales, the raters assigned significantly different scores to the texts. On the whole, the results revealed that the way the raters perceived the scale components will affect their judgement of the texts. The study also provides several implications for rater training programs and EFL writing assessment. (As Provided).
Anmerkungen	Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01