What Predicts Variation in Reliability and Validity of Online Peer Assessment? A Large-Scale Cross-Context Study

Autor/inn/en	Xiong, Yao; Schunn, Christian D.; Wu, Yong
Titel	What Predicts Variation in Reliability and Validity of Online Peer Assessment? A Large-Scale Cross-Context Study
Quelle	In: Journal of Computer Assisted Learning, 39 (2023) 6, S.2004-2024 (21 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Schunn, Christian D.) ORCID (Wu, Yong)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0266-4909
DOI	10.1111/jcal.12861
Schlagwörter	Peer Evaluation; Computer Assisted Testing; Test Reliability; Test Validity; Prediction + Suchen Sie Ihr Suchwort? Testreliabilität; Testvalidität; Vorhersage
Abstract	Background: For peer assessment, reliability (i.e., consistency in ratings across peers) and validity (i.e., consistency of peer ratings with instructors or experts) are frequently examined in the research literature to address a central concern of instructors and students. Although the average levels are generally promising, both reliability and validity can vary substantially from context to context. Meta-analyses have identified a few moderators that are related to peer assessment reliability/validity, but they have lacked statistical power to systematically investigate many moderators or disentangle correlated moderators. Objectives: The current study fills this gap by addressing what variables influence peer assessment reliability/validity using a large-scale, cross-context dataset from a shared online peer assessment platform. Methods: Using multi-level structural equation models, we examined three categories of variables: (1) variables related to the context of peer assessment; (2) variables related to the peer assessment task itself; and (3) variables related to rating rubrics of peer assessment. Results and Conclusions: We found that the extent to which assessment documents varied in quality on the given rubric played a central role in mediating the effect from different predictors to peer assessment reliability/validity. Other variables that are significantly associated with reliability and validity included: Education Level, Language, Discipline, Average Ability of Peer Raters, Draft Number, Assignment Number, Class Size, Average Number of Raters, and Length of Rubric Description. The results provide information to guide practitioners on how to improve reliability and validity of peer assessments. (As Provided).
Anmerkungen	Wiley. Available from: John Wiley & Sons, Inc. 111 River Street, Hoboken, NJ 07030. Tel: 800-835-6770; e-mail: cs-journals@wiley.com; Web site: https://www.wiley.com/en-us
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01