Peer Review in Agricultural Education: Interrater Reliability of Manuscript Reviews for the 2014 National Agricultural Education Research Conference

Autor/inn/en	Shoulders, Catherine W.; Johnson, Donald M.; Flowers, Jim
Titel	Peer Review in Agricultural Education: Interrater Reliability of Manuscript Reviews for the 2014 National Agricultural Education Research Conference
Quelle	In: Journal of Agricultural Education, 56 (2015) 3, S.121-133 (13 Seiten) PDF als Volltext kostenfreie Datei Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1042-0541
Schlagwörter	Peer Evaluation; Agricultural Education; Educational Research; Interrater Reliability; Publications; Conferences (Gatherings) + Suchen Sie Ihr Suchwort? Agriculture; Education; Landwirtschaftliche Ausbildung; Landwirtschaft; Ausbildung; Bildungsforschung; Pädagogische Forschung; Interrater-Reliabilität
Abstract	This study analyzed 336 peer reviews of 112 manuscripts submitted for possible presentation at the 2014 National Agricultural Education Research Conference (NAERC). There were scoring errors on 6.8% of the reviews; the most frequent errors were failure to record a score or assigning a score above the range of points possible for one or more of the review criteria. The coefficient of variation (CV) for same-paper ratings of manuscript quality ranged from 3.94% to 96.43% with a mean of 19.65% (SD = 15.22%). The interrater reliability for same-paper evaluations of manuscript quality was 0.15. The CVs for reviewer same-paper reject-accept recommendations ranged from 0.00% to 98.97% with a mean of 38.80% (SD = 22.72%). The interrater reliability for same-paper reject-accept recommendations was .09. On 82 (73.21%) manuscripts all three reviewers agreed on the manuscript's relevance to agricultural education. Mean manuscript quality scores explained 68.3% of the variance in mean reject-accept recommendations leaving 31.7% of the variance unexplained. There were no significant (p > 0.05) differences by academic rank in ratings of manuscript quality, reject-accept recommendations, or assessments of relevance to agricultural education. The authors offered recommendations for improvement of the NAERC manuscript review process. (As Provided).
Anmerkungen	American Association for Agricultural Education. P.O. Box 7607, Department of Agricultural and Extension Education, Raleigh, NC 27695. Web site: http://www.aaaeonline.org
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01