An Experimental Study of the Internal Consistency of Judgments Made in Bookmark Standard Setting.

Autor/inn/en	Clauser, Brian E; Baldwin, Peter; Margolis, Melissa J; Mee, Janet; Winward, Marcia
Titel	An Experimental Study of the Internal Consistency of Judgments Made in Bookmark Standard Setting.
Quelle	In: Journal of educational measurement, (2017) 4, S.481-497Infoseite zur Zeitschrift PDF als Volltext (1); PDF als Volltext (2); PDF als Volltext (3) Verfügbarkeit
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0022-0655
DOI	10.1111/jedm.12157
Schlagwörter	Educational evaluation; Performance standards; Experts; Judgments; Educational tests & measurements
Abstract	Validating performance standards is challenging and complex. Because of the difficulties associated with collecting evidence related to external criteria, validity arguments rely heavily on evidence related to internal criteria—especially evidence that expert judgments are internally consistent. Given its importance, it is somewhat surprising that evidence of this kind has rarely been published in the context of the widely used bookmark standard‐setting procedure. In this article we examined the effect of ordered item booklet difficulty on content experts’ bookmark judgments. If panelists make internally consistent judgments, their resultant cut scores should be unaffected by the difficulty of their respective booklets. This internal consistency was not observed: the results suggest that substantial systematic differences in the resultant cut scores can arise when the difficulty of the ordered item booklets varies. These findings raise questions about the ability of content experts to make the judgments required by the bookmark procedure.
Erfasst von	OLC
Update	2023/3/07