Setting standards to a scientific literacy test for adults using Item-Descriptor (ID) matching method.

Autor/inn/en	Haschke, Linda; Kampa, Nele; Hahn, Inga; Köller, Olaf
Titel	Setting standards to a scientific literacy test for adults using Item-Descriptor (ID) matching method.
Quelle	Aus: Blömeke, Sigrid (Hrsg.); Gustafsson, Jan-Eric (Hrsg.): Standard Setting in Education. The Nordic Countries in an International Perspective. Cham: Springer (2017) S. 319-339 PDF als Volltext Link als defekt meldenVerfügbarkeit
Reihe	Methodology of Educational Measurement and Assessment. 7
Beigaben	Literaturangaben
Sprache	englisch
Dokumenttyp	online; gedruckt; Sammelwerksbeitrag
ISBN	978-3-319-50855-9; 978-3-319-50856-6
DOI	10.1007/978-3-319-50856-6_18
Schlagwörter	Methode; Itemanalyse; Standardisierter Test; Erwachsener; Methode; Itemanalyse; Item; Naturwissenschaftliche Bildung; Naturwissenschaftliche Bildung; Item; Standard; Erwachsener
Abstract	Common standard setting methods such as the Angoff or the Bookmark method require panellists to imagine minimally competent persons or to estimate response probabilities, in order to define cut scores. Imagining these persons and how they would perform is criticised as cognitively demanding. These already challenging judgemental tasks become even more difficult, when experts have to deal with very heterogeneous or insufficiently studied populations, such as adults. The Item-Descriptor (ID) Matching method can reduce the arbitrariness of such subjective evaluations by focusing on rather objective judgements about the content of tests. At our standard setting workshop, seven experts had to match the item demands of 22 items of a scientific literacy test for adults with abilities described by performance level descriptions (PLDs) of the two proficiency levels Basic and Advanced. Since the ID Matching method has hardly been used in European standard settings, the method has not been evaluated comprehensively. In order to evaluate the appropriateness and correct interpretation of cut scores, information about the validity of standard setting methods is essential. In this chapter, we aim to provide procedural and internal evidence for the use and interpretation of the derived cut scores and PLDs using the ID Matching method. With regard to procedural validity, we report high and consensual agreement of the experts regarding explicitness, practicability, implementation, and feedback, which we assessed by detailed questionnaires. The inter-rater reliability for the panellists' classification of items was low, but increased during subsequent rounds (k = .38 to k = .63). The values are consistent with findings of earlier studies which support internal validity. We argue that the cut scores and PLDs derived from the application of the ID Matching method are appropriate to categorise adults as scientifically illiterate, literate, and advanced literate (Orig.).
Erfasst von	DIPF \| Leibniz-Institut für Bildungsforschung und Bildungsinformation, Frankfurt am Main
Update	2020/3