Item Calibration Samples and the Stability of Achievement Estimates and System Rankings: Another Look at the PISA Model

Autor/inn/en	Rutkowski, Leslie; Rutkowski, David; Zhou, Yan
Titel	Item Calibration Samples and the Stability of Achievement Estimates and System Rankings: Another Look at the PISA Model
Quelle	In: International Journal of Testing, 16 (2016) 1, S.1-20 (20 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1530-5058
DOI	10.1080/15305058.2015.1036163
Schlagwörter	Simulation; International Programs; Adolescents; Student Evaluation; Academic Achievement; Achievement Tests; Test Items; Test Bias; Testing Accommodations; Item Response Theory; Foreign Countries; Computation; Evaluation Methods; Program for International Student Assessment + Suchen Sie Ihr Suchwort? Simulation program; Simulationsprogramm; Adolescent; Adolescence; Adoleszenz; Jugend; Jugendalter; Jugendlicher; Schulnote; Studentische Bewertung; Schulleistung; Achievement test; Achievement; Testing; Test; Tests; Leistungsbeurteilung; Leistungsüberprüfung; Leistung; Testdurchführung; Testen; Test content; Testaufgabe; Testkritik; Testing process; Accessibility (for disabled); Accessibility; Disabled person; Barrierefreiheit; Zugänglichkeit; Behinderter; Item-Response-Theorie; Ausland
Abstract	Using an empirically-based simulation study, we show that typically used methods of choosing an item calibration sample have significant impacts on achievement bias and system rankings. We examine whether recent PISA accommodations, especially for lower performing participants, can mitigate some of this bias. Our findings indicate that standard operational methods, while not ideal, recover underlying proficiency reasonably well and generally outperform methods that more completely include all participants. Translating results onto the PISA scale, the calibration sample can induce bias of up to 12.49 points, which is important given that standard errors are around three points. Although ranking correlations are at least 0.95, we note the policy implications of slight ranking changes. Our findings indicate that limited accommodations targeted at low achieving educational systems do not outperform either of the other methods considered. Research that further explores accommodations for heterogeneous populations is recommended. (As Provided).
Anmerkungen	Routledge. Available from: Taylor & Francis, Ltd. 325 Chestnut Street Suite 800, Philadelphia, PA 19106. Tel: 800-354-1420; Fax: 215-625-2940; Web site: http://www.tandf.co.uk/journals
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2020/1/01