A Recipe for Disappointment: Policy, Effect Size, and the Winner's Curse

Autor/in	Simpson, Adrian
Titel	A Recipe for Disappointment: Policy, Effect Size, and the Winner's Curse
Quelle	In: Journal of Research on Educational Effectiveness, 16 (2023) 4, S.643-662 (20 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Simpson, Adrian)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1934-5747
DOI	10.1080/19345747.2022.2066588
Schlagwörter	Effect Size; Educational Research; Evidence Based Practice; Foreign Countries; Intervention; Educational Policy; Research Problems; Computation; Randomized Controlled Trials; Statistical Analysis; United Kingdom + Suchen Sie Ihr Suchwort? Bildungsforschung; Pädagogische Forschung; Ausland; Politics of education; Bildungspolitik; Forschungskritik; Statistische Analyse; Großbritannien
Abstract	Evidence-based education aims to support policy makers choosing between potential interventions. This rarely involves considering each in isolation; instead, sets of evidence regarding many potential policy interventions are considered. Filtering a set on any quantity measured with error risks the "winner's curse": conditional on selecting higher valued measures, the measurement likely overestimates the latent value. This article explains the winner's curse, illustrates it for one constrained and complete set of educational trials--the UK's Education Endowment Foundation's projects, where evidence is summarized with standardized effect size--and shows the results of adjusting for the curse on this set. This analysis suggests selecting policies for higher effect size can result in substantial effect size inflation and in some cases order reversals: one intervention ranking above another on estimated effect size but below it when adjusted. The issue has implications for evaluation programs, power analyses, and policy decisions. For example, even in the absence of other problems with interpreting effect size, it can help explain why policies tend to deliver less than promised. (As Provided).
Anmerkungen	Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01