Is It Who You Are or Where You Are? Accounting for Compositional Differences in Cross-Site Treatment Effect Variation

Autor/inn/en	Lu, Benjamin; Ben-Michael, Eli; Feller, Avi; Miratrix, Luke
Titel	Is It Who You Are or Where You Are? Accounting for Compositional Differences in Cross-Site Treatment Effect Variation
Quelle	In: Journal of Educational and Behavioral Statistics, 48 (2023) 4, S.420-453 (34 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	Weitere Informationen
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1076-9986
DOI	10.3102/10769986231155427
Schlagwörter	Statistical Analysis; Statistical Distributions; Program Implementation; Comparative Analysis; Program Evaluation; Sample Size; Adult Education; Vocational Education; Computation; Mathematical Formulas; Randomized Controlled Trials + Suchen Sie Ihr Suchwort? Statistische Analyse; Wahrscheinlichkeitsverteilung; Programme evaluation; Programmevaluation; Adult; Adults; Education; Adult basic education; Adult training; Erwachsenenbildung; Ausbildung; Berufsbildung; Mathematische Formel
Abstract	In multisite trials, learning about treatment effect variation across sites is critical for understanding where and for whom a program works. Unadjusted comparisons, however, capture "compositional" differences in the distributions of unit-level features as well as "contextual" differences in site-level features, including possible differences in program implementation. Our goal in this article is to adjust site-level estimates for differences in the distribution of observed unit-level features: If we can reweight (or "transport") each site to have a common distribution of observed unit-level covariates, the remaining treatment effect variation captures contextual and unobserved compositional differences across sites. This allows us to make apples-to-apples comparisons across sites, parceling out the amount of cross-site effect variation explained by systematic differences in populations served. In this article, we develop a framework for transporting effects using approximate balancing weights, where the weights are chosen to directly optimize unit-level covariate balance between each site and the common target distribution. We first develop our approach for the general setting of transporting the effect of a single-site trial. We then extend our method to multisite trials, assess its performance via simulation, and use it to analyze a series of multisite trials of adult education and vocational training programs. In our application, we find that distributional differences are potentially masking cross-site variation. Our method is available in the balancer R package. [For the corresponding grantee submission, see ED628289.] (As Provided).
Anmerkungen	SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01