Evaluating False Positive Rates of Standard and Hierarchical Measures of Metacognitive Accuracy

Autor/inn/en	Rausch, Manuel; Zehetleitner, Michael
Titel	Evaluating False Positive Rates of Standard and Hierarchical Measures of Metacognitive Accuracy
Quelle	In: Metacognition and Learning, 18 (2023) 3, S.863-889 (27 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Rausch, Manuel) ORCID (Zehetleitner, Michael)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1556-1623
DOI	10.1007/s11409-023-09353-y
Schlagwörter	Metacognition; Measurement; Accuracy; Vertical Organization; Research Design; Evaluation + Suchen Sie Ihr Suchwort? Meta cognitive ability; Meta-cognition; Metakognitive Fähigkeit; Metakognition; Messverfahren; Forschungsdesign; Evaluierung
Abstract	A key aspect of metacognition is metacognitive accuracy, i.e., the degree to which confidence judgments differentiate between correct and incorrect trials. To quantify metacognitive accuracy, researchers are faced with an increasing number of different methods. The present study investigated false positive rates associated with various measures of metacognitive accuracy by hierarchical resampling from the confidence database to accurately represent the statistical properties of confidence judgements. We found that most measures based on the computation of summary-statistics separately for each participant and subsequent group-level analysis performed adequately in terms of false positive rate, including gamma correlations, meta-d', and the area under type 2 ROC curves. Meta-d'/d' is associated with a false positive rate even below 5%, but log-transformed meta-d'/d' performs adequately. The false positive rate of HMeta-d depends on the study design and on prior specification: For group designs, the false positive rate is above 5% when independent priors are placed on both groups, but the false positive rate is adequate when a prior was placed on the difference between groups. For continuous predictor variables, default priors resulted in a false positive rate below 5%, but the false positive rate was not distinguishable from 5% when close-to-flat priors were used. Logistic mixed model regression analysis is associated with dramatically inflated false positive rates when random slopes are omitted from model specification. In general, we argue that no measure of metacognitive accuracy should be used unless the false positive rate has been demonstrated to be adequate. (As Provided).
Anmerkungen	Springer. Available from: Springer Nature. One New York Plaza, Suite 4600, New York, NY 10004. Tel: 800-777-4643; Tel: 212-460-1500; Fax: 212-460-1700; e-mail: customerservice@springernature.com; Web site: https://link.springer.com/
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01