An Analytic Framework for Evaluating the Validity of Concept Inventory Claims

Autor/inn/en	Jorion, Natalie; Gane, Brian D.; James, Katie; Schroeder, Lianne; DiBello, Louis V.; Pellegrino, James W.
Titel	An Analytic Framework for Evaluating the Validity of Concept Inventory Claims
Quelle	In: Journal of Engineering Education, 104 (2015) 4, S.454-496 (43 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1069-4730
DOI	10.1002/jee.20104
Schlagwörter	Guidelines; Validity; Inferences; Concept Formation; Scores; Misconceptions; Error Patterns; Measures (Individuals); Statistics; Reliability; Engineering Education; Student Evaluation + Suchen Sie Ihr Suchwort? Richtlinien; Gültigkeit; Inference; Inferenz; Concept learning; Begriffsbildung; Missverständnis; Fehlertyp; Messdaten; Statistik; Reliabilität; Ingenieurausbildung; Schulnote; Studentische Bewertung
Abstract	Background: Concept inventories (CIs) are commonly used in engineering disciplines to assess students' conceptual understanding and to evaluate instruction, but educators often use CIs without sufficient evidence that a structured approach has been applied to validate inferences about student thinking. Purpose: We propose an analytic framework for evaluating the validity arguments of CIs. We focus on three types of claims: that CI scores enable one to infer (1) students' overall understanding of all concepts identified in the CI, (2) students' understanding of specific concepts, and (3) students' propensity for misconceptions or common errors. Method: We applied our analytic framework to three CIs: the Concept Assessment Tool for Statics (CATS), the Statistics Concept Inventory (SCI), and the Dynamics Concept Inventory (DCI). Results: Using our analytic framework, we found varying degrees of support for each type of claim. CATS and DCI analyses indicated that the CIs could reliably measure students' overall understanding of all concepts identified in the CI, whereas SCI analyses provided limited evidence for this claim. Analyses revealed that the CATS could accurately measure students' understanding of specific concepts; analyses for the other two CIs did not support this claim. None of the CI analyses provided evidence that the instruments could reliably measure students' misconceptions and common errors. Conclusions: Our analytic framework provides a structure for evaluating CI validity. Engineering educators can apply this framework to evaluate aspects of CI validity and make more warranted uses and interpretations of CI outcome scores. (As Provided).
Anmerkungen	Wiley Periodicals, Inc. 350 Main Street, Malden, MA 02148. Tel: 800-835-6770; Tel: 781-388-8598; Fax: 781-388-8232; e-mail: cs-journals@wiley.com; Web site: http://www.wiley.com/WileyCDA
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01