Improving Test Security and Efficiency of Computerized Adaptive Testing for the Force Concept Inventory

Autor/inn/en	Yasuda, Jun-ichiro; Hull, Michael M.; Mae, Naohiro
Titel	Improving Test Security and Efficiency of Computerized Adaptive Testing for the Force Concept Inventory
Quelle	In: Physical Review Physics Education Research, 18 (2022) 1, Artikel 010112 (15 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Yasuda, Jun-ichiro) ORCID (Hull, Michael M.)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
Schlagwörter	Adaptive Testing; Computer Assisted Testing; Test Items; Risk Management; Test Length; Accuracy; Measurement; Science Tests; Physics; Scientific Concepts; College Students; Foreign Countries; Test Validity; Japan; Force Concept Inventory + Suchen Sie Ihr Suchwort? Test content; Testaufgabe; Risikomanagement; Messverfahren; Physik; Collegestudent; Ausland; Testvalidität
Abstract	This paper presents improvements made to a computerized adaptive testing (CAT)-based version of the FCI (FCI-CAT) in regards to test security and test efficiency. First, we will discuss measures to enhance test security by controlling for item overexposure, decreasing the risk that respondents may (i) memorize the content of a pretest for use on the post-test or (ii) share information about the items with their classmates who take the assessment later. Second, we will discuss measures to enhance test efficiency, so that a shorter test length can yield a desired accuracy and precision of the measurement. Specifically, we utilized collateral information in the form of a pretest proficiency estimate of each respondent for selecting items and estimating respondent proficiency level in the post-test. To shorten the total testing time further, we also allowed the test lengths to be different for the pre- and post-test. To analyze how these improvements affect the accuracy and precision (which we measure in terms of root-mean-square error) of Cohen's d, we conducted a Monte Carlo simulation and a "post hoc" simulation. Then, we calculated the minimal test length of the FCI-CAT whose accuracy and precision are equivalent to that of the paper-and-pencil version of the FCI. Consequently, we obtained the following three findings: (i) By using collateral information, we can achieve the accuracy and precision of the full-length FCI with fewer items via the FCI-CAT. (ii) For a class size of 40, we can control for test security while still reducing the sum of the pre- and post-test lengths of the FCI-CAT to a total of 33 items (17 items on the pretest and 16 items on the post-test), thereby reducing the testing time to 55%. (iii) If one's goal is to maximize test efficiency, the pretest length should be slightly larger than the post-test length. On the other hand, if the goal is to maximize test security, the pretest length should be smaller and the post-test length should be larger. If one desires a balance of these two goals, it would be reasonable to choose equal pre- and post-test lengths. (As Provided).
Anmerkungen	American Physical Society. One Physics Ellipse 4th Floor, College Park, MD 20740-3844. Tel: 301-209-3200; Fax: 301-209-0865; e-mail: assocpub@aps.org; Web site: http://prst-per.aps.org
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01