Literaturnachweis - Detailanzeige
| Autor/in | Ross, Linette P. |
|---|---|
| Titel | Test Fraud: Practical Applications and Operational Considerations for the Detection of Item Preknowledge and Compromised Content with Real Data |
| Quelle | (2022), (217 Seiten)
PDF als Volltext Ph.D. Dissertation, University of Delaware |
| Sprache | englisch |
| Dokumenttyp | gedruckt; online; Monographie |
| ISBN | 979-8-2098-9217-5 |
| Schlagwörter | Hochschulschrift; Dissertation; Testing; Deception; Cheating; Identification; Prior Learning; Item Response Theory; Difficulty Level; Test Items; Error Patterns; Bayesian Statistics; Statistical Inference |
| Abstract | One of the most serious forms of cheating occurs when examinees have item preknowledge and prior access to secure test material before taking an exam for the purpose of obtaining an inflated test score. Examinees that cheat and have prior knowledge of test content before testing may have an unfair advantage over examinees that do not cheat. Item preknowledge leads to compromised content and threatens the integrity of testing programs and the validity of the scores reported. Monitoring methods for detecting item preknowledge are needed to detect unexpected group behavior and identify potentially compromised content. This study examined the effectiveness of using data forensics and sequential monitoring of item performance with real data to detect item preknowledge and known compromised content. The study also examined the effect of compromised content on item performance and examinee ability estimates. The main detection methods used were sequential monitoring of item difficulty using both CTT and IRT methodologies. Two exams with known compromised content administered to school level cohorts of examinees were studied. Items on each exam were classified as exposed or non-exposed. For each exam, the power and Type 1 error rates for the two methodologies were compared for three levels of item exposure (Low, Medium, and High) represented by three forms of the exam. The item performance and ability estimates of examinee groups were considered when evaluating the two methodologies. Sequential analyses for the two monitoring procedures were unable to identify optimal cut points that would maximize power and control Type I error, regardless of level of item exposure or exam studied. Sequential plots showed fluctuations in performance over time which may have attributed to the difficulty in detecting compromised content. Since the two exams studied are typically administered to all examinees within a school at the same time, the varying ability of school cohorts was confounded by differences in ability across schools making accurate detection difficult. Analyses of the effect of compromised content on item performance and ability estimates indicated that as the level of item exposure increased, the degree of contamination in the ability estimates and performance of non-exposed items may have also disproportionately increased. However, differences in estimated abilities for exposed and non-exposed items were found to be useful for flagging aberrant examinees that may have benefited from preknowledge when the level of item exposure was not too high. These findings suggest that it is important to monitor performance early and often to identify exposed items so that they can be removed to reduce the impact of compromised content on ability estimates. A major limitation of the study was that we didn't know when items were truly exposed or the extent of item exposure on the exams. Items were confirmed as exposed through web monitoring or anonymous tips, but we didn't know if other items on a form were also contaminated or if the extent of a breach was limited to a small cohort of examinees or more widespread. Another limitation of the study is that the exams under study were administered by cohort. Identifying exposed items was confounded by sequential administration to school cohorts where the level of ability varied by school. There were fluctuations in performance across time and multiple change points for low and high performing schools within groups that made it difficult to identify the extent or frequency of exposure. Next steps for future research are to investigate the sequential monitoring of response times to see if monitoring response latency can better detect breached items for these exams, as well as conducting simulation studies using the two procedures that reflects the real data parameters, ability of cohorts and level of exposure. Future research should examine how well the sequential monitoring procedures perform on exams with real data that are not administered to cohorts that vary in ability and that wouldn't systematically introduce fluctuations in the abilities of the moving sample. Other promising areas of research for detecting examinees with item preknowledge include using Bayesian inference on ability estimates and outlier detection using response times (Sinharay, 2020) as well as, conducting studies exploring the usefulness and effectiveness of a new mixture Rasch-Lognormal response time model (MixRL) for examining response accuracy (Lee & Wollack 2020). This research gives insight into the sensitivity of the different methods for detection of item preknowledge and compromised content when using real data for subgroup populations. One expected outcome of this research was a list of recommendations and procedures based on data forensics and statistical analyses that can be used by operational testing programs when monitoring changes in item performance for subgroup populations to detect compromised content and identify aberrant examinee groups with item preknowledge. Although sequential monitoring procedures did not work well for the exams studied, this study does provide recommendations for practitioners and highlights the importance of accounting for operational considerations and group characteristics when implementing monitoring procedures that can help with setup and the interpretation of results. [The dissertation citations contained here are published with the permission of ProQuest LLC. Further reproduction is prohibited without permission. Copies of dissertations may be obtained by Telephone (800) 1-800-521-0600. Web page: http://www.proquest.com/en-US/products/dissertations/individuals.shtml.] (As Provided). |
| Anmerkungen | ProQuest LLC. 789 East Eisenhower Parkway, P.O. Box 1346, Ann Arbor, MI 48106. Tel: 800-521-0600; Web site: http://www.proquest.com/en-US/products/dissertations/individuals.shtml |
| Erfasst von | ERIC (Education Resources Information Center), Washington, DC |
| Update | 2024/1/01 |
Literaturbeschaffung und Bestandsnachweise in Bibliotheken prüfen
Permalink als QR-Code
Inhalt auf sozialen Plattformen teilen (nur vorhanden, wenn Javascript eingeschaltet ist)
Teile diese Seite:
