Assessing Sources of Score Variability in a Multi-Site Medical Performance Assessment: An Application of Hierarchical Linear Modeling.

Autor/inn/en	Floreck, Lisa M.; De Champlain, Andre F.; Kaplan, David
Titel	Assessing Sources of Score Variability in a Multi-Site Medical Performance Assessment: An Application of Hierarchical Linear Modeling.
Quelle	(2001), (24 Seiten) PDF als Volltext kostenfreie Datei Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Monographie
Schlagwörter	Clinical Experience; Computer Software; Licensing Examinations (Professions); Patients; Performance Based Assessment; Physicians; Reliability; Scores; Skill Development + Suchen Sie Ihr Suchwort? Patient; Leistungsermittlung; Physician; Doctor; Arzt; Reliabilität; Kompetenzentwicklung; Qualifikationsentwicklung
Abstract	The purpose of the current study was to use multilevel modeling to quantify and explain the sources of score variation in standardized patient (SP) encounters. Through laypersons trained to portray SPs and record medical student actions, SP examinations allow the measurement of examinees' clinical and interpersonal skills. In this study, the SP test assesses the clinical skills of physicians about to enter supervised practice. Four cases were drawn from the SP bank. The number of examinees who saw each of these cases ranged from 357 to 565 with this number being reduced in the checklist (objectively scored) models to those who had already taken step 2 of the U.S. Medical Licensing Examination. The multilevel modeling software package HLM5 was used to estimate the proportion of score variation between SPs and training sites, assess the relationship between the skill scores and SP characteristics, and quantify the proportion of variation explained when SP characteristics are added into the model. Results of previous generalizability analyses have demonstrated that there is little variation in scores across SPs and sites, and that most of the variation in scores is due to case specificity. The findings from this study show that although SP variability was negligible for checklist scores, variability was quite large for interpersonal scores. Although a major advantage of using multilevel modeling is to explain variation at various levels, the variables used in this study were not helpful in explaining the variation between SPs. Overall, however, this study should be seen as an important first step in using multilevel modeling to explore the variability of SP examinations. Careful consideration of SP and site characteristics should be captured and analyzed statistically so that steps can be taken to implement fair and reliable examinations. (Contains 4 tables and 17 references.) (SLD)
Erfasst von	ERIC (Education Resources Information Center), Washington, DC