Applying Computerized-Scoring Models of Written Biological Explanations across Courses and Colleges: Prospects and Limitations

Autor/inn/en	Ha, Minsu; Nehm, Ross H.; Urban-Lurain, Mark; Merrill, John E.
Titel	Applying Computerized-Scoring Models of Written Biological Explanations across Courses and Colleges: Prospects and Limitations
Quelle	In: CBE - Life Sciences Education, 10 (2011) 4, S.379-393 (15 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1931-7913
DOI	10.1187/cbe.11-08-0081
Schlagwörter	Expertise; Nonmajors; Majors (Students); Computer Software; Computers; Scoring; Biology; Feedback (Response); Universities; Comparative Analysis; Writing (Composition); Evolution; Introductory Courses; Questioning Techniques; Ohio + Suchen Sie Ihr Suchwort? Expert appraisal; Digitalrechner; Bewertung; Biologie; University; Universität; Schreibübung; Einführungskurs; Befragungstechnik; Fragetechnik
Abstract	Our study explored the prospects and limitations of using machine-learning software to score introductory biology students' written explanations of evolutionary change. We investigated three research questions: 1) Do scoring models built using student responses at one university function effectively at another university? 2) How many human-scored student responses are needed to build scoring models suitable for cross-institutional application? 3) What factors limit computer-scoring efficacy, and how can these factors be mitigated? To answer these questions, two biology experts scored a corpus of 2556 short-answer explanations (from biology majors and nonmajors) at two universities for the presence or absence of five key concepts of evolution. Human- and computer-generated scores were compared using kappa agreement statistics. We found that machine-learning software was capable in most cases of accurately evaluating the degree of scientific sophistication in undergraduate majors' and nonmajors' written explanations of evolutionary change. In cases in which the software did not perform at the benchmark of "near-perfect" agreement (kappa greater than 0.80), we located the causes of poor performance and identified a series of strategies for their mitigation. Machine-learning software holds promise as an assessment tool for use in undergraduate biology education, but like most assessment tools, it is also characterized by limitations. (Contains 4 tables and 4 figures.) (As Provided).
Anmerkungen	American Society for Cell Biology. 8120 Woodmont Avenue Suite 750, Bethesda, MD 20814-2762. Tel: 301-347-9300; Fax: 301-347-9310; e-mail: ascbinfo@ascb.org; Website: http://www.ascb.org
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2017/4/10