Utility-Value Score: A Case Study in System Generalization for Writing Analytics

Autor/inn/en	Beigman Klebanov, Beata; Priniski, Stacy; Burstein, Jill; Gyawali, Binod; Harackiewicz, Judith; Thoman, Dustin
Titel	Utility-Value Score: A Case Study in System Generalization for Writing Analytics
Quelle	2 (2018), S.314-328 (15 Seiten)Infoseite zur Zeitschrift PDF als Volltext kostenfreie Datei Verfügbarkeit
Zusatzinformation	Weitere Informationen
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
Schlagwörter	Learning Analytics; Context Effect; Automation; Generalization; Value Judgment; College Freshmen; Essays; Writing Evaluation; STEM Education; Relevance (Education); Content Area Writing; Introductory Courses; Psychology; Biology; Student Motivation; Natural Language Processing; Data Collection; California (Long Beach); Wisconsin (Madison) + Suchen Sie Ihr Suchwort? Werturteil; Studienanfänger; Essay; Aufsatzunterricht; STEM; Relevance; Relevanz; Schriftliche Übung; Einführungskurs; Psychologie; Biologie; Schulische Motivation; Natürliche Sprache; Data capture; Datensammlung
Abstract	Collection and analysis of students' writing samples on a large scale is a part of the research agenda of the emerging writing analytics community that promises to deliver an unprecedented insight into characteristics of student writing. Yet with a large scale often comes variability of contexts in which the samples were produced--different institutions, different purposes of writing, different author demographics, to name just a few possible dimensions of variation. What are the implications of such variation for the ability of automated methods to create indices/features based on the writing samples that would be valid and meaningful? This paper presents a case study in system generalization. Building on a system developed to assess the expression of "utility value" (a social-psychology-based construct) in essays written by first-year biology students at one postsecondary institution, we vary data parameters and observe system performance. From the point of view of social psychology, all these variants represent the same underlying construct (i.e., utility value), and it is thus very tempting to think that an automatically produced "utility-value" score could provide a meaningful analytic, consistently, on a large collection of essays. However, findings from this research show that there are challenges: Some variations are easier to deal with than others, and some components of the automated system generalize better than others. The findings are then discussed both in the context of the case study and more generally. (As Provided).
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01