Forced-Choice Ranking Models for Raters' Ranking Data

Autor/inn/en	Hung, Su-Pin; Huang, Hung-Yu
Titel	Forced-Choice Ranking Models for Raters' Ranking Data
Quelle	In: Journal of Educational and Behavioral Statistics, 47 (2022) 5, S.603-634 (32 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Huang, Hung-Yu)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1076-9986
DOI	10.3102/10769986221104207
Schlagwörter	Evaluation Methods; Rating Scales; Item Analysis; Preferences; Responses; Evaluators; Decision Making; Models; Probability; Item Response Theory; Simulation; Sample Size; Creativity; Correlation + Suchen Sie Ihr Suchwort? Rating-Skala; Itemanalyse; Decision-making; Entscheidungsfindung; Analogiemodell; Wahrscheinlichkeitsrechnung; Wahrscheinlichkeitstheorie; Item-Response-Theorie; Simulation program; Simulationsprogramm; Kreativität; Korrelation
Abstract	To address response style or bias in rating scales, forced-choice items are often used to request that respondents rank their attitudes or preferences among a limited set of options. The rating scales used by raters to render judgments on ratees' performance also contribute to rater bias or errors; consequently, forced-choice items have recently been employed for raters to rate how a ratee performs in certain defined traits. This study develops forced-choice ranking models (FCRMs) for data analysis when performance is evaluated by external raters or experts in a forced-choice ranking format. The proposed FCRMs consider different degrees of raters' leniency/severity when modeling the selection probability in the generalized unfolding item response theory framework. They include an additional topic facet when multiple tasks are evaluated and incorporate variations in leniency parameters to capture the interactions between ratees and raters. The simulation results indicate that the parameters of the new models can be satisfactorily recovered and that better parameter recovery is associated with more item blocks, larger sample sizes, and a complete ranking design. A technological creativity assessment is presented as an empirical example with which to demonstrate the applicability and implications of the new models. (As Provided).
Anmerkungen	SAGE Publications. 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243; Tel: 805-499-9774; Fax: 800-583-2665; e-mail: journals@sagepub.com; Web site: https://sagepub.com
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01