A Measure of Agreement for Interval or Nominal Multivariate Observations by Different Sets of Judges

Autor/inn/en	Janson, Harald; Olsson, Ulf
Titel	A Measure of Agreement for Interval or Nominal Multivariate Observations by Different Sets of Judges
Quelle	In: Educational and Psychological Measurement, 64 (2004) 1, S.62-70 (9 Seiten)Infoseite zur Zeitschrift PDF als Volltext Verfügbarkeit
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	0013-1644
DOI	10.1177/0013164403260195
Schlagwörter	Interrater Reliability; Multivariate Analysis; Evaluation Methods; Measurement Techniques; Rating Scales + Suchen Sie Ihr Suchwort? Interrater-Reliabilität; Multivariate Analyse; Messtechnik; Rating-Skala
Abstract	This article addresses the problem of accounting overall multivariate chance-corrected interobserver agreement when targets have been rated by different sets of judges (not necessarily equal in number). The proposed approach builds on Janson and Olsson's multivariate generalization of Cohen's kappa but incorporates weighting for number of judges and applies an expression for expected disagreement suitable for the case with different judges. The authors suggest that the attractiveness of this approach to multivariate agreement measurement lies in the interpretability of the terms of expected and observed disagreement as average distances between observations, and that addressing agreement without regard to the covariance structure among variables has advantages in simplicity and interpretability. Correspondences to earlier approaches are noted, and application of the proposed measure is exemplified using hypothetical data sets. (Author).
Anmerkungen	Sage Publications, 2455 Teller Road, Thousand Oaks, CA 91320. Tel: 800-818-7243 (Toll Free); Fax: 800-583-2665 (Toll Free).
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2017/4/10