Content Analysis of Test Taker Responses on an AP Japanese Computer-Simulated Conversation Test: A Mixed Methods Approach for a Validity Argument

Autor/in	Suzumura, Nana
Titel	Content Analysis of Test Taker Responses on an AP Japanese Computer-Simulated Conversation Test: A Mixed Methods Approach for a Validity Argument
Quelle	In: Language Assessment Quarterly, 19 (2022) 5, S.480-502 (23 Seiten) PDF als Volltext Verfügbarkeit
Zusatzinformation	ORCID (Suzumura, Nana)
Sprache	englisch
Dokumenttyp	gedruckt; online; Zeitschriftenaufsatz
ISSN	1543-4303
DOI	10.1080/15434303.2022.2130326
Schlagwörter	Content Analysis; Test Wiseness; Advanced Placement; Computer Assisted Testing; Computer Simulation; Second Language Learning; Language Tests; Japanese; Test Validity; Item Response Theory; Information Seeking; Prompting; Evaluation Criteria; Test Construction + Suchen Sie Ihr Suchwort? Inhaltsanalyse; Computergrafik; Computersimulation; Zweitsprachenerwerb; Language test; Sprachtest; Japaner; Japanisch; Testvalidität; Item-Response-Theorie; Informationserschließung; Benutzerführung; Testaufbau
Abstract	The present study is part of a larger mixed methods project that investigated the speaking section of the Advanced Placement (AP) Japanese Language and Culture Exam. It investigated assumptions for the evaluation inference through a content analysis of test taker responses. Results of the content analysis were integrated with those of a many-facet Rasch analysis of the same speech data. This study found that most information-seeking prompts elicited a good sized ratable speech sample with relevant content, and the rating criteria seemed to fit with the nature of the interaction. Therefore, information-seeking prompts generally provided appropriate evidence of test takers' ability. In contrast, non-information-seeking prompts such as requests and expressive prompts tended to have issues with eliciting a good sized ratable speech sample with relevant content, and their response expectations realized in the rating criteria did not fit with the nature of the interaction. Thus, non-information-seeking prompts showed greater potential of becoming sources of measurement error with the current test design. This article discusses possible solutions to increase the validity of the evaluation inference. Findings from the present study would be useful for future test development of computer-based L2 tests that aim to assess interpersonal communication skills. (As Provided).
Anmerkungen	Routledge. Available from: Taylor & Francis, Ltd. 530 Walnut Street Suite 850, Philadelphia, PA 19106. Tel: 800-354-1420; Tel: 215-625-8900; Fax: 215-207-0050; Web site: http://www.tandf.co.uk/journals
Erfasst von	ERIC (Education Resources Information Center), Washington, DC
Update	2024/1/01