Investigating rater severity/leniency in interpreter performance testing: A multifaceted Rasch measurement approach

Interpreting

Volumn 17, Issue 2, 2015, Pages 255-283

  Han, Chao ^a  

^a MACQUARIE UNIVERSITY   (Australia)

Author keywords

Interpreter certification; Multifaceted Rasch measurement; Performance testing; Rater severity leniency; Rater training; Rater variability

Indexed keywords

EID: 84941013696     PISSN: 13846647     EISSN: 1569982X     Source Type: Journal    
DOI: 10.1075/intp.17.2.05han     Document Type: Conference Paper

References (68)

1. Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika 43 (4), 561-573. DOI: 10.1007/BF02293814
2. Angelelli, C. (2009). Using a rubric to assess translation ability: Defining the construct. In C. Angelelli & H. E. Jacobson (Eds.), Testing and assessment in translation and interpreting studies. Amsterdam: John Benjamins, 13-47. DOI: 10.1075/ata.xiv.03ang
3. Arjona-Tseng, E. (1993). A psychometric approach to the selection of translation and interpreting students in Taiwan. Perspectives 1 (1), 91-104. DOI: 10.1080/0907676X.1993.9961203
4. Arocha, I. S. & Joyce, L. (2013). Patient safety, professionalization, and reimbursement as primary drivers for National Medical Interpreter Certification in the United States. Translation & Interpreting 5 (1), 127-142. DOI: ti.105201.2013.a07
5. Bond, T. G. & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). London: Lawrence Erlbaum.
6. Bonk, W. J. & Ockey, G. J. (2003). A many-facet Rasch analysis of the second language group oral discussion task. Language Testing 20 (1), 89-110. DOI: 10.1191/0265532203lt245oa
7. Brown, A. (1995). The effect of rater variables in the development of an occupation-specific language performance test. Language Testing 12 (1), 1-15. DOI: 10.1177/026553229501200101
8. Campbell, S. & Hale, S. (2003). Translation and interpreting assessment in the context of educational measurement. In G. Anderman & M. Rogers (Eds.), Translation today: Trends and perspectives. Clevedon: Multilingual Matters, 205-224.
9. Certification Commission for Healthcare Interpreters (2010). Job task analysis study and results. http://www.cchicertification.org/images/webinars/cchi%20jta%20report-public.pdf (accessed 22 May 2015).
10. Certification Commission for Healthcare Interpreters (2011). Technical Report on the Development and Pilot Testing of the CCHI Examinations. http://www.cchicertification.org/images/pdfs/cchi%20technical%20report%20-%20public%20final.pdf (accessed 22 May 2015).
11. Certification Commission for Healthcare Interpreters (2012). Technical Report on the Development and Pilot Testing of the Certified Healthcare Interpreter™ (CHI™) Examination for Arabic and Mandarin. http://www.cchicertification.org/images/pdfs/cchi%20arabic% 20and%20mandarin%20technical%20report-final.pdf (accessed 22 May 2015).
12. Certification Commission for Healthcare Interpreters (2014). Candidate's Examination Handbook. http://www.cchicertification.org/images/pdfs/candidatehandbook.pdf (accessed 22 May 2015).
13. Clifford, A. (2005). Putting the exam to the test: Psychometric validation and interpreter certification. Interpreting 7 (1), 97-13. DOI: 10.1075/intp.7.1.06cli
14. Eckes, T. (2005). Examining rater effects in TestDaF writing and speaking performance assessments: A many-facet Rasch analysis. Language Assessment Quarterly 2 (3), 197-221. DOI: 10.1207/s15434311laq0203-2
15. Eckes, T. (2008). Rater types in writing performance assessments: A classification approach to rater variability. Language Testing 25 (2), 155-185. DOI: 10.1177/0265532207086780
16. Eckes, T. (2011). Introduction to many-facet Rasch measurement: Analyzing and evaluating ratermediated assessments. Frankfurt am Main: Peter Lang.
17. Elder, C., Barkhuizen, G., Knoch, U. & von Randow, J. (2007). Evaluating rater responses to an online training program for L2 writing assessment. Language Testing 24 (1), 37-64. DOI: 10.1177/0265532207071511
18. Feng, J. Z. (2005).[Towards the standardization of interpretation testing]. 89, 54-58.
19. Feuerle, L. (2013). Testing interpreters: Developing, administering, and scoring court interpreter certification exams. Translation & Interpreting 5 (1), 80-93. DOI: 10.12807/ti.105201.2013.a04
20. Fox, C. & Jones, J. (1998). Uses of Rasch modeling in counseling psychology research. Journal of Counseling Psychology 45 (1), 30-45. DOI: 10.1037/0022-0167.45.1.30
21. Gile, D. (1995). Basic concepts and models for interpreter and translator training. Amsterdam: John Benjamins. DOI: 10.7202/002162ar
22. Green, R. (2013). Statistical analysis for language testers. Basingstoke: Palgrave Macmillan. DOI: 10.1057/9781137018298
23. Hale, S., Garcia, I., Hlavac, J., Kim, M., Lai, M., Turner, B. & Slatyer, H. (2012). Development of a conceptual overview for a new model for NAATI standards, testing and assessment. Sydney, Australia. http://www.naati.com.au/PDF/INT/INTFinalReport.pdf (accessed 22 May 2015).
24. Han, C. & Mehdi, R. (2015). The effects of speech rate and accent on interpreter performance quality: A mixed-methods replication study. Manuscript submitted for publication.
25. Henning, G. (1992). Dimensionality and construct validity of language tests. Language Testing 9 (1), 1-11. DOI: 10.1177/026553229200900102
26. Hlavac, J. (2013). A cross-national overview of translator and interpreter certification procedures. Translation & Interpreting 5, 32-65. DOI: 10.12807/ti.105201.2013.a02
27. IoL Educational Trust. (2010). Diploma in Public Service Interpreting: Handbook for candidates. London, UK. http://www.iol.org.uk/qualifications/DPSI/Handbook/DPSIHB11.pdf (accessed 22 May 2015).
28. Jacobs, E. A., Lauderdale, D. S., Meltzer, D., Shorey, J. M., Levinson, W. & Thisted, R. A. (2001). Impact of interpreter services on delivery of health care to limited-English-proficient patients. Journal of General Internal Medicine 16 (7), 468-474. DOI: 10.1046/j.1525-1497.2001.016007468.x
29. Knoch, U. (2011). Investigating the effectiveness of individualized feedback to rating behavior - a longitudinal study. Language Testing 28 (2), 179-200. DOI: 10.1177/0265532210384252
30. Kondo-Brown, K. (2002). A FACETS analysis of rater bias in measuring Japanese second language writing performance. Language Testing 19 (1), 3-31. DOI: 10.1191/0265532202lt218oa
31. Linacre, J. M. (2002). What do infit and outfit, mean-square and standardized mean? Rasch Measurement Transactions 16 (2), 878.
32. Linacre, J. M. (2013). A user's guide to FACETS: Program manual 3.71.2. http://www.winsteps.com/a/facets-manual.pdf (accessed 22 May 2015).
33. Liu, M. (2013). Design and analysis of Taiwan's interpretation certification examination. In D. Tsagari & R. van Deemter (Eds.), Assessment issues in language translation and interpreting. Frankfurt: Peter Lang, 163-178.
34. Lu, M., Liu, C. & Gong, X. F. (2007). [How to maintain consistency in CATTI's interpretation tests: A research report]., 5, 57-61.
35. Lumley, T. & McNamara, T. F. (1995). Rater characteristics and rater bias: implications for training. Language Testing 12 (1), 54-71. DOI: 10.1177/026553229501200104
36. Lunz, M. E. & Stahl, J. A. (1990). Judge consistency and severity across grading periods. Evaluation and the Health Professions 13 (4), 425-444. DOI: 10.1177/016327879001300405
37. Lynch, B. K. & McNamara, T. F. (1998). Using G-theory and many-facet Rasch measurement in the development of performance assessments of the ESL speaking skills of immigrants. Language Testing 15 (2), 158-180.
38. Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika 47 (2), 149-174. DOI: 10.1007/BF02296272
39. McNamara, T. F. (1996). Measuring second language performance. London: Longman.
40. McNamara, T. F. & Knoch, U. (2012). The Rasch wars: The emergence of Rasch measurement in language testing. Language Testing 29 (4) 555-576. DOI: 10.1177/0265532211430367
41. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed.). New York: American Council on Education and Macmillan, 13-103.
42. Mortensen, D. (1998). Establishing a scheme for interpreter certification: The Norwegian experience. http://folk.uio.no/dianem/report1998.pdf (accessed 22 May 2015).
43. Mortensen, D. (2001). Measuring quality in interpreting: A report on the Norwegian Interpreter Certification Examination (NICE). Oslo, Norway. http://folk.uio.no/dianem/IntQuality- Internet.pdf (accessed 22 May 2015).
44. Myford, C. M. & Wolfe, E. W. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement 4 (4), 386-422.
45. National Accreditation Authority for Translators and Interpreters. (2013). INT Project discussion paper. http://www.naati.com.au/PDF/INT/INT%20Project%20Discussion%20Paper%20-%20November%202013.pdf (accessed 22 May 2015).
46. National Association of the Deaf (2014). NAD and RID releases the NAD-RID National Interpreter Certification (NIC) Credential Validity, Reliability, & Candidate Performance Report. http://nad.org/news/2014/2/nad-and-rid-release-performance-report (accessed 22 May 2015).
47. National Board of Certification for Medical Interpreters (2014). The National Board of Certification for Medical Interpreters: Certified Medical Interpreter candidate handbook. http://www.certifiedmedicalinterpreters.org//sites/default/files/national-board-candidatehandbook.pdf (accessed 22 May 2015).
48. National Center for States Courts (2013). Federal Court Interpreter Certification Examination for Spanish/English: Examinee handbook. http://www.ncsc.org/sitecore/content/microsites/fcice/home/About-the-program/-/media/Microsites/Files/FCICE/Final%20Examinee%20Handbook%201-23-2013%20for%20online.ashx (accessed 22 May 2015).
49. Office of China Accreditation Tests for Translators and Interpreters. (2005). [Syllabus of CATTI Level-two Simultaneous Interpreting Test]. Beijing: Foreign Languages Press.
50. PSI Services LLC (2010). Development and validation of oral and written examinations for medical interpreter certification: Technical report. Burbank, California, USA. http://www.certifiedmedicalinterpreters.org//sites/default/files/oral-and-written-medical-interpretertechnical- report-final.pdf (accessed 22 May 2015).
51. PSI Services LLC (2013). Development and validation of oral examinations for Medical Interpreter Certification: Mandarin, Russian, Cantonese, Korean, and Vietnamese forms. http://www.certifiedmedicalinterpreters.org/sites/default/files/tech-report-development-validationlanguage- forms.pdf (accessed 22 May 2015).
52. Roat, C. E. (2006). Certification of health care interpreters in the United States: A primer, a status report and considerations for national certification. Los Angeles, CA. http://www.calendow.org/uploadedFiles/certification-of-health-care-interpretors.pdf (accessed 22 May 2015).
53. Russell, D. & Malcolm, K. (2009). Assessing ASL-English interpreters: The Canadian model of national certification. In C. V. Angelelli & H. E. Jacobson (Eds.), Testing and assessment in translation and interpreting studies: A call for dialogue between research and practice. Amsterdam: John Benjamins, 331-376. DOI: 10.1075/ata.xiv.15rus
54. Schaefer, E. (2008). Rater bias pattern in an EFL writing assessment. Language Testing 25 (4), 465-493. DOI: 10.1177/0265532208094273
55. Schumacker, R. E. (1999). Many-facet Rasch analysis with crossed, nested and mixed designs. Journal of Outcome Measurement 3 (4), 323-338.
56. South African Translators' Institute (2007a). Guidelines: SASL interpreter accreditation testing. http://translators.org.za/sati-cms/downloads/dynamic/sati-accreditation-for-sasl-interpreting- english.pdf (accessed 22 May 2015).
57. South African Translators' Institute (2007b). Guidelines: Simultaneous interpreter accreditation testing. http://translators.org.za/sati-cms/downloads/dynamic/sati-accreditation-for- sim-interpreting-english.pdf (accessed 22 May 2015).
58. Stansfield, C.W. & Hewitt, W. (2005). Examining the predictive validity of cut scores on a screening test for court interpreters. Language Testing 22 (2), 1-25. DOI: 10.1191/0265532205lt304oa
59. Sudweeks, R., Reeve, S. & Bradshaw, W. S. (2005). A comparison of generalizability theory and many-facet Rasch measurement in an analysis of college sophomore writing. Assessing Writing 9, 239-261. DOI: 10.1016/j.asw.2004.11.001
60. Turner, B., Lai, M. & Huang, N. (2010). Error deduction and descriptors - a comparison of two methods of translation test assessment. Translation & Interpreting 2 (1), 11-23.
61. Upshur, J. A. & Turner, C. E. (1999). Systematic effects in the rating of second-language speaking ability: test method and leaner discourse. Language Testing 16 (1), 82-111.
62. Vermeiren, H., Gucht, J. V. & De Bontridder, L. (2009). Standards as critical success factors in assessments: Certifying social interpreters in Flanders, Belgium. In C. V. Angelelli & H. E. Jacobson (Eds.), Testing and assessment in translation and interpreting studies: A call for dialogue between research and practice. Amsterdam: John Benjamins, 291-330. DOI: 10.1075/ata.xiv.14ver
63. Weigle, S. C. (1994). Effects of training on raters of ESL compositions. Language Testing 11(2), 197-223. DOI: 10.1177/026553229401100206
64. Weigle, S. C. (1998). Using FACETS to model rater training effects. Language Testing 15(2), 263-287. DOI: 10.1177/026553229801500205
65. Wigglesworth, G. (1993). Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction. Language Testing 10 (3), 305-319. DOI: 10.1177/026553229301000306
66. Wu, S. C. (2010). Assessing simultaneous interpreting: A study on test reliability and examiners' assessment behavior. PhD thesis, Newcastle University.
67. Youdelman, M. (2013). The development of certification for healthcare interpreters in the United States. Translation & Interpreting 5 (1), 114-126. DOI: ti.105201.2013.a06
68. Yu, D. R. (2005). T&I labor market in China. Sydney, Australia. http://www.ling.mq.edu.au/translation/lmtip-china.htm (accessed 22 May 2015).