A Meta-Analysis of Reliability Coefficients in Second Language Research

Modern Language Journal

Volumn 100, Issue 2, 2016, Pages 538-553

  Plonsky, Luke ^a   Derrick, Deirdre J ^b  

^a UNIVERSITY OF LONDON   (United Kingdom)

^b NORTHERN ARIZONA UNIVERSITY   (United States)

Author keywords

instrumentation; meta analysis; psychometrics; reliability; second language; testing

Indexed keywords

EID: 84977476165     PISSN: 00267902     EISSN: 15404781     Source Type: Journal    
DOI: 10.1111/modl.12335     Document Type: Article

References (56)

1. Adesope, O. O., Lavin, T., Thompson, T., & Ungerleider, C. (2010). A systematic review and meta-analysis of the cognitive correlates of bilingualism. Review of Educational Research, 80, 207–245.
2. Adesope, O. O., Lavin, T., Thompson, T., & Ungerleider, C. (2011). Best practices in teaching literacy to ESL immigrant students: A meta-analysis. British Journal of Educational Psychology, 81, 629–653.
3. Bachman, L. F. (2004). Statistical analyses for language assessment. New York: Cambridge University Press.
4. Brennan, R. L. (2001). Generalizability theory. New York: Springer Verlag.
5. Brown, D. (2016). The type and linguistic foci of oral corrective feedback in the L2 classroom: A meta-analysis. Language Teaching Research, doi:10.1177/1362168814563200.
6. Brown, J. D. (2014). Classical theory reliability. In A. J. Kunnan (Ed.), The companion to language assessment (pp. 1165–1181). Oxford, UK: Wiley–Blackwell.
7. Brown, J. D., & Bailey, K. M. (2008). Language testing courses: What are they in 2007? Language Testing, 25, 349–383.
8. Chaudron, C. (2001). Progress in language classroom research: Evidence from The Modern Language Journal, 1916–2000. Modern Language Journal, 85, 57–76.
9. Cho, E., & Kim, S. (2015). Cronbach's coefficient alpha: Well known but poorly understood. Organizational Research Methods, 18, 207–230.
10. Cohen, A. D., & Macaro, E. (2013). Research methods in second language acquisition. In E. Macaro (Ed.), Continuum companion to second language acquisition (pp. 107–136). London: Continuum.
11. Coniam, D. (2009). Investigating the quality of teacher-produced tests for EFL students and the effects of training in test development principles and practices on improving test quality. System, 37, 226–242.
12. Cortina, J. M. (1994). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, 98–104.
13. Cronbach, L. J., Nageswari, R., & Gleser, G. C. (1963). Theory of generalizability: A liberation of reliability theory. The British Journal of Statistical Psychology, 16, 137–163.
14. Derrick, D. J. (2016). Instrument reporting practices in second language research. TESOL Quarterly, 50, 132–153.
15. Douglas, D. (2001). Performance consistency in second language acquisition and language testing: A conceptual gap. Second Language Research, 17, 442–456.
16. Ellis, R. (2005). Measuring implicit and explicit knowledge of a second language: A psychometric study. Studies in Second Language Acquisition, 27, 141–172.
17. Feng, G. C. (2013). Factors affecting intercoder reliability: A Monte Carlo experiment. Quality and Quantity, 47, 2959–2982.
18. Feng, G. C. (2014). Intercoder reliability indices: Disuse, misuse, and abuse. Quality and Quantity, 48, 1803–1815.
19. Gass, S. M., with Behney, J., & Plonsky, L. (2013). Second language acquisition: An introductory course (4th ed.). New York: Routledge/Taylor & Francis.
20. Green, R. (2013). Statistical analyses for language testers. New York: Palgrave Macmillan.
21. Grabowski, K., & Dakin, J. W. (2014). Test development literacy. In A. J. Kunnan (Ed.), The companion to language assessment (pp. 751–768). Oxford, UK: Wiley–Blackwell.
22. Haertel, E. H. (2006). Reliability. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 65–110). Westport, CT: American Council on Education and Praeger.
23. Hunter, F. L., & Schmidt, J. E. (2014). Methods of meta-analysis: Correcting error and bias in research findings (3rd ed.). Thousand Oaks, CA: SAGE.
24. Hughes, A. (2003). Testing for language teachers (2nd ed.). New York: Cambridge University Press.
25. Jeon, E. H., & Kaya, T. (2006). Effects of L2 instruction on interlanguage pragmatic development: A meta-analysis. In J. M. Norris, & L. Ortega (Eds.), Synthesizing research on language learning and teaching (pp. 165–211). Philadelphia: John Benjamins.
26. Jeon, E. H., & Yamashita, J. (2014). L2 reading comprehension and its correlates: A meta-analysis. Language Learning, 64, 160–212.
27. Larson–Hall, J. (2010). A guide to doing statistics in second language research using SPSS. New York: Routledge/Taylor & Francis.
28. Larson–Hall, J., & Plonsky, L. (2015). Reporting and interpreting quantitative research findings: What gets reported and recommendations for the field. Language Learning, 65, Supp. 1, 127–159.
29. Lee, J., Jang, J., & Plonsky, L. (2015). The effectiveness of second language pronunciation instruction: A meta-analysis. Applied Linguistics, 36, 345–366.
30. Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: SAGE.
31. Loewen, S., Lavolette, B., Spino, L. A., Papi, M., Schmidtke, J., Sterling, S., & Wolff, D. (2014). Statistical literacy among applied linguists and second language acquisition researchers. TESOL Quarterly, 48, 360–388.
32. Mackey, A., & Goo, J. (2007). Interaction research in SLA: A meta-analysis and research synthesis. In A. Mackey (Ed.), Conversational interaction in second language acquisition: A collection of empirical studies (pp. 407–451). New York: Oxford University Press.
33. Morgan, G. B., Zhu, M., Johnson, R. L., & Hodge, K. J. (2014). Interrater reliability estimators commonly used in scoring language assessments: A Monte Carlo investigation of estimator accuracy. Language Assessment Quarterly, 11, 303–324.
34. Muchinsky, P. M. (1996). The correction for attenuation. Educational and Psychological Measurement, 56, 3–75.
35. Nekrasova, T., & Becker, T. (2009). Effectiveness of practice: A research synthesis and quantitative meta-analysis. Unpublished manuscript.
36. Norris, J. M., & Ortega, L. (2000). Effectiveness of L2 instruction: A research synthesis and quantitative meta-analysis. Language Learning, 50, 417–528.
37. Oller, J. W., Jr., & Perkins, K. (1978). Language proficiency as a source of variance in self-reported affective variables. In J. W. Oller, Jr., & K. Perkins (Eds.), Language in education: Testing the tests (pp. 103–125). Rowley, MA: Newbury House Publishers.
38. Perkins, K., Brutten, S. R., & Gass, S. M. (1996). An investigation of patterns of discontinuous learning: Implications for ESL measurement. Language Testing, 13, 63–82.
39. Plonsky, L. (2011). The effectiveness of second language strategy instruction: A meta-analysis. Language Learning, 61, 993–1038.
40. Plonsky, L. (2013). Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition, 35, 655–687.
41. Plonsky, L., & Gass, S. (2011). Quantitative research methods, study quality, and outcomes: The case of interaction research. Language Learning, 61, 325–366.
42. Plonsky, L., & Oswald, F. L. (2014). How big is ‘big’? Interpreting effect sizes in L2 research. Language Learning, 64, 878–912.
43. Rodriguez, M. C., & Meada, Y. (2006). Meta-analysis of coefficient alpha. Psychological Methods, 11, 306–322.
44. Russell, J., & Spada, N. (2006). The effectiveness of corrective feedback for the acquisition of L2 grammar: A meta-analysis of the research. In J. M. Norris & L. Ortega(Eds.), Synthesizing research on language learning and teaching (pp. 133–164). Philadelphia: John Benjamins.
45. Shrout, P. E. (1998). Measurement reliability and agreement in psychiatry. Statistical Methods in Medical Research, 7, 201–317.
46. Subkoviak, M. J. (1988). A practitioner's guide to computation and interpretation of reliability indices for mastery. Journal of Educational Measurement, 25, 47–55.
47. Thompson, B., & Snyder, P. A. (1998). Statistical significance and reliability analyses in recent JCD research articles. Journal of Counseling and Development, 76, 436–441.
48. Vacha–Haase, T., Ness, C., Nilsson, J., & Reetz, D. (1999). Practices regarding reporting of reliability coefficients: A review of three journals. Journal of Experimental Education, 67, 335–341.
49. VanPatten, B., & Williams, J. (2002). Research criteria for tenure in second language acquisition: Results from a survey of the field. (Unpublished manuscript). University of Illinois, Chicago, IL.
50. Vogt, K., & Tsagaro, D. (2014). Assessment literacy of foreign language teachers: Findings of a European study. Language Assessment Quarterly, 11, 374–402.
51. Watanabe, Y., & Koyama, D. (2008). A meta-analysis of second language cloze testing research. Second Language Studies, 26, 103–133.
52. Webb, N. M., Shavelson, R. J., & Haertel, E. H. (2006). Reliability coefficients and Generalizability Theory. Handbook of Statistics, 26, 81–124.
53. Weir, C. (2005). Language testing and validation. Basingstoke, UK: Palgrave Macmillan.
54. Wheeler, D. L., Vassar, M., Worley, J. A., & Barnes, L. L. B. (2011). A reliability generalization meta-analysis of coefficient alpha for the Maslach Burnout Inventory. Educational and Psychological Measurement, 71, 231–244.
55. Willson, V. L. (1980). Research techniques in AERJ articles: 1969 to 1978. Educational Researcher, 9, 5–10.
56. Ziegler, N. (2013). Synchronous computer-mediated communication and interaction: A research synthesis and meta-analysis. (Unpublished doctoral dissertation). Georgetown University, Washington, DC.