Study quality in quantitative l2 research (1990-2010): A methodological synthesis and call for reform

Modern Language Journal

Volumn 98, Issue 1, 2014, Pages 450-470

  Plonsky, Luke ^a  

^a NORTHERN ARIZONA UNIVERSITY   (United States)

Author keywords

Meta analysis; Quantitative research methods; Research synthesis; Statistical reform; Study quality

Indexed keywords

EID: 84894107678     PISSN: 00267902     EISSN: 15404781     Source Type: Journal    
DOI: 10.1111/j.1540-4781.2014.12058.x     Document Type: Article

References (87)

1. Altman, M. (2004). Statistical significance, path dependency, and the culture of journal publication. Journal of Socio-Economics, 33, 651-663.
2. Bigelow, M., & Watson, J. (2012). The role of educational level, literacy, and orality in L2 learning. In S. M. Gass & A. Mackey (Eds.), The Routledge handbook of second language acquisition (pp. 461-475). New York: Routledge.
3. Brown, J. D. (1990). The use of multiple t tests in language research. TESOL Quarterly, 24, 770-773.
4. Byrnes, H. (2013). Editor's introduction. Modern Language Journal, 97, 105-108.
5. Chan, A.-W., Hróbjartsson, A., Haahr, M. T., Gøtzsche, P. C., & Altman, D. G. (2004). Empirical evidence for selective reporting of outcomes in randomized trials. Journal of the American Medical Association, 291, 2457-2465.
6. Chapelle, C. A., & Duff, P. A. (2003). Some guidelines for conducting quantitative and qualitative research in TESOL. TESOL Quarterly, 37, 157-178.
7. Chaudron, C. (2001). Progress in language classroom research: Evidence from The Modern Language Journal, 1916-2000. Modern Language Journal, 85, 57-76.
8. Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65, 145-153.
9. Cohen, J. (1968). Multiple regression as a general data-analytic system. Psychological Bulletin, 70, 426-443.
10. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
11. Cook, T. D., & Payne, M. R. (2002). Objecting to the objections to using random assignment in educational research. In F. Mosteller & R. Boruch (Eds.), Evidence matters: Randomized trials in educational research (pp. 150-178). Washington, DC: Brookings Institute.
12. Cronbach, L. J. (1957). The two disciplines of scientific psychology. American Psychologist, 12, 671-684.
13. Crookes, G. (1991). Power, effect size, and second language research. Another researcher comments. TESOL Quarterly, 25, 762-765.
14. Cumming, G., Fidler, F., Kalinowski, P., & Lai, J. (2012). The statistical recommendations of the American Psychological Association Publication Manual: Effect sizes, confidence intervals, and meta-analysis. Australian Journal of Psychology, 64, 138-146.
15. DeKeyser, R. (2009, October). Variable interaction in SLA: Much more than a nuisance. Plenary address given at the Second Language Research Forum, East Lansing, MI.
16. Ellis, N. C. (2000). Editorial statement. Language Learning, 50, xi-xiii.
17. Gass, S. (2009). A survey of SLA research. In W. C. Ritchie & T. K. Bhatia (Eds.), Handbook of second language acquisition (pp. 3-28). Bingley, UK: Emerald.
18. Gelman, A., & Weakliem, D. (2009). Of beauty, sex, and power: Too little attention has been paid to the statistical challenges in estimating small effects. American Scientist, 97, 310-316.
19. Goo, J. (2012). Corrective feedback and working memory capacity in interaction-driven L2 learning. Studies in Second Language Acquisition, 34, 445-474.
20. Goodwin, L. D., & Goodwin, W. L. (1985). An analysis of statistical techniques used in the Journal of Educational Psychology, 1979-1983. Educational Psychologist, 20, 13-21.
21. Hatch, E., & Lazaraton, A. (1991). The research manual: Design and statistics for applied linguistics. Boston, MA: Heinle & Heinle.
22. Hauser, E. (2001). The statistical power of second language acquisition research: A review. Paper presented at the Pacific Second Language Research Forum, University of Hawai'i at Manoa, Honolulu, HI.
23. Henning, G. (1986). Quantitative methods in language acquisition research. TESOL Quarterly, 20, 701-708.
24. Journal Article Reporting Standards Working Group. (2008). Reporting standards for research in psychology: Why do we need them? What might they be? American Psychologist, 63, 839-851.
25. Keck, C., Iberri-Shea, G., Tracy-Ventura, N., & Wa-Mbaleka, S. (2006). Investigating the empirical link between task-based interaction and acquisition: A meta-analysis. In J. M. Norris & L. Ortega (Eds.), Synthesizing research on language learning and teaching (pp. 91-131). Philadelphia/Amsterdam: John Benjamins.
26. Kieffer, K. M., Reese, R. J., & Thompson, B. (2001). Statistical techniques employed in AERJ and JCP articles from 1988 to 1997: A methodological review. Journal of Experimental Education, 69, 280-309.
27. Language Learning journal overview. (n.d.). In Online Wiley Library. Accessed at http://onlinelibrary.wiley.com/journal/10.1111/(ISSN)1467-9922/homepage/ProductInformation.html
28. Larson-Hall, J. (2010). A guide to doing statistics in second language research using SPSS. New York: Routledge.
29. Lazaraton, A. (1991). Power, effect size, and second language research. A researcher comments. TESOL Quarterly, 25, 759-762.
30. Lazaraton, A. (2005). Quantitative research methods. In E. Hinkel (Ed.), Handbook of research in second language teaching and learning (pp. 209-224). Mahwah, NJ: Lawrence Erlbaum.
31. Lazaraton, A., Riggenbach, H., & Ediger, A. (1987). Forming a discipline: Applied linguists' literacy in research methodology and statistics. TESOL Quarterly, 21, 263-277.
32. Lent, R. H., Aurbach, H. A., & Levin, L. S. (1971). Research design and validity assessment. Personnel Psychology, 24, 247-274.
33. Li, S. (2013). The interactions between the effects of implicit and explicit feedback and individual differences in language analytic ability and working memory. Modern Language Journal, 97, 634-654.
34. Loewen, S. (2005). Incidental focus on form and second language learning. Studies in Second Language Acquisition, 27, 361-386.
35. Loewen, S., Lavolette, E., Spino, L., Papi, M., Schmidtke, J., Sterling, S., & Wolff, D. (2014). A discipline formed?: An update on applied linguists' statistical literacy. TESOL Quarterly. http://onlinelibrary.wiley.com/doi/10.1002/tesq.128/abstract
36. Long, M. H. (1996). The role of the linguistic environment in second language acquisition. In W. C. Ritchie & T. K. Bhatia (Eds.), Handbook of research on language acquisition: Vol. 2, Second language acquisition (pp. 413-468). San Diego, CA: Academic Press.
37. Lyster, R., & Izquierdo, J. (2009). Prompts versus recasts in dyadic interaction. Language Learning, 59, 453-498.
38. Mackey, A. (2007). Introduction: The role of conversational interaction in second language acquisition. In A. Mackey (Ed.), Conversational interaction in second language acquisition: A collection of empirical studies (pp. 1-26). Oxford: Oxford University Press.
39. Mackey, A., & Gass, S. M. (2006). Pushing the methodological boundaries in interaction research: An introduction to the special issue. Studies in Second Language Acquisition, 28, 169-178.
40. Mackey, A., & Gass, S. (Eds.). (2012). Research methods in second language acquisition: A practical guide. London: Basil Blackwell.
41. Mackey, A., & Goo, J. (2007). Interaction research in SLA: A meta-analysis and research synthesis. In A. Mackey (Ed.), Conversational interaction in second language acquisition: A collection of empirical studies (pp. 407-449). Oxford: Oxford University Press.
42. Mackey, A., & Polio, C. (Eds.). (2009). Multiple perspectives on interaction: Second language research in honor of Susan M. Gass. New York: Routledge.
43. Mackey, A., & Sachs, R. (2012). Older learners in SLA research: A first look at working memory, feedback, and L2 development. Language Learning, 62, 704-740.
44. Magnan, S. S. (1994). From the editor: The MLJ tradition and the challenges ahead. Modern Language Journal, 78, 7-9.
45. Meara, P. (1995). [Book review of Statistical techniques for the study of language and language behavior, by T. Rietveld & R. van Hout]. Language Learning, 45, 341-343.
46. Meier, S. T., & Davis, S. R. (1990). Trends in reporting psychometric properties of scales used in counseling psychology research. Journal of Counseling Psychology, 37, 113-115.
47. Nassaji, H. (2012). Significance tests and generalizability of research results: A case for replication. In G. Porte (Ed.), Replication research in applied linguistics (pp. 92-115). Cambridge: Cambridge University Press.
48. Nekrasova, T., & Becker, T. (2009). Effectiveness of practice: A research synthesis and quantitative meta-analysis. Unpublished manuscript.
49. New York Times, Business Day. (2001). Accessed at http://www.nytimes.com/2001/03/28/business/greenspan-calls-for-better-data-collection.html
50. Norris, J. M., & Ortega, L. (2000). Effectiveness of L2 instruction: A research synthesis and quantitative meta-analysis. Language Learning, 50, 417-528.
51. Norris, J. M., & Ortega, L. (2006). The value and practice of research synthesis for language learning and teaching. In J. M. Norris & L. Ortega (Eds.), Synthesizing research on language learning and teaching (pp. 3-50). Philadelphia/Amsterdam: John Benjamins.
52. Nunan, D. (1991). Methods in second language classroom-oriented research: A critical review. Studies in Second Language Acquisition, 13, 249-274.
53. Oswald, F. L., & Plonsky, L. (2010). Meta-analysis in second language research: Choices and challenges. Annual Review of Applied Linguistics, 30, 85-110.
54. Peng, C.-Y. J., Chen, L.-T., Chiang, H.-M., & Chiang, Y.-C. (2013). The impact of APA and AERA guidelines on effect size reporting. Educational Psychology Review, 25, 157-209.
55. Plonsky, L. (2011a). The effectiveness of second language strategy instruction: A meta-analysis. Language Learning, 61, 993-1038.
56. Plonsky, L. (2011b). Study quality in SLA: A cumulative and developmental assessment of designs, analyses, reporting practices, and outcomes in quantitative L2 research (Unpublished doctoral dissertation). Michigan State University, East Lansing, MI.
57. Plonsky, L. (2012). Replication, meta-analysis, and generalizability. In G. Porte (Ed.), Replication research in applied linguistics (pp. 116-132). Cambridge: Cambridge University Press.
58. Plonsky, L. (2013). Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition, 35, 655-687.
59. Plonsky, L., & Gass, S. (2011). Quantitative research methods, study quality, and outcomes: The case of interaction research. Language Learning, 61, 325-366.
60. Plonsky, L., Egbert, J., & LaFlair, G. T. (2015). Bootstrapping in applied linguistics: Assessing its potential using shared data. Applied Linguistics.
61. Plonsky, L., & Oswald, F. L. (2012). How to do a meta-analysis. In A. Mackey & S. Gass (Eds.), Research methods in second language acquisition: A practical guide (pp. 275-295). London: Basil Blackwell.
62. Plonsky, L., & Oswald, F. L. (2014). How big is 'big'? Interpreting effect sizes in L2 research. Manuscript under review.
63. Polio, C. (2012). Replication in published applied linguistics research: An historical perspective. In G. Porte (Ed.), Replication research in applied linguistics (pp. 47-91). Cambridge: Cambridge University Press.
64. Porte, G. (2010). Appraising research in second language learning: A practical approach to critical analysis of quantitative research (2nd ed.). Philadelphia/Amsterdam: John Benjamins.
65. Porte, G. (Ed.). (2012). Replication research in applied linguistics. Cambridge: Cambridge University Press.
66. Psychonomic Society. (2012). New Statistical Guidelines for Journals of the Psychonomic Society. Accessed 22 October 2013 at http://www.springer.com/?SGWID=0-102-2-1390050-preview&dynamic=true
67. Publication manual of the American Psychological Association (6th ed.). (2010). Washington, DC: American Psychological Association.
68. Russell, J., & Spada, N. (2006). The effectiveness of corrective feedback for the acquisition of L2 grammar: A meta-analysis of the research. In J. M. Norris & L. Ortega (Eds.), Synthesizing research on language learning and teaching (pp. 133-164). Philadelphia/Amsterdam: John Benjamins.
69. Salgado, J. F. (1998). Sample size in validity studies of personnel selection. Journal of Occupational and Organizational Psychology, 71, 161-164.
70. Sheen, Y. (2007). The effects of corrective feedback, language aptitude, and learner attributes on the acquisition of English articles. In A. Mackey (Ed.), Conversational interaction in second language acquisition: A collection of empirical studies (pp. 301-322). Oxford: Oxford University Press.
71. Shadish, W. R., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.
72. Skidmore, S. T., & Thompson, B. (2010). Statistical techniques used in published articles: A historical review of reviews. Educational and Psychological Measurement, 70, 777-795.
73. Stafford, C. A. (2013). What's on your mind? How private speech mediates cognition during initial non-primary language learning. Applied Linguistics, 34, 151-172.
74. Studies in Second Language Acquisition journal home. (n.d.). In Cambridge Journals Online. Accessed at http://journals.cambridge.org/action/displayJournal?jid=SLA
75. Statistics Quotes. (n.d.). Accessed at http://www.math.wpi.edu/Course_Materials/SAS/quotes.html
76. Tarone, E. (2013). Applied linguistics without borders. Language Teaching, 46, 355-364.
77. Teleni, V., & Baldauf, R. B. (1989). Statistical techniques used in three applied linguistics journals: Language Learning, Applied Linguistics, and TESOL Quarterly, 1980-1986: Implications for readers and researchers. Unpublished research report. (ERIC Document Reproduction Service No. ED312905).
78. Thompson, B., & Snyder, P. A. (1998). Statistical significance and reliability analyses in recent JCD research articles. Journal of Counseling and Development, 76, 436-441.
79. Trikalinos, T. A., Churchill, R., Ferri, M., Leucht, S., Tuunainen, A., Wahlbeck, K., et al. (2004). Effect sizes in cumulative meta-analyses of mental health randomized trials evolved over time. Journal of Clinical Epidemiology, 57, 1124-1130.
80. Vacha-Haase, T., Ness, C., Nilsson, J., & Reetz, D. (1999). Practices regarding reporting of reliability coefficients: A review of three journals. Journal of Experimental Education, 67, 335-341.
81. Valentine, J. C., & Cooper, H. (2008). A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: The study design and implementation assessment device (Study DIAD). Psychological Methods, 13, 130-149.
82. VanPatten, B., & Williams, J. (2002). Research criteria for tenure in second language acquisition: Results from a survey of the field. Unpublished manuscript. University of Illinois at Chicago, Chicago, IL.
83. Wa-Mbaleka, S. (2006). A meta-analysis investigating the effects of reading on second language vocabulary learning. (Unpublished doctoral dissertation). Northern Arizona University, Flagstaff, AZ.
84. Wells, K., & Littell, J. H. (2009). Study quality assessment in systematic reviews of research on intervention effects. Research on Social Work Practice, 19, 52-62.
85. Wilkinson, L., & Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.
86. Willson, V. L. (1980). Research techniques in AERJ articles: 1969 to 1978. Educational Researcher, 9, 5-10.
87. Yilmaz, Y. (2013). The role of working memory capacity and language analytic ability in the effectiveness of explicit correction and recasts. Applied Linguistics, 34, 344-368.