Reporting and Interpreting Quantitative Research Findings: What Gets Reported and Recommendations for the Field

Language Learning

Volumn 65, Issue S1, 2015, Pages 127-159

  Larson Hall, Jenifer ^a   Plonsky, Luke ^b  

^a Fukuoka Jo Gakuin   (Japan)

^b NORTHERN ARIZONA UNIVERSITY   (United States)

Author keywords

Confidence intervals; Effect sizes; L2; Language; Mean; Meta analysis; Methodology; Quantitative; Raw data; Reporting practices; Standard deviation

Indexed keywords

EID: 84929883176     PISSN: 00238333     EISSN: 14679922     Source Type: Journal    
DOI: 10.1111/lang.12115     Document Type: Article

References (68)

1. Abraham, L. B. (2008). Computer-mediated glosses in second language reading comprehension and vocabulary learning: A meta-analysis. Computer Assisted Language Learning, 21, 199-226.
2. Adesope, O. O., Lavin, T., Thompson, T., & Ungerleider, C. (2010). A systematic review and meta-analysis of the cognitive correlates of bilingualism. Review of Educational Research, 80, 207-245.
3. Adesope, O. O., Lavin, T., Thompson, T., & Ungerleider, C. (2011). Best practices in teaching literacy to ESL immigrant students: A meta-analysis. British Journal of Educational Psychology, 81, 629-653.
4. American Educational Research Association. (2006). Standards for reporting on empirical social science research in AERA publications. Educational Researcher, 35(6), 33-40.
5. American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.
6. APA Publications and Communications Board Working Group on Journal Article Reporting Standards. (December 2008). American Psychologist, 63(9), 839-851.
7. Badges to acknowledge open practices. (2013, February 20). Open Science Framework Beta. Retrieved January 11, 2014, from https://osf.io/tvyxz/wiki/aandi
8. Biber, D., Nekrasova, T., & Horn, B. (2011). The effectiveness of feedback for L1-English and L2 writing development: A meta-analysis. TOEFL iBT Re-search Report No. TOEFLiBT-14. Princeton, NJ: Educational Testing Service.
9. Bowles, M. (2010). Features that make a task amenable to think-aloud: A meta-analysis of studies investigating the validity of think-alouds on verbal tasks. In The think-aloud controversy in second language research (Chap. 3). New York: Routledge.
10. Breckler, S. (2009). Dealing with data. American Psychological Association: Science Directions. Retrieved February 10, 2014, from http://www.apa.org/monitor/2009/02/sd.aspx
11. Brown, D. (in press). The type and linguistic foci of oral corrective feedback in the L2 classroom: A meta-analysis. Language Teaching Research.
12. Byrnes, H. (2013). Notes from the editor. Modern Language Journal, 97, 825-827.
13. Catford, J.C. (1998). Language Learning and the applied linguistics: A historical sketch. Language Learning, 48(4), 465-496.
14. Chaudron, C. (2001). Progress in language classroom research: Evidence from The Modern Language Journal, 1916-2000. The Modern Language Journal, 85, 57-76.
15. Cooper, H. (2011). Reporting research in psychology: How to meet journal article reporting standards. Washington, DC: American Psychological Association.
16. Crookes, G. (1991). Power, effect size, and second language research: Another researcher comments. TESOL Quarterly, 25, 762-765.
17. Cumming, G. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. New York: Routledge.
18. DeKeyser, R. (1997). Beyond explicit rule learning: Automatizing second language morphosyntax. Studies in Second Language Acquisition, 19, 195-221.
19. Dinsmore, T. H. (2006). Principles, parameters, and SLA: A retrospective meta-analytic investigation into adult L2 learners' access to Universal Grammar. In J. M. Norris & L. Ortega (Eds.), Synthesizing research on language learning and teaching (pp. 53-90). Philadelphia: John Benjamins.
20. Ellis, N. C. (2000). Editorial statement. Language Learning, 50, xi-xiii.
21. Funder, D., Levin, J., Mackie, D., Morf, C., Vazire, S., & West, S. (2013). Improving the dependability of research in personality and social psychology: Recommendations for research and educational practices. Personality and Social Psychology Review, 18(1), 3-12.
22. Goldschneider, J. M., & DeKeyser, R. M. (2001). Explaining the natural order of L2 morpheme acquisition: A meta-analysis of multiple determinants. Language Learning, 51, 1-50.
23. Good, J. (2010, January 11). Linguistics society debates data sharing. Open Access News. Retrieved February 10, 2014, from http://legacy.earlham.edu/peters/fos/2010/01/linguistics-society-debates-data.html.
24. Grgurović, M., Chapelle, C. A., & Shelley, M. (2013). A meta-analysis of effectiveness studies on computer technology-supported language learning. ReCALL, 25, 165-198.
25. Henning, G. (1986). Quantitative methods in language acquisition research. TESOL Quarterly, 20, 701-708.
26. Hudson, T. (in press). Presenting quantitative data visually. In L. Plonsky (Ed.), Advancing quantitative methods in second language research. New York: Routledge.
27. Jeon, E. H., & Kaya, T. (2006). Effects of L2 instruction on interlanguage pragmatic development: A meta-analysis. In J. M. Norris & L. Ortega (Eds.), Synthesizing research on language learning and teaching (pp. 165-211). Philadelphia: John Benjamins.
28. Kalinowski, P. (2014). Understanding confidence intervals (CIs) and effect size estimation. Retrieved February 6, 2014, from http://www.psychologicalscience.org/index.php/publications/observer/2010/april-10/understanding-confidence-intervals-cis-and-effect-size-estimation.html
29. Keck, C. M., Iberri-Shea, G., Tracy-Ventura, N., & Wa-Mbaleka, S. (2006). Investigating the empirical link between task-based interaction and acquisition: A meta-analysis. In J. M. Norris & L. Ortega (Eds.), Synthesizing research on language learning and teaching (pp. 91-131). Philadelphia: John Benjamins.
30. Kieffer, M. T., Lesaux, N. K., Rivera, M., & Francis, D. J. (2009). Accommodations for English language learners taking large-scale assessments: A meta-analysis on effectiveness and validity. Review of Educational Research, 79, 1168-1201.
31. Larson-Hall, J. (2008). Weighing the benefits of studying a foreign language at a younger starting age in a minimal input situation. Second Language Research, 24, 35-63.
32. Larson-Hall, J. (2010). A guide to doing statistics in second language research using SPSS. New York: Routledge.
33. Larson-Hall, J. (2015a). A guide to doing statistics in second language research using SPSS and R. New York: Routledge.
34. Larson-Hall, J. (2015b). Moving beyond the bar plot and line graph to create informative and attractive graphics. Manuscript in preparation.
35. Larson-Hall, J., & Herrington, R. (2009). Improving data analysis in second language acquisition by utilizing modern developments in applied statistics. Applied Linguistics, 31, 368-390.
36. Lee, S.-K., & Huang, H.-T. (2008). Visual input enhancement and grammar learning: A meta-analytic review. Studies in Second Language Acquisition, 30, 307-331.
37. Lee, J., Jang, J., & Plonsky, L. (in press). The effectiveness of second language pronunciation instruction: A meta-analysis. Applied Linguistics.
38. Li, S. (2010). The effectiveness of corrective feedback in SLA: A meta-analysis. Language Learning, 60, 309-365.
39. Lin, W.-C., Huang, H.-T., & Liou, H.-C. (2013). The effects of text-based SCMC on SLA: A meta-analysis. Language Learning & Technology, 17, 123-142.
40. Loewen, S., & Gass, S. (2009). Research timeline: The use of statistics in L2 acquisition research. Language Teaching, 42(2), 181-196.
41. Lyster, R. (2004). Differential effects of prompts and recasts in form-focused instruction. Studies in Second Language Acquisition, 26, 399-432.
42. Mackey, A., & Goo, J. (2007). Interaction research in SLA: A meta-analysis and research synthesis. In A. Mackey (Ed.), Conversational interaction in second language acquisition: A collection of empirical studies (pp. 407-451). New York: Oxford University Press.
43. Nekrasova, T., & Becker, T. (2009). Effectiveness of practice: A research synthesis and quantitative meta-analysis. Unpublished manuscript.
44. Nicol, A. A. M., & Pexman, P. M. (2010a). Presenting your findings: A practical guide for creating tables. Washington, DC: American Psychological Association.
45. Nicol, A. A. M., & Pexman, P. M. (2010b). Displaying your findings: A practical guide for creating figures, posters, and presentations. Washington, DC: American Psychological Association.
46. Norris, J. M. (2015). Statistical significance testing in second language research: Basic problems and suggestions for reform. Language Learning, 65(S1), 97-126.
47. Norris, J. M., & Ortega, L. (2000). Effectiveness of L2 instruction: A research synthesis and quantitative meta-analysis. Language Learning, 50, 417-528.
48. Norris, J. M., & Ortega, L. (2006). The value and practice of research synthesis for language learning and teaching. In J. M. Norris & L. Ortega (Eds.), Synthesizing research on language learning and teaching (pp. 3-50). Philadelphia: John Benjamins.
49. Nunan, D. (1991). Methods in second language classroom-oriented research: A critical review. Studies in Second Language Acquisition, 13, 249-274.
50. Pennock-Roman, M., & Rivera, C. (2011). Mean effects of test accommodations for ELLs and non-ELLs: A meta-analysis of experimental studies. Educational Measurement: Issues and Practice, 30, 10-28.
51. Phakiti, A. (2010). Analysing quantitative data. In B. Paltridge & A. Phakiti (Eds.), Continuum companion to research methods in applied linguistics (pp. 39-49). London: Continuum.
52. Plonsky, L. (2011). The effectiveness of second language strategy instruction: A meta-analysis. Language Learning, 61, 993-1038.
53. Plonsky, L. (2013). Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition, 35, 655-687.
54. Plonsky, L. (2014). Study quality in quantitative L2 research (1990-2010): A methodological synthesis and call for reform. Modern Language Journal, 98, 450-470.
55. Plonsky, L., & Derrick, D. J. (2015). A meta-analysis of reliability coefficients in second language research. Manuscript submitted for publication.
56. Plonsky, L., Egbert, J., & LaFlair, G. T. (in press). Bootstrapping in applied linguistics: Assessing its potential using shared data. Applied Linguistics.
57. Plonsky, L., & Gass, S. (2011). Quantitative research methods, study quality, and outcomes: The case of interaction research. Language Learning, 61, 325-366.
58. Plonsky, L., & Oswald, F. L. (2014). How big is "big"? Interpreting effect sizes in L2 research. Language Learning, 64, 878-912.
59. Russell, J., & Spada, N. (2006). The effectiveness of corrective feedback for the acquisition of L2 grammar: A meta-analysis of the research. In J. M. Norris & L. Ortega (Eds.), Synthesizing research on language learning and teaching (pp. 133-164). Philadelphia: John Benjamins.
60. Tufte, E. (2001). The visual display of quantitative information (2nd ed.). Cheshire, CT: Graphics Press.
61. Tufte, E. (2014, January 16). Sparkline theory and practice Edward Tufte [Online forum comment]. Retrieved from http://www.edwardtufte.com/bboard/q-and-a-fetch-msg?msg_id=0001OR
62. Vines, T., Albert, A., Andrew, R., Débarre, F., Bock, D., Franklin, M., et al. (2014). The availability of research data declines rapidly with article age. Current Biology, 24(1), 94-97.
63. Wainer, H. (1996). Depicting error. American Statistician, 50, 101-111.
64. Wa-Mbaleka, S. (2006). A meta-analysis investigating the effects of reading on second language vocabulary learning. Unpublished doctoral dissertation, Northern Arizona University, Flagstaff, AZ.
65. Wang, L.-J. (2010). A meta-analysis of empirical studies on the effects of extensive reading. Unpublished master's thesis, National Tsing Hua University, Taiwan.
66. Wilkinson, L., & Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604. Available from tinyurl.com/tfsi1999
67. Wu, C. (1991). A meta-analysis of intensive audio-lingual language programs: A meta-analysis of intensive audio-lingual language programs: Implications for English as a foreign language instruction in China. Unpublished doctoral dissertation, University of San Francisco.
68. Ziegler, N. (2013). Synchronous computer-mediated communication and interaction: A research synthesis and meta-analysis. Unpublished doctoral dissertation, Georgetown University.