How to do a Meta-Analysis

Research Methods in Second Language Acquisition: A Practical Guide

Volumn , Issue , 2012, Pages 275-295

  Plonsky, Luke ^a   Oswald, Frederick L ^b  

^a MICHIGAN STATE UNIVERSITY   (United States)

^b RICE UNIVERSITY   (United States)

Author keywords

Analysis phase, missing data; Coding in meta analysis; Coding sheet, data collection; Corrective feedback; Doing a meta analysis; L2 self assessment validity; Meta analysis, and sla toolbox; Metaanalysis; Narrative reviews, nhst reliance; Statistical method, mean and variance

Indexed keywords

EID: 84874459755     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1002/9781444347340.ch14     Document Type: Chapter

References (64)

1. Abraham, L. B. (2008). Computer-mediated glosses in second language reading comprehension and vocabulary learning: A meta-analysis. Computer Assisted Language Learning, 21(3), 199-226.
2. APA Publications and Communications Board Working Group on Journal Article Reporting Standards. (2008). Reporting standards for research in psychology: Why do we need them? What might they be? American Psychologist, 63, 839-851.
3. Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R. (2009). Introduction to meta-analysis. Chichester, England: Wiley.
4. Cheung, S. F., & Chan, D. K-S. (2004). Dependent effect sizes in meta-analysis: Incorporating the degree of interdependence. Journal of Applied Psychology, 89(5), 780-791.
5. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
6. Cohn, L. D., & Becker, B. J. (2003). How meta-analysis increases statistical power. Psychological Methods, 8(3), 243-253.
7. Cooper, H. (1998). Synthesizing research: A guide for literature reviews. Thousand Oaks, CA: Sage.
8. Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.). (2009). The handbook of research synthesis and meta-analysis (2nd ed.). New York, NY: Russell Sage Foundation.
9. Cooper, H. M., & Rosenthal, R. (1980). Statistical versus traditional procedures for summarizing research findings. Psychological Bulletin, 87(3), 442-449.
10. Dalton, D. R., & Dalton, C. M. (2008). Meta-analyses: Some very good steps toward a bit longer journey. Organizational Research Methods, 11(1), 127-147.
11. Fern, E. F., & Monroe, K. B. (1996). Effect-size estimates: Issues and problems in interpretation. Journal of Consumer Research, 23(2), 89-105.
12. Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3-8.
13. Gleser, L. J., & Olkin, I. (2009). Stochastically dependent effect sizes. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 357-376). New York, NY: Sage.
14. Hattie, J. A., Biggs, J., & Purdie, N. (1996). Effects of learning skills interventions on student learning: A meta-analysis. Review of Educational Research, 66(2), 99-136.
15. Hedges, L. V. (2008). What are effect sizes and why do we need them? Child Development Perspectives, 2(3), 167-171.
16. Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
17. Hedges, L. V., & Pigott, T. D. (2001). The power of statistical tests in meta-analysis. Psychological Methods, 6(3), 203-217.
18. Higgins, J. P. T., White, I. R., & Wood, A. M. (2008). Imputation methods for missing outcome data in meta-analysis of clinical trials. Clinical Trials, 5(3), 225-239.
19. Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage.
20. In'nami, Y., & Koizumi, R. (2009). A meta-analysis of test format effects on reading and listening test performance: Focus on multiple-choice and open-ended formats. Language Testing, 26(2), 219-244.
21. In'nami, Y., & Koizumi, R. (2010). Database selection guidelines for meta-analysis in applied linguistics. TESOL Quarterly, 44(1), 169-184.
22. Kirk, R. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56(5), 746-759.
23. Kline, R. B. (2004). Beyond significance testing: Reforming data analysis methods in behavioral research. Washington, DC: American Psychological Association.
24. Lee, S-K., & Huang, H-T. (2008). Visual input enhancement and grammar learning: A metaanalytic review. Studies in Second Language Acquisition, 30(3), 307-331.
25. Li, S. (2010). The effectiveness of corrective feedback in SLA: A meta-analysis. Language Learning, 60(2), 309-365.
26. Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage. Lyster, R., & Saito, K. (2010). Oral feedback in classroom SLA: A meta-analysis. Studies in Second Language Acquisition, 32(2), 265-302.
27. Mackey, A., & Goo, J. (2007). Interaction research in SLA: A meta-analysis and research synthesis. In A. Mackey (Ed.), Conversational interaction in second language acquisition: A collection of empirical studies (pp. 407-451). New York, NY: Oxford University Press.
28. Masgoret, A-M., & Gardner, R. C. (2003). Attitudes, motivation, and second language learning: A meta-analysis of studies conducted by Gardner and associates. Language Learning, 53(1), 123-163.
29. McManus, R. J., Wilson, S., Delaney, B. C., Fitzmaurice, D. A., Hyde, C. J., Tobias, R. S., Jowett, S., & Hobbs, F. D. R. (1998). Review of the usefulness of contacting other experts when conducting a literature search for systematic reviews. British Medical Journal, 317, 1562-1563.
30. Morris, S. B. (2008). Estimating effect sizes from pretest-posttest-control group designs. Organizational Research Methods, 11(2), 364-386.
31. Norris, J. M., & Ortega, L. (2000). Effectiveness of L2 instruction: A research synthesis and quantitative meta-analysis. Language Learning, 50(3), 417-528.
32. Norris, J. M., & Ortega, L. (2006a). Synthesizing research on language learning and teaching. Philadelphia, PA: John Benjamins.
33. Norris, J. M., & Ortega, L. (2006b). The value and practice of research synthesis for language learning and teaching. In J. M. Norris & L. Ortega (Eds.), Synthesizing research on language learning and teaching (pp. 3-50). Philadelphia, PA: John Benjamins.
34. Norris, J. M., & Ortega, L. (2007). The future of research synthesis in applied linguistics: Beyond art or science. TESOL Quarterly, 41(4), 805-815.
35. Norris, J. M., & Ortega, L. (2010). Research timeline: Research synthesis. Language Teaching, 43, 461-479.
36. Ortega, L. (2010). Research synthesis. In B. Paltridge & A. Phakiti (Eds.), Companion to research methods in applied linguistics (pp. 111-126). London, England: Continuum.
37. Orwin, R. G. (1994). Evaluating coding decisions. In H. Cooper & L. V. Hedges (Eds.), Handbook of research synthesis (pp. 139-162). New York, NY: Russell Sage Foundation.
38. Orwin, R., & Cordray, D. S. (1985). Effects of deficient reporting on meta-analysis: A conceptual framework and reanalysis. Psychological Bulletin, 97(1), 134-147.
39. Oswald, F. L., & Johnson, J. W. (1998). On the robustness, bias, and stability of statistics from meta-analysis of correlation coefficients: Some initial Monte Carlo findings. Journal of Applied Psychology, 83(2), 164-178.
40. Oswald, F. L., & McCloy, R. A. (2003). Meta-analysis and the art of the average. In K. R. Murphy (Ed.), Validity generalization: A critical review (pp. 311-338). Mahwah, NJ:Lawrence Erlbaum.
41. Oswald, F. L., & Plonsky, L. (2010). Meta-analysis in second language research: Choices and challenges. Annual Review of Applied Linguistics, 30, 85-110.
42. Pearson, K. (1904). Report on certain enteric fever inoculation statistics. British Medical Journal, 3, 1243-1246.
43. Plonsky, L. (2011). Study quality in SLA: A cumulative and developmental assessment of designs, analyses, reporting practices, and outcomes in quantitative L2 research. Unpublished doctoral dissertation, Michigan State University, East Lansing, MI.
44. Plonsky, L. (2011). Replication, meta-analysis, and generalizability. In G. Porte (Ed.), A guide to replication in applied linguistics. New York, NY: Cambridge University Press.
45. Plonsky, L. (in press). The effectiveness of second language strategy instruction: A meta-analysis. Language Learning.
46. Plonsky, L., & Gass, S. M. (2011). Quantitative research methods, study quality, and outcomes: The case of interaction research. Language Learning, 61, 325-366.
47. Poltavtchenko, E., & Johnson, M. D. (2009, March). Feedback and second language writing: A meta-analysis. Poster session presented at the annual meeting of TESOL, Denver, CO.
48. Prentice, D. A., & Miller, D. T. (1992). When small effects are impressive. Psychological Bulletin, 112(1), 160-164.
49. Rosenthal, R. (1978). Combining results of independent studies. Psychological Bulletin, 85(1), 185-193.
50. Ross, S. (1998). Self-assessment in second language testing: A meta-analysis and analysis of experiential factors. Language Testing, 15(1), 1-20.
51. Rothstein, H. R., Sutton, A. J., & Borenstein, M. (Eds.). (2005). Publication bias in metaanalysis: Prevention, assessment and adjustments. Chichester, England: Wiley.
52. Russell, J., & Spada, N. (2006). The effectiveness of corrective feedback for the acquisition of L2 grammar: A meta-analysis of the research. In J. M. Norris & L. Ortega (Eds.), Synthesizing research on language learning and teaching (pp. 133-164). Philadelphia, PA: John Benjamins.
53. Schmidt, F. L., & Hunter, J. E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62(5), 529-540.
54. Schmidt, F. L., Le, H., & Oh, I-S. (2009). Correcting for the distorting effects of study artifacts in meta-analysis. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (2nd ed., pp. 317-333). New York, NY: Russell Sage Foundation.
55. Schmidt, F. L., Oh, I-S., & Hayes, T. (2009). Fixed versus random effects models in metaanalysis: Model properties and an empirical comparison of differences in results. British Journal of Mathematical and Statistical Psychology, 62(1), 97-128.
56. Spada, N., & Tomita, Y. (2010). Interactions between type of instruction and type of language feature: A meta-analysis. Language Learning, 60(2), 263-308.
57. Sutton, A. J., & Higgins, J. P. T. (2008). Recent development in meta-analysis. Statistics in Medicine, 27(5), 625-650.
58. Taylor, A., Stevens, J. R., & Asher, J. W. (2006). The effects of explicit reading strategy training on L2 reading comprehension: A meta-analysis. In J. M. Norris & L. Ortega (Eds.), Synthesizing research on language learning and teaching (pp. 213-244). Philadelphia, PA: John Benjamins.
59. Truscott, J. (2007). The effect of error correction on learners' ability to write accurately. Journal of Second Language Writing, 16(4), 255-272.
60. Valentine, J. C., Pigott, T. D., & Rothstein, H. R. (2010). How many studies do you need?: A primer on statistical power for meta-analysis. Journal of Educational and Behavioral Statistics, 35(2), 215-247.
61. Volker, M. A. (2006). Reporting effect size estimates in school psychology research. Psychology in the Schools, 43(6), 653-672.
62. White, H. D. (2009). Scientific communication and literature retrieval. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis (2nd ed., pp. 51-71). New York, NY: Russell Sage Foundation.
63. Wilkinson, L., & Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54(8), 594-604.
64. Wilson, D. B. (2009). Systematic coding. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis (2nd ed., pp. 159-176). New York, NY: Russell Sage Foundation.