Statistical Significance Testing in Second Language Research: Basic Problems and Suggestions for Reform

Language Learning

Volumn 65, Issue S1, 2015, Pages 97-126

  Norris, John M ^a  

^a GEORGETOWN UNIVERSITY   (United States)

Author keywords

Inferential statistics; Null hypothesis significance testing; p value; Recommended practices

Indexed keywords

EID: 84929929050     PISSN: 00238333     EISSN: 14679922     Source Type: Journal    
DOI: 10.1111/lang.12114     Document Type: Article

References (42)

1. Adler, P. B., Seabloom, E. W., Borer, E. T., Hillebrand, H., Hautier, Y., Hector, A., et al. (2011). Productivity is a poor predictor of plant species richness. Science, 333, 1750-1753.
2. American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.
3. Brown, J. D. (1990). The use of multiple t-tests in language research. TESOL Quarterly, 24, 770-773.
4. Brown, J. D. (1991). Statistics as a foreign language: What to look for in reading statistical language studies-Part 1. TESOL Quarterly, 25, 569-586.
5. Brown, J. D. (1992). Statistics as a foreign language-Part 2: More things to look for in reading statistical language studies. TESOL Quarterly, 26, 629-664.
6. Carver, R. P. (1978). The case against statistical significance testing. Harvard Educational Review, 48, 378-399.
7. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
8. Cohen, J. (1990). Things I have learned (so far). American Psychologist, 45, 1304-1312.
9. Cohen, J. (1994). The earth is round (p < .05). American Psychologist, 49, 997-1003.
10. Cowles, M., & Davis, C. (1982). On the origins of the .05 level of statistical significance. American Psychologist, 37, 553-558.
11. Crookes, G. (1991). Power, effect size, and second language research: Another researcher comments. TESOL Quarterly, 25, 762-765.
12. Cumming, G. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. New York: Routledge.
13. Eich, E. (2014). Business not as usual. Psychological Science, 25, 3-6.
14. Franco, A., Malhotra, N., & Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345, 1502-1505.
15. Gass, S., Mackey, A., Alvarez-Torres, M., & Fernández-García, M. (1999). The effects of task repetition on linguistic output. Language Learning, 49, 549-581.
16. Harlow, L. L., Mulaik, S. A., & Steiger, J. H. (Eds.). (1997). What if there were no significance tests? Mahwah, NJ: Erlbaum.
17. Hudson, T., & Llosa, L. (2015). Design issues and inference in experimental L2 research. Language Learning, 65(S1), 76-96.
18. Kirk, R. E. (1996). Practical significance: A concept whose time has come. Educational and Psychological Measurement, 56, 746-759.
19. Kline, R. B. (2004). Beyond significance testing: Reforming data analysis methods in behavioral research. Washington, DC: American Psychological Association.
20. Larson-Hall, J., & Plonsky, L. (2015). Reporting and interpreting quantitative research findings: What gets reported and recommendations for the field. Language Learning, 65(S1), 127-159.
21. Li, S. (2010). The effectiveness of corrective feedback in SLA: A meta-analysis. Language Learning, 60, 309-365.
22. Limentani, G. B., Ringo, M. C., Ye, F., Bergquist, M. L., & McSorley, E. O. (2005). Beyond the t-test: Statistical equivalence testing. Analytical Chemistry, 77, 221A-226A.
23. Mackey, A., & Goo, J. (2007). Interaction research in SLA: A meta-analysis and research synthesis. In A. Mackey (Ed.), Conversational interaction in second language acquisition: A collection of empirical studies (pp. 407-449). Oxford, UK: Oxford University Press.
24. Meehl, P. E. (1967). Theory-testing in psychology and physics: A methodological paradox. Philosophy of Science, 34, 103-115.
25. Norris, J. M., & Ortega, L. (2000). Effectiveness of L2 instruction: A research synthesis and quantitative meta-analysis. Language Learning, 50, 417-528.
26. Norris, J. M., & Ortega, L. (Eds.). (2006). Synthesizing research on language learning and teaching. Amsterdam: John Benjamins.
27. Oakes, M. W. (1986). Statistical inference: A commentary for the social and behavioral sciences. Chichester, UK: Wiley.
28. Plonsky, L. (2011). The effectiveness of second language strategy instruction: A meta-analysis. Language Learning, 61, 993-1038.
29. Plonsky, L. (2013). Study quality in SLA. Studies in Second Language Acquisition, 35, 655-687.
30. Plonsky, L. (2014). Study quality in quantitative L2 research (1990-2010): A methodological synthesis and call for reform. The Modern Language Journal, 98, 450-470.
31. Plonsky, L. (in press). Statistical power, p values, descriptive statistics, and effect sizes: A "back-to-basics" approach to advancing quantitative methods in L2 research. In L. Plonsky (Ed.), Advancing quantitative methods in second language research. New York: Routledge.
32. Plonsky, L., & Gass, S. (2011). Quantitative research methods, study quality, and outcomes: The case of interaction research. Language Learning, 61, 325-366.
33. Plonsky, L., & Oswald, F. (2014). How big is "big"? Interpreting effect sizes in L2 research. Language Learning, 64, 878-912.
34. Purpura, J. E., Brown, J. D., & Schoonen, R. (2015). Improving the validity of quantitative measures in applied linguistics research. Language Learning, 65(S1), 37-75.
35. Ross, S. J., & Mackey, B. (2015). Bayesian approaches to imputation, hypothesis testing, and parameter estimation. Language Learning, 65(S1), 208-227.
36. Salsburg, D. (2001). The lady tasting tea: How statistics revolutionized science in the twentieth century. New York: Macmillan.
37. Sullivan, G. M., & Feinn, R. (2012). Using effect size-Or why the p value is not enough. Journal of Graduate Medical Education, 4, 279-282.
38. Tabachnick, B. G., & Fidell, L. S. (2013). Using multivariate statistics (6th ed.). Boston: Pearson.
39. Thompson, B. (1996). Research news and comment: AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researcher, 25, 26-30.
40. Thompson, B. (2006). Foundations of behavioral statistics: An insight-based approach. New York: Guilford Press.
41. Wilcox, R. R. (2012). Introduction to robust estimation and hypothesis testing (3rd ed.). Waltham, MA: Academic Press.
42. Wilkinson, L., & Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.