Methodological Synthesis in Quantitative L2 Research: A Review of Reviews and a Case Study of Exploratory Factor Analysis

Language Learning

Volumn 65, Issue S1, 2015, Pages 9-36

  Plonsky, Luke ^a   Gonulal, Talip ^b  

^a NORTHERN ARIZONA UNIVERSITY   (United States)

^b MICHIGAN STATE UNIVERSITY   (United States)

Author keywords

Factor analysis; Meta analysis; Research methods; Research synthesis

Indexed keywords

EID: 84929918248     PISSN: 00238333     EISSN: 14679922     Source Type: Journal    
DOI: 10.1111/lang.12111     Document Type: Article

References (85)

1. American Psychological Association. (2010). Publication manual of the American Psychological Association (6th ed.). Washington, DC: Author.
2. Asención-Delaney, Y., & Collentine, J. (2011). A multidimensional analysis of a written L2 Spanish corpus. Applied Linguistics, 32, 299-322.
3. Bachman, L. F. (2004). Statistical analyses for language assessment. Cambridge, UK: Cambridge University Press.
4. Bandalos, D. L., & Boehm-Kaufman, M. R. (2009). Four common misconceptions in exploratory factor analysis. In C. E. Lance & R. J. Vandenberg (Eds.), Statistical and methodological myths and urban legends: Doctrine, verity and fable in the organizational and social sciences (pp. 61-87). New York: Routledge.
5. Brown, J. D. (2004). Resources on quantitative/statistical research for applied linguists. Second Language Research, 20, 372-393.
6. Brown, J. D. (2009a). Statistics Corner. Questions and answers about language testing statistics: Principal components analysis and exploratory factor analysis-Definitions, differences, and choices. Shiken: JALT Testing & Evaluation SIG Newsletter, 13(1), 26-30.
7. Brown, J. D. (2009b). Statistics Corner. Questions and answers about language testing statistics: Choosing the right number of components or factors in PCA and EFA. Shiken: JALT Testing & Evaluation SIG Newsletter, 13(2), 19-23.
8. Brown, J. D. (2009c). Statistics Corner. Questions and answers about language testing statistics: Choosing the right type of rotation in PCA and EFA. Shiken: JALT Testing & Evaluation SIG Newsletter, 13(3), 20-25.
9. Brown, J. D. (2010a). Statistics Corner. Questions and answers about language testing statistics: How are PCA and EFA used in language test and questionnaire development? Shiken: JALT Testing & Evaluation SIG Newsletter, 14(2), 22-27.
10. Brown, J. D. (2010b). Statistics Corner. Questions and answers about language testing statistics: How are PCA and EFA used in language research? Shiken: JALT Testing & Evaluation SIG Newsletter, 14(1), 19-23.
11. Chapelle, C. A., & Duff, P. A. (2003). Some guidelines for conducting quantitative and qualitative research in TESOL. TESOL Quarterly, 37, 157-178.
12. Chaudron, C. (2001). Progress in language classroom research: Evidence from the Modern Language Journal, 1916-2000. Modern Language Journal, 85, 57-76.
13. Cohen, D., & Macaro, E. (2010). Research methods in second language acquisition. In E. Macaro (Ed.), Continuum companion to second language acquisition (pp. 107-136). London: Continuum.
14. Cohen, J. (1968). Multiple regression as a general data-analytic system. Psychological Bulletin, 70, 426-443.
15. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
16. Comrey, A. L., & Lee, H. B. (1992). A first course in factor analysis (2nd ed.). Hillsdale, NJ: Erlbaum.
17. Conway, J. M., & Huffcutt, A. I. (2003). A review and evaluation of exploratory factor analysis practices in organizational research. Organizational Research Methods, 6, 147-168.
18. Cortina, J. M. (2002). Big things have small beginnings: An assortment of "minor" methodological misunderstandings. Journal of Management, 28, 339-362.
19. Costello, A., & Osborne, J. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research & Evaluation, 10(7), 1-9.
20. DeKeyser, R., Alfi-Shabtay, I., & Ravid, D. (2010). Cross-linguistic evidence for the nature of age effects in second language acquisition. Applied Psycholinguistics, 31, 413-438.
21. Ellis, N. C. (2000). Editorial statement. Language Learning, 50, xi-xiii.
22. Field, A. (2009). Discovering statistics using SPSS. London: SAGE.
23. Fabrigar, L. R., & Wegener, D. T. (2012). Exploratory factor analysis. New York: Oxford University Press.
24. Fabrigar, L. R., Wegener, D. T., MacCallum, R. C., & Strahan, E. J. (1999). Evaluating the use of exploratory factor analysis in psychological research. Psychological Methods, 4, 272-299.
25. Gass, S. (2009). A survey of SLA research. In W. Ritchie & T. Bhatia (Eds.), Handbook of second language acquisition (pp. 3-28). Bingley, UK: Emerald.
26. Gorsuch, R. L. (1983). Factor analysis (2nd ed.). Hillsdale, NJ: Erlbaum.
27. Gorsuch, R. L. (1990). Common factor-analysis versus component analysis: Some well and little known facts. Multivariate Behavioral Research, 25(1), 33-39.
28. Gorsuch, R. L. (2003). Factor analysis. In J. A. Schinka & W. F. Velicer (Eds.), Handbook of psychology: Vol. 2. Research methods in psychology (pp. 143-164). Hoboken, NJ: Wiley.
29. Hair, J., Anderson, R. E., Tatham, R. L., & Black, W. C. (1995). Multivariate data analysis (4th ed.). Englewood Cliffs, NJ: Prentice-Hall.
30. Hatch, E., & Lazaraton, A. (1991). The research manual: Design and statistics for applied linguistics. Rowley, MA: Newbury House.
31. Henson, K. R., Capraro, R. M., & Capraro, M. M. (2004). Reporting practice and use of exploratory factor analysis in educational research journals: Errors and explanation. Research in the Schools, 11(2), 61-72.
32. Henson, K. R., & Roberts, J. K. (2006). Use of exploratory factor analysis in published research: Common errors and some comment on improved practice. Educational and Psychological Measurement, 66, 393-416.
33. Journal Article Reporting Standards Working Group. (2008). Reporting standards for research in psychology: Why do we need them? What might they be? American Psychologist, 63, 839-851.
34. Kieffer, K. M. (1999). An introductory primer on the appropriate use of exploratory and confirmatory factor analysis. Research in the Schools, 6, 75-92.
35. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174.
36. Larson-Hall, J. (2010). A guide to doing statistics in second language research using SPSS. New York: Routledge.
37. Larson-Hall, J., & Plonsky, L. (2015). Reporting and interpreting quantitative research findings: What gets reported and recommendations for the field. Language Learning, 65(S1), 127-159.
38. Lazaraton, A. (2000). Current trends in research methodology and statistics in applied linguistics. TESOL Quarterly, 34, 175-181.
39. Lazaraton, A. (2005). Quantitative research methods. In E. Hinkel (Ed.), Handbook of research in second language teaching and learning (pp. 109-224). Mahwah, NJ: Erlbaum.
40. Lee, J., Jang, J., & Plonsky, L. (in press). The effectiveness of second language pronunciation instruction: A meta-analysis. Applied Linguistics.
41. Li, S. (2010). The effectiveness of corrective feedback in SLA: A meta-analysis. Language Learning, 60, 309-365.
42. Lipsey, M. W. (2009). Identifying interesting variables and analysis opportunities. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis (2nd ed., pp. 147-158). New York: Russell Sage Foundation.
43. Loewen, S., & Gass, S. (2009). Research timeline: The use of statistics in L2 acquisition research. Language Teaching, 42, 181-196.
44. Loewen, S., & Gonulal, T. (in press). Principal component analysis and factor analysis. In L. Plonsky (Ed.), Advancing quantitative methods in second language research. New York: Routledge.
45. Loewen, S., Lavolette, B., Spino, L. A. Papi, M., Schmidtke, J., Sterling, S., et al. (2014). Statistical literacy among applied linguistics and second language acquisition researchers. TESOL Quarterly, 48, 360-388.
46. Loewen, S., Li, S., Fei, F., Thompson, A., Nakatsukasa, K., Ahn, S., & Chen, X. (2009). Second language learners' beliefs about grammar instruction and error correction. Modern Language Journal, 93(1), 91-104.
47. Macaro, E., Handley, Z., & Walter, C. (2012). A systematic review of CALL in English as a second language: Focus on primary and secondary education. Language Teaching, 45, 1-43.
48. Mackey, A., & Goo, J. (2007). Interaction research in SLA: A meta-analysis and research synthesis. In A. Mackey (Ed.), Conversational interaction in second language acquisition: A collection of empirical studies (pp. 407-449). Oxford, UK: Oxford University Press.
49. Norris, J. M. (2015). Statistical significance testing in second language research: Basic problems and suggestions for reform. Language Learning, 65(S1), 97-126.
50. Norris, J. M., & Ortega, L. (2000). Effectiveness of L2 instruction: A research synthesis and quantitative meta-analysis. Language Learning, 50, 417-528.
51. Norris, J. M., & Ortega, L. (2006). The value and practice of research synthesis for language learning and teaching. In J. M. Norris & L. Ortega (Eds.), Synthesizing research on language learning and teaching (pp. 3-50). Philadelphia: John Benjamins.
52. Nunan, D. (1991). Methods in second language classroom-oriented research: A critical review. Studies in Second Language Acquisition, 13, 249-274.
53. Ortega, L. (2005). For what and for whom is our research? The ethical as transformative lens in instructed SLA. Modern Language Journal, 89, 427-443.
54. Ortega, L. (2009). Understanding second language acquisition. London: Hodder Education.
55. Oswald, F. L., & Plonsky, L. (2010). Meta-analysis in second language research: Choices and challenges. Annual Review of Applied Linguistics, 30, 85-110.
56. Pett, M. A., Lackey, N. R., & Sullivan, J. J. (2003). Making sense of factor analysis: The use of factor analysis for instrument development in health care research. Thousand Oaks, CA: SAGE.
57. Plonsky, L. (2011). The effectiveness of second language strategy instruction: A meta-analysis. Language Learning, 61, 993-1038.
58. Plonsky, L. (2012). Replication, meta-analysis, and generalizability. In G. Porte (Ed.), Replication research in applied linguistics (pp. 116-132). New York: Cambridge University Press.
59. Plonsky, L. (2013). Study quality in SLA: An assessment of designs, analyses, and reporting practices in quantitative L2 research. Studies in Second Language Acquisition, 35, 655-687.
60. Plonsky, L. (2014a, February). Sampling, power, and generalizability in L2 research (Or, why we might as well be flipping coins). Keynote presentation at the Second Language Studies Symposium, East Lansing, MI.
61. Plonsky, L. (2014b). Study quality in quantitative L2 research (1990-2010): A methodological synthesis and call for reform. Modern Language Journal, 98, 450-470.
62. Plonsky, L. (in press). Statistical power, p values, descriptive statistics, and effect sizes: A "back-to-basics" approach to advancing quantitative methods in L2 research. In L. Plonsky (Ed.), Advancing quantitative methods in second language research. New York: Routledge.
63. Plonsky, L., Egbert, J., & LaFlair, G. T. (in press). Bootstrapping in applied linguistics: Assessing its potential using shared data. Applied Linguistics.
64. Plonsky, L., & Gass, S. (2011). Quantitative research methods, study quality, and outcomes: The case of interaction research. Language Learning, 61(2), 325-366.
65. Plonsky, L., & Gurzynski-Weiss, L. (2014). Research methods. In C. Fäcke (Ed.), Manual of language acquisition (pp. 31-49). Berlin: De Gruyter.
66. Plonsky, L., & Oswald, F. L. (2014). How big is "big"? Interpreting effect sizes in L2 research. Language Learning, 64, 878-912.
67. Plonsky, L., & Oswald, F. L. (in press). Meta-analyzing second language research. In L. Plonsky (Ed.), Advancing quantitative methods in second language research. New York: Routledge.
68. Porte, G. (Ed.). (2012). Replication research in applied linguistics. Cambridge, UK: Cambridge University Press.
69. Preacher, K. J., & MacCallum, R. C. (2003). Repairing Tom Swift's electric factor analysis machine. Understanding Statistics, 2, 13-43.
70. Raykov, T., & Marcoulides, G. A. (2008). An introduction to applied multivariate analysis. New York: Taylor & Francis.
71. Richards, K., Ross, S., & Seedhouse, P. (2012). Research methods for applied language studies. New York: Routledge.
72. Selinker, L., & Lakshmanan, U. (2001). How do we know what we know? Why do we believe what we believe? Second Language Research, 17, 323-325.
73. Shen, W., Kiger, T. B., Davies, S. E., Rasch, R. L., Simon, K. M., & Ones, D. S. (2011). Samples in applied psychology: Over a decade of research in review. Journal of Applied Psychology, 96, 1055-1065.
74. Stevens, J. (1996). Applied multivariate statistics for the social sciences (3rd ed.). Mahwah, NJ: Erlbaum.
75. Stevens, J. P. (2009). Applied multivariate statistics for the social sciences (5th ed.). New York: Routledge.
76. Spinner, P. (2011). Review article: Second language acquisition of Bantu languages: A (mostly) untapped research opportunity. Second Language Research, 27, 418-430.
77. Tabachnick, B., & Fidell, L. (2013). Using multivariate statistics (6th ed.). Boston: Pearson Education.
78. Tarone, E. (2013). Applied linguistics without borders. Language Teaching, 46, 355-364.
79. Thompson, B. (2004). Exploratory and confirmatory factor analysis: Understanding concepts and applications. Washington, DC: American Psychological Association.
80. Vacha-Haase, T., & Thompson, B. (2004). How to estimate and interpret various effect sizes. Journal of Counseling Psychology, 51, 473-481.
81. Valentine, J. C., & Cooper, H. (2008). A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: The study design and implementation assessment device (Study DIAD). Psychological Methods, 13, 130-149.
82. Velicer, W. F., Eato, C. A., & Fava, J. L. (2000). Construct explication through factor or component analysis: A review and evaluation of alternative procedures for determining the number of factors or components. In R. D. Goffin & E. Helmes (Eds.), Problems and solutions in human assessment: Honoring Douglas N. Jackson at seventy (pp. 41-71). Norwell, MA: Kluwer Academic.
83. Wilkinson, L., & Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.
84. Wintre, M. G., North, C., & Sugar, L. A. (2001). Psychologists' response to criticisms about research based on undergraduate participants: A developmental perspective. Canadian Psychology, 42, 216-225.
85. Wittenborn, J., & Larsen, R. (1944). A factorial study of achievement in college German. Journal of Educational Psychology, 35, 39-48.