Reconceptualizing Reactivity of Think-Alouds and Eye Tracking: Absence of Evidence Is Not Evidence of Absence

Language Learning

Volumn 65, Issue 4, 2015, Pages 896-928

  Godfroid, Aline ^a   Spino, Le Anne ^a  

^a MICHIGAN STATE UNIVERSITY   (United States)

Author keywords

Equivalence tests; Eye tracking; Null hypothesis significance testing; Reactivity; Think alouds

Indexed keywords

EID: 84945453685     PISSN: 00238333     EISSN: 14679922     Source Type: Journal    
DOI: 10.1111/lang.12136     Document Type: Article

References (68)

1. Altman, D. G., & Bland, J. M. (1995). Absence of evidence is not evidence of absence. British Medical Journal, 311, 485. doi:10.1136/bmj.311.7003.485
2. Bachelor of Arts in Linguistics and Literature: Objectives. (2014). Retrieved May 31, 2013, from https://caliweb.cumulus.vub.ac.be/caliweb/?page=program&id=00074&target=pr&language=en&output=html&year=1314
3. Bowles, M. A. (2010). The think-aloud controversy in second language research. New York: Routledge.
4. Bowles, M. A., & Leow, R. P. (2005). Reactivity and type of verbal report in SLA research methodology: Expanding the scope of investigation. Studies in Second Language Acquisition, 27, 415-440. doi:10.1017/S0272263105050187
5. Brown, J. D. (2005). Testing in language programs: A comprehensive guide to English language assessment. New York: McGraw-Hill.
6. Chen, J. J., Tsong, Y., & Kang, S. (2000). Tests for equivalence or noninferiority between two proportions. Drug Information Journal, 34, 569-578. doi:10.1177/009286150003400225
7. Coltheart, M., Rastle, K., Perry, C., Langdon, R., & Ziegler, J. (2001). DRC: A dual route cascaded model of visual word recognition and reading aloud. Psychological Review, 108, 204-256. doi:10.1037/0033-295x.108.1.204
8. Council of Europe. (2001). Common European framework of reference for languages: Learning, teaching, assessment. Cambridge, UK: Cambridge University Press.
9. Dörnyei, Z. (2009a). The L2 motivational self system. In Z. Dörnyei & E. Ushioda (Eds.), Motivation, language identity and the L2 self (pp. 9-42). Bristol, UK: Multilingual Matters.
10. Dörnyei, Z. (2009b). The psychology of second language acquisition. Oxford, UK: Oxford University Press.
11. Dussias, P. E. (2010). Uses of eye-tracking data in second language sentence processing research. Annual Review of Applied Linguistics, 30, 149-166. doi:10.1017/s026719051000005x
12. Engbert, R., & Kliegl, R. (2011). Parallel graded attention models of reading. In S. P. Liversedge, I. D. Gilchrist, & S. Everling (Eds.), The Oxford handbook of eye movements (pp. 787-800). New York: Oxford University Press.
13. Engbert, R., Nuthmann, A., Richter, E. M., & Kliegl, R. (2005). SWIFT: A dynamical model of saccade generation during reading. Psychological Review, 112, 777-813. doi:10.1037/0033-295X.112.4.777
14. Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.
15. Ferguson, C. J. (2009). An effect size primer: A guide for clinians and researchers. Professional Psychology: Research and Practice, 40, 532-538. doi:10.1037/a0015808
16. Field, A. (2013). Discovering statistics using SPSS (4th ed.). Thousand Oaks, CA: SAGE.
17. Fox, M. C., Ericsson, K. A., & Best, R. (2011). Do procedures for verbal reporting of thinking have to be reactive? A meta-analysis and recommendations for best reporting methods. Psychological Bulletin, 137, 316-344. doi:10.1037/a0021663
18. Gass, S., & Mackey, A. (2000). Stimulated recall methodology in second language research. Mahwah, NJ: Erlbaum.
19. Godfroid, A., Boers, F., & Housen, A. (2013). An eye for words: Gauging the role of attention in L2 vocabulary acquisition by means of eye-tracking. Studies in Second Language Acquisition, 35(2), 483-517.
20. Godfroid, A., & Schmidtke, J. (2013). What do eye movements tell us about awareness? A triangulation of eye-movement data, verbal reports and vocabulary learning scores. Honolulu: University of Hawai'i at Manoa, National Foreign Language Resource Center.
21. Godfroid, A., & Winke, P. (in press). Investigating implicit and explicit processing using L2 learners' eye-movement data. In P. Rebuschat (Ed.), Implicit and explicit learning of languages. Amsterdam: John Benjamins.
22. Godfroid, A., Winke, P., & Gass, S. (2013). Thematic issue on eye tracking in second language acquisition research [Special issue]. Studies in Second Language Acquisition, 35.
23. Grabe, W., & Stoller, F. L. (2011). Teaching and researching reading (2nd ed.). Harlow, UK: Longman.
24. Haddock, C. K., Rindskopf, D., & Shadish, W. R. (1998). Using odds ratios as effect sizes for meta-analysis of dichotomous data: A primer on methods and issues. Psychological Methods, 3, 339-353. doi:10.1037/1082-989x.3.3.339
25. Hama, M., & Leow, R. P. (2010). Learning without awareness revisited: Extending Williams (2005). Studies in Second Language Acquisition, 32, 465-491. doi:10.1017/S0272263110000045
26. Hardin, J. W., & Hilbe, J. M. (2013). Generalized estimating equations (2nd ed.). Boca Raton, FL: CRC Press.
27. Hinkelmann, K., & Kempthorne, O. (2007). Design and analysis of experiments: Introduction to experimental design (Vol. 1). Hoboken, NJ: Wiley.
28. Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., & van de Weijer, J. (2011). Eye-tracking: A comprehensive guide to methods and measures. Oxford, UK: Oxford University Press.
29. Hulstijn, J. H. (2003). Incidental and intentional learning. In C. Doughty & M. H. Long (Eds.), The handbook of second language acquisition (pp. 349-375). Malden, MA: Blackwell.
30. IBM Knowledge Center. (2011). Goodness of fit, from http://www-01.ibm.com/support/knowledgecenter/SSLVMB_20.0.0/com.ibm.spss.statistics.cs/gee_wheeze_fit.htm
31. Jegerski, J., & VanPatten, B. (2014). Research methods for second language psycholinguistics. New York: Routledge.
32. Just, M. A., & Carpenter, P. A. (1980). A theory of reading: From eye fixations to comprehension. Psychological Review, 87, 326-354. doi:10.1037/0033-295x.87.4.329
33. Kaakinen, J. K., & Hyönä, J. (2005). Perspective effects on expository test comprehension: Evidence from think-aloud protocols, eye-tracking, and recall. Discourse Processes, 40, 239-257. doi:10.1207/s15326950dp4003_4
34. Khalifa, H., & Weir, C. (2009). Examining reading: Research and practice in assessing second language reading. Cambridge, UK: Cambridge University Press.
35. Kline, R. (2004). Beyond significance testing: Reforming data analysis methods in behavioral research. Washington, DC: American Psychological Association.
36. Labov, W. (1972). Sociolinguistic patterns. Philadelphia: University of Pennsylvania.
37. Larson-Hall, J. (2010). A guide to doing statistics in second language research using SPSS. New York: Routledge.
38. Leow, R. P., Grey, S., Marijuan, S., & Moorman, C. (2014). Concurrent data elicitation procedures, processes, and the early stages of L2 learning: A critical overview. Second Language Research, 30, 111-127. doi:10.1177/0267658313511979
39. Leow, R. P., Hsieh, H., & Moreno, N. (2008). Attention to form and meaning revisited. Language Learning, 58, 665-695. doi:10.1111/j.1467-9922.2008.00453.x
40. Leow, R. P., & Morgan-Short, K. (2004). To think aloud or not to think aloud: The issue of reactivity in SLA research methodology. Studies in Second Language Acquisition, 26, 35-57. doi:10.1017/S0272263104026129
41. Leung, J. H. C., & Williams, J. N. (2011). The implicit learning of mappings between forms and contextually derived meanings. Studies in Second Language Acquisition, 33, 33-55. doi:10.1017/S0272263110000525
42. Limentani, G. B., Ringo, M. C., Ye, F., Bergquist, M. L., & McSorley, E. O. (2005). Beyond the T-test: Statistical equivalence testing. Analytical Chemistry, 77, 221-226. doi:10.1021/ac053390m
43. Loewen, S., Lavolette, E., Spino, L., Papi, M., Schmidtke, J., Sterling, S., & Wolff, D. (2013). Statistical literacy among applied linguists and second language acquisition researchers. TESOL Quarterly, 48, 360-388. doi:10.1002/tesq.128
44. Martínez-Fernández, A. (2008). Revisiting the involvement load hypothesis: Awareness, type of task, and type of item. In M. Bowles (Ed.), Selected proceedings of the 2007 Second Language Research Forum (pp. 210-228). Somerville, MA: Cascadilla Proceedings Project.
45. Mascha, E. J., & Sessler, D. I. (2011). Equivalence and noninferiority testing in regression models and repeated-measures designs. Anesthesia & Analgesia, 112, 678-687. doi:10.1213/ANE.0b013e318206f872
46. Morgan-Short, K., Heil, J., Botero-Moriarty, A., & Ebert, S. (2012). Allocation of attention to second language form and meaning: Issues of think-alouds and depth of processing. Studies in Second Language Acquisition, 34, 659-685. doi:10.1017/s027226311200037x
47. Morrison, R. E. (1983). Retinal image size and the perceptual span in reading. In K. Rayner (Ed.), Eye movements in reading: Perceptual and language processes (pp. 31-40). New York, NY: Academic Press.
48. Murphy, K. R., & Myors, B. (2004). Statistical power analysis. Mahwah, NJ: Erlbaum.
49. Oswald, F. L., & Plonsky, L. (2010). Meta-analysis in second language research: Choices and challenges. Annual Review of Applied Linguistics, 30, 85-110. doi:10.1017/s0267190510000115
50. Plonsky, L. (2013). Study quality in SLA: An assessment of designs, analyses and reporting practices in quantitative L2 research. Studies in Second Language Acquisition, 35, 655-687. doi:10.1017/S0272263113000399
51. Plonsky, L., & Gass, S. (2011). Quantitative research methods, study quality, and outcomes: The case of interaction research. Language Learning, 61, 325-366. doi:10.1111/j.1467-9922.2011.00640.x
52. Plonsky, L., & Oswald, F. L. (2014). How big is "big"? Interpreting effect sizes in L2 research. Language Learning, 64, 878-912. doi:10.1111/lang.12079
53. Polio, C., & Wang, J. (under review). The reactivity of concurrent verbal protocols in second language reading research: A study of advanced second language readers.
54. Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372-422. doi:10.1037/0033-2909.124.3.372
55. Rayner, K. (2009). Eye movements and attention in reading, scene perception, and visual search. The Quarterly Journal of Experimental Psychology, 62, 1457-1506. doi:10.1080/17470210902816461
56. Reichle, E. D. (2011). Serial-attention models of reading. In S. P. Liversedge, I. D. Gilchrist, & S. Everling (Eds.), The Oxford handbook of eye movements (pp. 767-786). New York: Oxford University Press.
57. Reichle, E. D., Liversedge, S. P., Drieghe, D., Blythe, H. I., Joseph, H., White, S. J., & Rayner, K. (2013). Using E-Z Reader to examine the concurrent development of eye-movement control and reading skill. Developmental Review, 33, 110-149. doi:10.1016/j.dr.2013.03.001
58. Rogers, J. L., Howard, K. I., & Vessey, J. T. (1993). Using significance tests to evaluate equivalence between two experimental groups. Quantitative Methods in Psychology, 113, 553-565. doi:10.1037/0033-2909.113.3.553
59. Sachs, R., & Polio, C. (2007). Learners' use of two types of written feedback on a L2 writing revision task. Studies in Second Language Acquisition, 29, 67-100. doi:10.1017/s0272263107070039
60. Sanz, C., Lin, H.-J., Lado, B., Bowden, H. W., & Stafford, C. A. (2009). Concurrent verbalizations, pedagogical conditions, and reactivity: Two CALL studies. Language Learning, 59, 33-71. doi:10.1111/j.1467-9922.2009.00500.x
61. Schmidt, R. (1990). The role of consciousness in second language learning. Applied Linguistics, 11, 129-158. doi:10.1093/applin/11.2.129
62. Smith, B. (2012). Eye tracking as a measure of noticing: A study of explicit recasts in SCMC. Language Learning & Technology, 16, 53-81.
63. Spinner, P., Gass, S. M., & Behney, J. (2013). Ecological validity in eye-tracking: An empirical study. Studies in Second Language Acquisition, 35, 389-415. doi:10.1017/s0272263112000927
64. Streiner, D. L. (2003). Unicorns do exist: A tutorial on "Proving" the null hypothesis. Canadian Journal of Psychiatry, 48, 756-761.
65. Tinker, M. A. (1936). Reliability and validity of eye-movement measures of reading. Journal of Experimental Psychology, 19, 732-746.
66. Van Assche, E., Drieghe, D., Duyck, W., Welvaert, M., & Hartsuiker, R. J. (2011). The influence of semantic constraints on bilingual word recognition during sentence reading. Journal of Memory and Language, 64, 88-107. doi:10.1016/j.jml.2010.08.006
67. VanPatten, B. (1990). Attending to form and content in the input: An experiment in consciousness. Studies in Second Language Acquisition, 12, 287-301. doi:10.1017/S0272263100009177
68. Wellek, S. (2010). Testing statistical hypotheses of equivalence and noninferiority (2nd ed.). Boca Raton, FL: CRC Press.