Megastudies, crowdsourcing, and large datasets in psycholinguistics: An overview of recent developments

Quarterly Journal of Experimental Psychology

Volumn 68, Issue 8, 2015, Pages 1457-1468

  Keuleers, Emmanuel ^a   Balota, David A ^b  

^a GHENT UNIVERSITY   (Belgium)

^b WASHINGTON UNIVERSITY   (United States)

Author keywords

Corpora; Crowdsourcing; Distributional semantics; Language processing; Megastudies

Indexed keywords

CROWDSOURCING; HISTORY; HUMAN; LINGUISTICS; PROCEDURES; SEMANTICS; STATISTICS AND NUMERICAL DATA; TRENDS;

CROWDSOURCING; HISTORY, 20TH CENTURY; HISTORY, 21ST CENTURY; HUMANS; PSYCHOLINGUISTICS; SEMANTICS;

EID: 84936818172     PISSN: 17470218     EISSN: 17470226     Source Type: Journal    
DOI: 10.1080/17470218.2015.1051065     Document Type: Editorial

References (70)

1. J.S.Adelman,, & G.D.A.Brown, (2008). Modeling lexical decision: The form of frequency and diversity effects. Psychological Review, 115(1), 214–227. http://doi.org/10.1037/0033-295X.115.1.214 doi: 10.1037/0033-295X.115.1.214
2. J.S.Adelman,, G.D.A.Brown,, & J.F.Quesada, (2006). Contextual diversity, not word frequency, determines word-naming and lexical decision times. Psychological Science, 17(9), 814–823. http://doi.org/10.1111/j.1467-9280.2006.01787.x doi: 10.1111/j.1467-9280.2006.01787.x
3. J.S.Adelman,, R.L.Johnson,, S.F.McCormick,, M.McKague,, S.Kinoshita,, J.S.Bowers,, C.J.… Davis, (2014). A behavioral database for masked form priming. Behavior Research Methods, 46(4), 1052–1067. http://doi.org/10.3758/s13428-013-0442-y doi: 10.3758/s13428-013-0442-y
4. R.H.Baayen, (2010). A real experiment is a factorial experiment. The Mental Lexicon, 5(1), 149–157. doi:10.1075/ml.5.1.06baa
5. R.H.Baayen,, R.Piepenbrock,, & L.Gulikers, (1995). The CELEX lexical database (CD-ROM). Linguistic Data Consortium. Philadelphia, PA: University of Pennsylvania.
6. D.A.Balota,, M.J.Cortese,, S.D.Sergent-Marshall,, D.H.Spieler,, & M.Yap, (2004). Visual word recognition of single-syllable words. Journal of Experimental Psychology: General, 133(2), 283–316. http://doi.org/10.1037/0096-3445.133.2.283 doi: 10.1037/0096-3445.133.2.283
7. D.A.Balota,, M.Pilotti,, & M.J.Cortese, (2001). Subjective frequency estimates for 2,938 monosyllabic words. Memory & Cognition, 29, 639–647. doi:10.3758/BF03200465
8. D.A.Balota,, M.J.Yap,, K.A.Hutchison,, & M.J.Cortese, (2012). Megastudies: What do millions (or so) of trials tell us about lexical processing? In J.S.Adelman (Ed.), Visual word recognition volume 1: Models and methods (pp. 90–115). Hove, East Sussex: Psychology Press.
9. D.A.Balota,, M.J.Yap,, K.A.Hutchison,, M.J.Cortese,, B.Kessler,, B.Loftis,, … R.Treiman, (2007). The English lexicon project. Behavior Research Methods, 39(3), 445–459. doi:10.3758/BF03193014
10. Y.Bestgen,, & N.Vincze, (2012). Checking and bootstrapping lexical norms by means of word similarity indexes. Behavior Research Methods, 44(4), 998–1006. http://doi.org/10.3758/s13428-012-0195-z doi: 10.3758/s13428-012-0195-z
11. M.Brysbaert,, M.Buchmeier,, M.Conrad,, A.M.Jacobs,, J.Bölte,, & A.Böhl, (2011). The word frequency effect: A review of recent developments and implications for the choice of frequency estimates in German. Experimental Psychology, 58(5), 412–424. http://doi.org/10.1027/1618-3169/a000123 doi: 10.1027/1618-3169/a000123
12. M.Brysbaert,, & B.New, (2009). Moving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behavior Research Methods, 41(4), 977–990. http://doi.org/10.3758/BRM.41.4.977 doi: 10.3758/BRM.41.4.977
13. M.Brysbaert,, A.B.Warriner,, & V.Kuperman, (2013). Concreteness ratings for 40 thousand generally known English word lemmas. Behavior Research Methods, 1–8.
14. Q.Cai,, & M.Brysbaert, (2010). SUBTLEX-CH: Chinese word and character frequencies based on film subtitles. PLoS ONE, 5(6), e10729. http://doi.org/10.1371/journal.pone.0010729 doi: 10.1371/journal.pone.0010729
15. F.Chetail,, D.Balota,, R.Treiman,, & A.Content, (2015). What can megastudies tell us about the orthographic structure of English words? The Quarterly Journal of Experimental Psychology, 1–22. http://doi.org/10.1080/17470218.2014.963628
16. M.Coltheart, (1981). The MRC psycholinguistic database. The Quarterly Journal of Experimental Psychology Section A, 33(4), 497–505. http://doi.org/10.1080/14640748108400805 doi: 10.1080/14640748108400805
17. M.J.Cortese,, S.Hacker,, J.Schock,, & J.B.Santo, (2015). Is reading-aloud performance in megastudies systematically influenced by the list context? The Quarterly Journal of Experimental Psychology, 1–12. http://doi.org/10.1080/17470218.2014.974624
18. M.J.Cortese,, M.M.Khanna,, & S.Hacker, (2010). Recognition memory for 2,578 monosyllabic words. Memory, 18(6), 595–609. doi:10.1080/09658211.2010.493892
19. M.J.Cortese,, D.P.McCarty,, & J.Schock, (2015). A mega recognition memory study of 2897 disyllabic words. The Quarterly Journal of Experimental Psychology, 1–13. http://doi.org/10.1080/17470218.2014.945096
20. F.Cuetos Vega,, M.González Nosti,, A.Barbón Gutiérrez,, & M.Brysbaert, (2011). SUBTLEX-ESP: Spanish word frequencies based on film subtitles. DIALNET. Retrieved from http://recopila.uniovi.es/dspace/handle/123456789/10272
21. S.De Deyne,, S.Verheyen,, & G.Storms, (2015). The role of corpus size and syntax in deriving lexico-semantic representations for a wide range of concepts. The Quarterly Journal of Experimental Psychology, 1–22. http://doi.org/10.1080/17470218.2014.994098 doi: 10.1080/17470218.2014.994098
22. M.Dimitropoulou,, J.A.Duñabeitia,, A.Avilés,, J.Corral,, & M.Carreiras, (2010). Subtitle-based word frequencies as the best estimate of reading behavior: The case of Greek. Frontiers in Language Sciences, 1, 218. http://doi.org/10.3389/fpsyg.2010.00218
23. S.Dufau,, J.A.Duñabeitia,, C.Moret-Tatay,, A.McGonigal,, D.Peeters,, F.-X.Alario,, … J.Grainger, (2011). Smart phone, smart science: How the use of smartphones can revolutionize research in cognitive science. PLoS ONE, 6(9), e24974. http://doi.org/10.1371/journal.pone.0024974 doi: 10.1371/journal.pone.0024974
24. M.Ernestus,, & A.Cutler, (2015). BALDEY: A database of auditory lexical decisions. The Quarterly Journal of Experimental Psychology, 1–20. http://doi.org/10.1080/17470218.2014.984730 doi: 10.1080/17470218.2014.984730
25. L.Ferrand,, B.New,, M.Brysbaert,, E.Keuleers,, P.Bonin,, A.Méot,, … C.Pallier, (2010). The French lexicon project: Lexical decision data for 38,840 French words and 38,840 pseudowords. Behavior Research Methods, 42(2), 488–496. http://doi.org/10.3758/BRM.42.2.488 doi: 10.3758/BRM.42.2.488
26. K.I.Forster, (2000). The potential for experimenter bias effects in word recognition experiments. Memory & Cognition, 28(7), 1109–1115. doi:10.3758/BF03211812
27. C.H.Haagen, (1949). Synonymity, vividness, familiarity, and association value ratings of 400 pairs of common adjectives. Journal of Psychology, 27, 453–463. Retrieved from http://search.proquest.com/docview/1290585641/citation/F4D1C9100B804CD1PQ/3?accountid=11077 doi: 10.1080/00223980.1949.9917435
28. D.H.Howes,, & R.L.Solomon, (1951). Visual duration threshold as a function of word-probability. Journal of Experimental Psychology, 41(6), 401. doi:10.1037/h0056020
29. K.A.Hutchison,, D.A.Balota,, J.H.Neely,, M.J.Cortese,, E.R.Cohen-Shikora,, C.-S.Tse,, … E.Buchanan, (2013). The semantic priming project. Behavior Research Methods, 45(4), 1099–1114. http://doi.org/10.3758/s13428-012-0304-z doi: 10.3758/s13428-012-0304-z
30. E.Keuleers,, M.Brysbaert,, & B.New, (2010). SUBTLEX-NL: A new measure for Dutch word frequency based on film subtitles. Behavior Research Methods, 42(3), 643–650. http://doi.org/10.3758/BRM.42.3.643 doi: 10.3758/BRM.42.3.643
31. E.Keuleers,, K.Diependaele,, & M.Brysbaert, (2010). Practice effects in large-scale visual word recognition studies: A lexical decision study on 14,000 dutch mono- and disyllabic words and nonwords. Frontiers in Psychology, 1. http://doi.org/10.3389/fpsyg.2010.00174 doi: 10.3389/fpsyg.2010.00174
32. E.Keuleers,, P.Lacey,, K.Rastle,, & M.Brysbaert, (2012). The British lexicon project: Lexical decision data for 28,730 monosyllabic and disyllabic English words. Behavior Research Methods, 44(1), 287–304. http://doi.org/10.3758/s13428-011-0118-4 doi: 10.3758/s13428-011-0118-4
33. E.Keuleers,, M.Stevens,, P.Mandera,, & M.Brysbaert, (2015). Word knowledge in the crowd: Measuring vocabulary size and word prevalence in a massive online experiment. The Quarterly Journal of Experimental Psychology, 1–28. http://doi.org/10.1080/17470218.2015.1022560
34. H.Kučera,, & N.Francis, (1967). Computational analysis of present-day American English. Providence, RI: Brown University Press.
35. V.Kuperman, (2015). Virtual experiments in megastudies: A case study of language and emotion. The Quarterly Journal of Experimental Psychology, 1–18. http://doi.org/10.1080/17470218.2014.989865 doi: 10.1080/17470218.2014.989865
36. V.Kuperman,, H.Stadthagen-Gonzalez,, & M.Brysbaert, (2012). Age-of-acquisition ratings for 30,000 English words. Behavior Research Methods. http://doi.org/10.3758/s13428-012-0210-4
37. V.Kuperman,, & J.A.Van Dyke, (2013). Reassessing word frequency as a determinant of word recognition for skilled and unskilled readers. Journal of Experimental Psychology: Human Perception and Performance. http://doi.org/10.1037/a0030859
38. T.K.Landauer,, & S.T.Dumais, (1997). A solution to Plato's problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychological Review, 104(2), 211. doi:10.1037/0033-295X.104.2.211
39. P.Mandera,, E.Keuleers,, & M.Brysbaert, (2015). How useful are corpus-based methods for extrapolating psycholinguistic variables? The Quarterly Journal of Experimental Psychology, 1–20. http://doi.org/10.1080/17470218.2014.988735 doi: 10.1080/17470218.2014.988735
40. P.Mandera,, E.Keuleers,, Z.Wodniecka,, & M.Brysbaert, (2014). Subtlex-pl: Subtitle-based word frequency estimates for Polish. Behavior Research Methods. http://doi.org/10.3758/s13428-014-0489-4
41. M.Marelli,, S.Amenta,, & D.Crepaldi, (2015). Semantic transparency in free stems: The effect of orthography-semantics consistency on word recognition. The Quarterly Journal of Experimental Psychology, 1–13. http://doi.org/10.1080/17470218.2014.959709
42. W.Mason,, & S.Suri, (2011). Conducting behavioral research on Amazon's Mechanical Turk. Behavior Research Methods, 44(1), 1–23. http://doi.org/10.3758/s13428-011-0124-6 doi: 10.3758/s13428-011-0124-6
43. R.Munro,, S.Bethard,, V.Kuperman,, V.T.Lai,, R.Melnick,, C.Potts,, … H.Tily, (2010). Crowdsourcing and language studies: The new generation of linguistic data. In Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon's Mechanical Turk (pp. 122–130). Retrieved from http://dl.acm.org/citation.cfm?id=1866715
44. B.New,, M.Brysbaert,, J.Veronis,, & C.Pallier, (2007). The use of film subtitles to estimate word frequencies. Applied Psycholinguistics, 28(04), 661–677. http://doi.org/10.1017/S014271640707035X doi: 10.1017/S014271640707035X
45. H.C.Nusbaum,, D.B.Pisoni,, & C.K.Davis, (1984). Sizing up the Hoosier mental lexicon: Measuring the familiarity of 20,000 words. Research on Speech Perception Progress Report, 10(10), 357–376.
46. N.Oostdijk, (2002). The design of the spoken Dutch Corpus. In P.Peters, P.Collins, & A.Smith (Eds.), New frontiers of corpus research (pp. 105–112). Amsterdam: Rodopi.
47. A.Paivio,, J.C.Yuille,, & S.A.Madigan, (1968). Concreteness, imagery, and meaningfulness values for 925 nouns. Journal of Experimental Psychology, 76(1p2), 1–25. doi: 10.1037/h0025327
48. C.Perry,, J.C.Ziegler,, & M.Zorzi, (2010). Beyond single syllables: Large-scale modeling of reading aloud with the Connectionist Dual Process (CDP++) model. Cognitive Psychology, 61(2), 106–151. http://doi.org/10.1016/j.cogpsych.2010.04.001 doi: 10.1016/j.cogpsych.2010.04.001
49. M.Radeau,, P.Mousty,, & A.Content, (1990). Brulex. Une base de données lexicales informatisée pour le français écrit et parlé. L'année Psychologique, 90(4), 551–566. http://doi.org/10.3406/psy.1990.29428 doi: 10.3406/psy.1990.29428
50. G.Recchia,, & M.M.Louwerse, (2015). Reproducing affective norms with lexical co-occurrence statistics: Predicting valence, arousal, and dominance. The Quarterly Journal of Experimental Psychology, 1–15. http://doi.org/10.1080/17470218.2014.941296
51. T.Schnoebelen,, & V.Kuperman, (2010). Using Amazon Mechanical Turk for linguistic research. Psihologija, 43(4), 441–464. http://doi.org/10.2298/PSI1004441S doi: 10.2298/PSI1004441S
52. M.S.Seidenberg,, & G.S.Waters, (1989). Reading words aloud-a mega study. Retrieved from http://philpapers.org/rec/SEIRWA
53. D.E.Sibley,, C.T.Kello,, & M.S.Seidenberg, (2009). Error, error everywhere: A look at megastudies of word reading. In Proceedings of the 31st annual conference of the cognitive science society (pp. 1036–1041). Retrieved from http://cogmech.ucmerced.edu/pubs/SibleyETAL09-csproc.pdf
54. A.P.Soares,, J.Machado,, A.Costa,, Á.Iriarte,, A.Simões,, J.J.de Almeida,, … M.Perea, (2014). On the advantages of word-frequency and contextual diversity measures extracted from subtitles: The case of Portuguese. The Quarterly Journal of Experimental Psychology, 1–41. http://doi.org/10.1080/17470218.2014.964271
55. D.H.Spieler,, & D.A.Balota, (1997). Bringing computational models of word naming down to the item level. Psychological Science, 8, 411–416. doi: 10.1111/j.1467-9280.1997.tb00453.x
56. D.H.Spieler,, & D.A.Balota, (2000). Factors influencing word naming in younger and older adults. Psychology and Aging, 15(2), 225–231. http://doi.org/10.1037//0882-7974.15.2.225 doi: 10.1037/0882-7974.15.2.225
57. W.P.Sze,, S.J.Rickard Liow,, & M.J.Yap, (2014). The Chinese lexicon project: A repository of lexical decision behavioral responses for 2,500 Chinese characters. Behavior Research Methods, 46(1), 263–273. http://doi.org/10.3758/s13428-013-0355-9 doi: 10.3758/s13428-013-0355-9
58. W.P.Sze,, M.J.Yap,, & S.J.Rickard Liow, (2015). The role of lexical variables in the visual recognition of Chinese characters: A megastudy analysis. The Quarterly Journal of Experimental Psychology, 1–30. http://doi.org/10.1080/17470218.2014.985234 doi: 10.1080/17470218.2014.985234
59. A.Szekely,, T.Jacobsen,, S.D'Amico,, A.Devescovi,, E.Andonova,, D.Herron,, … et al. (2004). A new on-line resource for psycholinguistic studies. Journal of Memory and Language, 51(2), 247–250. doi:10.1016/j.jml.2004.03.002
60. A.Taikh,, I.S.Hargreaves,, M.J.Yap,, & P.M.Pexman, (2015). Semantic classification of pictures and words. The Quarterly Journal of Experimental Psychology, 1–17. http://doi.org/10.1080/17470218.2014.975728
61. E.L.Thorndike,, & I.Lorge, (1944). The teacher's word book of 30,000 words. New York: Teachers College Press.
62. R.Treiman,, J.Mullennix,, R.Bijeljac-Babic,, & E.D.Richmond-Welty, (1995). The special role of rimes in the description, use, and acquisition of English orthography. Journal of Experimental Psychology: General, 124(2), 107–136. doi:10.1037/0096-3445.124.2.107
63. W.J.B.Van Heuven,, P.Mandera,, E.Keuleers,, & M.Brysbaert, (2014). SUBTLEX-UK: A new and improved word frequency database for British English. The Quarterly Journal of Experimental Psychology, 67(6), 1176–1190. http://doi.org/10.1080/17470218.2013.850521 doi: 10.1080/17470218.2013.850521
64. A.B.Warriner,, V.Kuperman,, & M.Brysbaert, (2013). Norms of valence, arousal, and dominance for 13,915 English lemmas. Behavior Research Methods, 45(4), 1191–1207. http://doi.org/10.3758/s13428-012-0314-x doi: 10.3758/s13428-012-0314-x
65. C.Westbury,, J.Keith,, B.B.Briesemeister,, M.J.Hofmann,, & A.M.Jacobs, (2015). Avoid violence, rioting, and outrage; approach celebration, delight, and strength: Using large text corpora to compute valence, arousal, and the basic emotions. The Quarterly Journal of Experimental Psychology, 1–24. http://doi.org/10.1080/17470218.2014.970204
66. S.Wood, (2006). Generalized additive models: An introduction with R. Boca Raton, FL: CRC press.
67. M.J.Yap,, D.A.Balota,, D.E.Sibley,, & R.Ratcliff, (2012). Individual differences in visual word recognition: Insights from the English Lexicon Project. Journal of Experimental Psychology: Human Perception and Performance, 38(1), 53–79. http://doi.org/10.1037/a0024177
68. M.J.Yap,, S.J.R.Liow,, S.B.Jalil,, & S.S.B.Faizal, (2010). The Malay Lexicon Project: A database of lexical statistics for 9,592 words. Behavior Research Methods, 42(4), 992–1003. http://doi.org/10.3758/BRM.42.4.992 doi: 10.3758/BRM.42.4.992
69. T.Yarkoni,, D.Balota,, & M.Yap, (2008). Moving beyond Coltheart's N: A new measure of orthographic similarity. Psychonomic Bulletin & Review, 15(5), 971–979. http://doi.org/10.3758/PBR.15.5.971 doi: 10.3758/PBR.15.5.971
70. J.C.Ziegler,, L.H.Tan,, C.Perry,, & M.Montant, (2000). Phonology matters: The phonological frequency effect in written Chinese. Psychological Science, 11(3), 234–238. doi:10.1111/1467-9280.00247