Dealing with zero word frequencies: A review of the existing rules of thumb and a suggestion for an evidence-based choice

Behavior Research Methods

Volumn 45, Issue 2, 2013, Pages 422-430

  Brysbaert, Marc ^a   Diependaele, Kevin ^a  

^a GHENT UNIVERSITY   (Belgium)

Author keywords

Good Turing algorithm; Laplace transformation; Word frequency; Zero frequency

Indexed keywords

ALGORITHM; ARTICLE; BEHAVIORAL RESEARCH; DECISION MAKING; EVIDENCE BASED MEDICINE; HUMAN; LANGUAGE; LINGUISTICS; METHODOLOGY; RECOGNITION;

ALGORITHMS; BEHAVIORAL RESEARCH; CHOICE BEHAVIOR; EVIDENCE-BASED MEDICINE; HUMANS; LANGUAGE; RECOGNITION (PSYCHOLOGY); VOCABULARY;

EID: 84878231848     PISSN: 1554351X     EISSN: 15543528     Source Type: Journal    
DOI: 10.3758/s13428-012-0270-5     Document Type: Article

References (16)

1. Baayen, R. H. (2001). Word frequency distributions. Dordrecht, The Netherlands: Kluwer Academic Publishers.
2. Baayen, R. H., Piepenbrock, R., & Gulikers, L. (1995). The CELEX lexical database (Release 2) [CD-ROM]. Philadelphia, PA: Linguistic Data Consortium, University of Pennsylvania.
3. Balota, D. A., Yap, M. J., Cortese, M. J., Hutchison, K. A., Kessler, B., Loftis, B., & Treiman, R. (2007). The English Lexicon Project. Behavior Research Methods, 39, 445-459. doi: 10. 3758/BF03193014.
4. Brysbaert, M., Buchmeier, M., Conrad, M., Jacobs, A. M., Bölte, J., & Böhl, A. (2011a). The word frequency effect: A review of recent developments and implications for the choice of frequency estimates in German. Experimental Psychology, 58, 412-424. doi: 10. 1027/1618-3169/a000123.
5. Brysbaert, M., Keuleers, E., & New, B. (2011b). Assessing the usefulness of Google Books' word frequencies for psycholinguistic research on word processing. Frontiers in Language Sciences, 2, 27. doi: 10. 3389/fpsyg. 2011. 00027.
6. Brysbaert, M., & New, B. (2009). Moving beyond Kučera and Francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for American English. Behavior Research Methods, 41, 977-990. doi: 10. 3758/BRM. 41. 4. 977.
7. Gale, W. A., & Sampson, G. (1995). Good-Turing frequency estimation without tears. Journal of Quantitative Linguistics, 2, 217-237.
8. Jurafsky, D., & Martin, J. H. (2009). Speech and language processing: An introduction to natural language processing, speech recognition, and computational linguistics (2nd ed.). New York, NY: Prentice Hall.
9. Keuleers, E., Diependaele, K., & Brysbaert, M. (2010). Practice effects in large-scale visual word recognition studies: A lexical decision study on 14, 000 Dutch mono- and disyllabic words and nonwords. Frontiers in Psychology, 1, 174. doi: 10. 3389/fpsyg. 2010. 00174.
10. Keuleers, E., Lacey, P., Rastle, K., & Brysbaert, M. (2012). The British Lexicon Project: Lexical decision data for 28, 730 monosyllabic and disyllabic English words. Behavior Research Methods, 44, 287-304. doi: 10. 3758/s13428-011-0118-4.
11. Kornai, A. (2002). How many words are there? Glottometrics, 4, 61-86.
12. Kučera, H., & Francis, W. N. (1967). Computational analysis of present day American English. Providence, RI: Brown University Press.
13. Kuperman, V., & van Dyke, J. A. (in press). Reassessing word frequency as a determinant of word recognition for skilled and unskilled readers. Journal of Experimental Psychology: Human Perception and Performance.
14. Michel, J. B., Shen, Y. K., Aiden, A. P., Veres, A., Gray, M. K., Google Books Team, . . . Aiden, E. L. (2011). Quantitative analysis of culture using millions of digitized books. Science, 331, 176-182. doi: 10. 1126/science. 1199644.
15. Rayner, K., & Pollatsek, A. (1989). The psychology of reading. Englewood Cliffs, NJ: Prentice Hall.
16. Wang, J. P. (2011). SPECIES: An R package for species richness estimation. Journal of Statistical Software, 40, 1-15.