Significance testing of word frequencies in corpora

Digital Scholarship in the Humanities

Volumn 31, Issue 2, 2016, Pages 374-397

  Lijffijt, Jefrey ^a,b   Nevalainen, Terttu ^c   Säily, Tanja ^c   Papapetrou, Panagiotis ^a,d   Puolamäki, Kai ^a,e   Mannila, Heikki ^a  

^a AALTO UNIVERSITY   (Finland)

^b UNIVERSITY OF BRISTOL   (United Kingdom)

^c UNIVERSITY OF HELSINKI   (Finland)

^d STOCKHOLM UNIVERSITY   (Sweden)

^e FINNISH INSTITUTE OF OCCUPATIONAL HEALTH   (Finland)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84974696001     PISSN: 20557671     EISSN: 2055768X     Source Type: Journal    
DOI: 10.1093/llc/fqu064     Document Type: Article

References (46)

1. Altmann, E. G., Pierrehumbert, J. B., and Motter, A. E. (2009). Beyond word frequency: bursts, lulls, and scaling in the temporal distributions of words. PLoS One, 4(11): e7678.
2. Argamon, S., Koppel, M., Fine, J., and Shimoni, A. R. (2003). Gender, genre, and writing style in formal written texts. Text, 23(3): 321-46.
3. Beaujean, F., Caldwell, A., Kollár, D., and Kröninger, K. (2011). P-values for model evaluation. Physical Review D, 83(1): 012004.
4. Bell, A. (1984). Language style as audience design. Language in Society, 13: 145-204.
5. Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, 57(1): 289-300.
6. Berry, G. and Armitage, P. (1995). Mid-P confidence intervals: a brief review. The Statistician, 44(4): 417-23.
7. Biber, D. (1995). Dimensions of Register Variation: A Cross-linguistic Comparison. Cambridge: Cambridge University Press.
8. Biber, D. and Burges, J. (2000). Historical change in the language use of women and men: gender differences in dramatic dialogue. Journal of English Linguistics, 28(1): 21-37.
9. Blocker, C., Conway, J., Demortier, L., Heinrich, J., Junk, T., Lyons, L., and Punzig, G. (2006). Simple Facts about p-values. Technical Report CDF/MEMO/STATISTICS/PUBLIC/8023, Laboratory of Experimental High Energy Physics, The Rockefeller University.
10. BNC - The British National Corpus, version 3 (BNC XML Edition) (2007). Distributed by Oxford University Computing Services on behalf of the BNC Consortium. http://www.natcorp.ox.ac.uk/(accessed 26 November 2012).
11. Burnard, L. (2007). Reference Guide for the British National Corpus (XML Edition). Published for the British National Corpus Consortium by the Research Technologies Service at Oxford University Computing Services. http://www.natcorp.ox.ac.uk/docs/URG/ (accessed 26 November 2012).
12. Dudoit, S., Shaffer, J. P., and Boldrick, J. C. (2003). Multiple hypothesis testing in microarray experiments. Statistical Science, 18(1): 71-103.
13. Dunning, T. (1993). Accurate methods for the statistics of surprise and coincidence. Computational Linguistics, 19: 61-74.
14. Efron, B. and Tibshirani, R. J. (1994). An Introduction to the Bootstrap. New York: Chapman and Hall/CRC.
15. Evert, S. (2005). The Statistics of Word Cooccurrences: Word Pairs and Collocations. Dissertation, Institut für Maschinelle Sprachverarbeitung, University of Stuttgart.
16. Good, P. (2005). Permutation, Parametric, and Bootstrap Tests of Hypotheses. 3rd edn. New York/Heidelberg: Springer.
17. Gries, S. Th. (2005). Null-hypothesis significance testing of word frequencies: a follow-up on Kilgarriff. Corpus Linguistics and Linguistic Theory, 1(2): 277-94.
18. Gries, S. Th. (2008). Dispersions and adjusted frequencies in corpora. International Journal of Corpus Linguistics, 13(4): 403-37.
19. Herring, S. C. and Paolillo, J. C. (2006). Gender and genre variation in weblogs. Journal of Sociolinguistics, 10(4): 439-59.
20. Hinneburg, A., Mannila, H., Kaislaniemi, S., Nevalainen, T., and Raumolin-Brunberg, H. (2007). How to handle small samples: bootstrap and Bayesian methods in the analysis of linguistic change. Literary and Linguistic Computing, 22(2): 137-50.
21. Hoffmann, S., Evert, S., Smith, N., Lee, D., and Berglund Prytz, Y. (2008). Corpus Linguistics with BNCweb-a Practical Guide. Frankfurt am Main: Peter Lang.
22. Kilgarriff, A. (2001). Comparing corpora. International Journal of Corpus Linguistics, 6(1): 13-7.
23. Kilgarriff, A. (2005). Language is never, ever, ever, random. Corpus Linguistics and Linguistic Theory, 1(2): 263-76.
24. L'Ecuyer, P. and Simard, R. (2007). TestU01: a C library for empirical testing of random number generators. ACM Transactions on Mathematical Software, 33(4): 22.
25. Lee, D. Y. W. (2001). Genres, registers, text types, domains and styles: clarifying the concepts and navigating a path through the BNC jungle. Language Learning and Technology, 5(3): 37-72.
26. Lijffijt, J. (2012). Bootstrap test for R and Matlab. http://users.ics.aalto.fi/lijffijt/bootstraptest/(accessed 26 November 2012).
27. Lijffijt, J. (2013). A fast and simple method for mining subsequences with surprising event counts. In Blockeel, H., Kersting, K., Nijssen, S., and Železný, F. (eds), Proceedings of ECML-PKDD 2013-Part I. Berlin: Springer-Verlag, pp. 385-400.
28. Lijffijt, J. and Gries, S. Th. (2012). Correction to Stefan Th. Gries' "Dispersions and adjusted frequencies in corpora". International Journal of Corpus Linguistics, 17(1): 147-9.
29. Lijffijt, J., Papapetrou, P., Puolamäki, K., and Mannila, H. (2011). Analyzing word frequencies in large text corpora using inter-arrival times and boot-strapping. In Gunopulos, D., Hofmann, T., Malerba, D., and Vazirgiannis, M. (eds), Proceedings of ECML-PKDD 2011-Part II. Berlin: Springer-Verlag, pp. 341-57.
30. Lijffijt, J., Säily, T., and Nevalainen, T. (2012). CEECing the baseline: lexical stability and significant change in a historical corpus. In Tyrkkö, J., Kilpiö, M., Nevalainen, T., and Rissanen, M. (eds), Outposts of Historical Corpus Linguistics: From the Helsinki Corpus to a Proliferation of Resources. Studies in Variation, Contacts and Change in English, Vol. 10. Helsinki: VARIENG. http://www.helsinki.fi/varieng/journal/volumes/10/lijffijt_saily_nevalainen/ (accessed 26 November 2012).
31. Mann, H. B. and Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. Annals of Mathematical Statistics, 18(1): 50-60.
32. Massey, F. (1951). The Kolmogorov-Smirnov test for goodness of fit. Journal of the American Statistical Association, 46(253): 68-78.
33. Newman, M. L., Groom, C. J., Handelman, L. D., and Pennebaker, J. W. (2008). Gender differences in language use: an analysis of 14,000 text samples. Discourse Processes, 45: 211-36.
34. North, B. V., Curtis, D., and Sham, P. C. (2002). A note on the calculation of empirical p-values from Monte Carlo procedures. The American Journal of Human Genetics, 71(2): 439-41.
35. Oakes, M. P. and Farrow, M. (2007). Use of the chisquared test to examine vocabulary differences in English-language corpora representing seven different countries. Literary and Linguistic Computing, 22(1): 85-100.
36. Paquot, M. and Bestgen, Y. (2009). Distinctive words in academic writing: a comparison of three statistical tests for keyword extraction. In Jucker, A., Schreier, D., and Hundt, M. (eds), Corpora: Pragmatics and Discourse. Amsterdam: Rodopi, pp. 247-69.
37. Rayson, P. (2008). From key words to key semantic domains. International Journal of Corpus Linguistics, 13(4): 519-49.
38. Rayson, P., Berridge, D., and Francis, B. (2004). Extending the Cochran rule for the comparison of word frequencies between corpora. In Purnelle, G., Fairon, C., and Dister, A. (eds), Le poids des mots: Proceedings of the 7th International Conference on Statistical Analysis of Textual Data (JADT 2004). Louvain-la-Neuve: Presses universitaires de Louvain, pp. 926-36.
39. Rayson, P. and Garside, R. (2000). Comparing corpora using frequency profiling. In Kilgarriff, A. and Berber Sardinha, T. (eds), Proceedings of the Workshop on Comparing Corpora. Stroudsburg: Association for Computational Linguistics, pp. 16.
40. Rayson, P., Leech, G., and Hodges, M. (1997). Social differentiation in the use of English vocabulary: some analyses of the conversational component of the British National Corpus. International Journal of Corpus Linguistics, 2(1): 133-52.
41. Savický, P. and Hlaváčová, J. (2002). Measures of word commonness. Journal of Quantitative Linguistics, 9(3): 215-31.
42. Schweder, T. and Spjøtvoll, E. (1982). Plots of p-values to evaluate many tests simultaneously. Biometrika, 69(3): 493-502.
43. Scott, M. (2012). WordSmith Tools, version 6. Liverpool: Lexical Analysis Software.
44. Shaffer, J. P. (1995). Multiple hypothesis testing. Annual Review of Psychology, 46: 561-84.
45. Welch, B. L. (1947). The generalization of 'Student's' problem when several different population variances are involved. Biometrika, 34(12): 28-35.
46. Wilcoxon, F. (1945). Individual comparisons by ranking methods. Biometrics Bulletin, 1(6): 80-3.