Measures for corpus similarity and homogeneity

Proceedings of the 3rd Conference on Empirical Methods in Natural Language Processing, EMNLP 1998

Volumn , Issue , 1998, Pages 46-52

  Kilgarriff, Adam ^a   Rose, Tony ^b  

^a UNIVERSITY OF BRIGHTON   (United Kingdom)

^b Canon Research Centre Europe

Author keywords

[No Author keywords available]

Indexed keywords

INFORMATION THEORY;

INFORMATION THEORETIC MEASURE; OPERATIONAL DEFINITION; SIMILARITY JUDGEMENTS; WORD FREQUENCIES;

NATURAL LANGUAGE PROCESSING SYSTEMS;

EID: 84864070583     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (17)

1. Biber, Douglas. 1988. Variation across speech and writing. Cambridge University Press.
2. Biber, Douglas. 1993. Using register-diversified corpora for genera] language studies. Computational Linguistics, 19(2):219 242.
3. Biber, Douglas. 1995. Dimensions in Register Variation. Cambridge University Press.
4. Chamiak Eugene. 1993. Statistical Language Learning. MIT Press, Cambridge, Mass.
5. Church, Kenneth W. and Robert L. Mercer: 1993. In troduetion to the special issue on computationallinguistics using large corpora. Computational Linquistics, 19(1):1-24.
6. Dunning., Ted. 1993. Accurate methods for the statistics of surprise and coincidence. Computational Linguistics, 19(1): 61-74.
7. Johansson, Stig and Knut Hofland, editors. 1989. Frequency Analysis of English vocabulary and grammar, based on the LOB corpus. Clarendon, Oxford.
8. Kilgarriff, Adam. 1996. Which words are particularly characteristic of a text? a survey of statistical approaches. In Language Engineering for Document Analysis and Recognition, pages 33-40, Brighton, England, April. AISB Workshop Series.
9. Kilgarriff, Adam. 1997. Using word frequency lists to measure corpus homogeneity and similarity between corpora. In Proceedings, ACL SIGDAT workshop on very large corpora, pages 231-245, Beijing and Hong Kong, August.
10. Mostelier, Frederick and David L. Wallace. 1964. Applied Bayesian and Classical Inference - The Case of The Federalist Papers. Springer Series in Safeties, Springer-Verlag.
11. Rayson, Paul. Geoffrey Leech, and Mary Hodges. 1997. jSocial differentiation in the use of English vocabulary: some analysis of the conversational component of the British National Corpus. International Journal of Corpus Linguistics, 2(1):133-152.
12. Rose, Tony, Nicholas Haddock, and Roger Tucker. 1997, The effects of corpus size and homogeneity on language model quality. In Proceedings, ACL SIGDAT workshop on very large corpora, pages 178-191, Beijing and Hong Kong, August.
13. Rosenfeld, Ronald. 1995. The CMU Statistical Language Modelling Toolkit and its use in the 1994 ARPA CSR. Evaluation. In Proc. Spoken Language Technology Workshop, Austin, Texas.
14. Roukos, Salim, 1996. Language Representation, chapter 1.6. National Science Foundation and European Commission, www. cse. ogi/CSLU/HLTsurvey. html.
15. Saltan, Gerard. 1989. Automatic Text Processing. Addison-Wesley.
16. Sekine, SatshL 1997. The domain dependence of parsing. In Proc. Fifth Conference on Applied Natural Language Processing, pages 96-102, Washington DC, April. ACL.
17. Ueberla, Joerg. 1997. Towards an improved performance measure for language models. Technical Report DERA/CIS/CIS5/TR97426, DERA, cmp-lg/9711009.