Compressed Web indexes

WWW'09 - Proceedings of the 18th International World Wide Web Conference

Volumn , Issue , 2009, Pages 451-460

  Chierichetti, Flavio ^a   Kumar, Ravi ^b   Raghavan, Prabhakar ^b  

^a SAPIENZA UNIVERSITY OF ROME   (Italy)

^b YAHOO RESEARCH   (United States)

Author keywords

Compression; Double pareto; Index size; Power law

Indexed keywords

ARBITRARY DISTRIBUTION; DOCUMENT COLLECTION; DOUBLE-PARETO; INDEX SIZE; POWER LAW; QUERY TERMS; SHARP BOUNDS; SPACE REQUIREMENTS; TEXT DOCUMENT; THEORETICAL PREDICTION; ZIPF'S LAW;

COMPACTION; COMPUTATIONAL LINGUISTICS; EXPERIMENTS; PROBABILITY DISTRIBUTIONS; SEARCH ENGINES;

WORLD WIDE WEB;

EID: 84856121275     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1526709.1526770     Document Type: Conference Paper

References (28)

1. T. M. Apostol. Introduction to Analytic Number Theory. Springer-Verlag, 1976.
2. R. Baeza-Yates and G. Navarro. Block-addressing indices for approximate text retrieval. Journal of the American Society for Information Science, 51(1):69-82, 2000.
3. R. Baeza-Yates and B. Ribeiro-Neto. Modern Information Retrieval. Addison Wesley, 1999.
4. A.-L. Barabasi. Linked: How Everything is Connected to Everything Else and What It Means. Penguin Group, 2003.
5. D. Bladford and G. Blelloch. Index compression through document reordering. In Proceedings of the Data Compression Conference, pages 342-351, 2002.
6. P. Boldi and S. Vigna. The Webgraph framework i: Compression techniques. In Proceedings of the 13th International Conference on World Wide Web, pages 595-602, 2004.
7. P. Boldi and S. Vigna. The Webgraph framework ii: Codes for the world-wide web. In Data Compression Conference, 2004.
8. A. Gelbukh and G. Sidorov. Zipf and Heaps laws' coefficients depend on language. In Proceedings of the 2nd International Conference on Computational Linguistics and Intelligent Text Processing, pages 332-335, 2001.
9. L. Q. Ha, E. I. Sicilia-Garcia, J. Ming, and F. J. Smith. Extension of Zipf's law to word and character n-grams for English and Chinese. Computational Linguistics and Chinese Language Processing, 8(1):77-102, 2003.
10. H. S. Heaps. Information Retrieval: Computational and Theoretical Aspects. Academic Press, New York, 1978.
11. R. Kumar, P. Raghavan, S. Rajagopalan, and A. Tomkins. Trawling the Web for emerging cyber-communities. Computer Networks, 31(11-16):1481-1493, 1999.
12. W. Li. Random texts exhibit Zipf's-law-like word frequency distribution. IEEE Transactions on Information Theory, 38(6):1842-1845, 1992.
13. B. Mandelbrot. An information theory of the statistical structure of language. In W. Jackson, editor, Communication Theory, pages 486-502. Academic Press, 1953.
14. C. D. Manning, P. Raghavan, and H. Schütze. Introduction to Information Retrieval. Cambridge University Press, 2008.
15. C. D. Manning and H. Schütze. Foundations of Statistical Natural Language Processing. MIT Press, Cambridge, MA, 1999.
16. M. Mitzenmacher. Dynamic models for file sizes and double Pareto distributions. Internet Mathematics, 1(3):305-333, 2003.
17. M. Molloy and B. Reed. Graph Coloring and the Probabilistic Method. Springer-Verlag, 2002.
18. M. Newman, A.-L. Barabasi, and D. J. Watts. The Structure and Dynamics of Networks. Princeton University Press, 2006.
19. W. J. Reed and M. Jorgensen. The double Pareto-lognormal distribution - A new parametric model for size distributions. Communications in Statistics: Theory and Methods, 33(8):1733-1753, 2004.
20. W.-Y. Shieh, T.-F. Chen, J. J.-J. Shann, and C.-P. Chung. Inverted file compression through document identifier reassignment. Information Processing and Management, 39(1):117-131, 2003.
21. F. Silvestri, R. Perego, and S. Orlando. Assigning document identifiers to enhance compressibility of web search indexes. In Proceedings of the Symposium on Applied Computing, pages 600-605, 2004.
22. H. A. Simon. On a class of skew distribution functions. Biometrika, 42:425-440, 1955.
23. D. C. van Leijenhorst and T. P. van der Weide. A formal derivation of Heap's law. Information Sciences, 170:263-272, 2005.
24. D. Watts. Six Degrees: The Science of a Connected Age. W. W. Norton, 2003.
25. H. E. Williams and J. Zobel. Searchable words on the web. International Journal on Digital Libraries, 5(2):99-105, 2005.
26. I. H. Witten and T. C. Bell. Source models for natural language text. International Journal Man-Machine Studies, 32(5):545-579, 1990.
27. I. H. Witten, A. Moffat, and T. C. Bell. Managing Gigabytes: Compressing and Indexing Documents and Images. Morgan Kaufmann, 1999.
28. G. K. Zipf. Human Behavior and the Principle of Least Effort. Addison-Wesley, Cambridge MA, 1949.