On compressing the textual web

WSDM 2010 - Proceedings of the 3rd ACM International Conference on Web Search and Data Mining

Volumn , Issue , 2010, Pages 391-400

  Ferragina, Paolo ^a   Manzini, Giovanni ^b  

^a UNIVERSITY OF PISA   (Italy)

^b UNIVERSITY OF EASTERN PIEDMONT   (Italy)

Author keywords

Burrows wheeler transform; Compressed (self )indexes; Lossless data compression; Text compression

Indexed keywords

BURROWS WHEELER TRANSFORM; COMPRESSION ALGORITHMS; DATA SETS; EXPERIMENTAL COMPARISON; FAST SCANNING; HUFFMAN; INVERTED LIST; KNOW-HOW; LARGE PARTS; LOSSLESS DATA COMPRESSION; MOVE-TO-FRONT; NEW TECHNOLOGIES; SOFTWARE DEVELOPER; STORAGE SOLUTIONS; TEXT COMPRESSIONS; WEB APPLICATION; WEB COLLECTIONS; WEB PAGE; WEB STRUCTURES;

DATA COMPRESSION; INFORMATION RETRIEVAL; SEARCH ENGINES;

WORLD WIDE WEB;

EID: 77950946921     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1718487.1718536     Document Type: Conference Paper

References (59)

1. Apache Foundation. Hadoop. http://hadoop.apache.org/.
2. R. Baeza-Yates and R. Ramakrishnan. Data challenges at Yahoo EDBT, 652-655, 2008.
3. R. Baeza-Yates and B. Ribeiro-Neto. Modern Information Retrieval, Addison Wesley, 1999.
4. L. Barroso, J. Dean, and U. Hölzle. Web search for a planet: The google cluster architecture. IEEE Micro, 23(2), 2003.
5. R. Bayardo, Y. Ma, and R. Srikant. Scaling up all pairs similarity search. WWW, 131-140, 2007.
6. J. Bentley and M. McIlroy. Data compression with long repeated strings. Information Sciences, 135(1-2):1-11, 2001.
7. D. Blandford and G. Blelloch. Dictionaries using variable-length keys and data, with applications. ACM-SIAM SODA, 1-10, 2005.
8. P. Boldi, B. Codenotti, M. Santini, and S. Vigna. Ubicrawler: A scalable fully distributed web crawler. Software: Practice & Experience, 34(8):711-726, 2004.
9. P. Boldi, M. Santini, and S. Vigna. Permuting Web Graphs. WAW, 116-126, 2009.
10. P. Boldi and S. Vigna. The webgraph framework I: compression techniques. WWW, 595-602, 2004.
11. P. Boldi and S. Vigna. MG4J at TREC 2005. http://mg4j.dsi.unimi.it/.
12. N. Brisaboa, A. Fariña, S. Ladra, and G. Navarro. Reorganizing compressed text. ACM SIGIR, 139-146, 2008.
13. N. Brisaboa, A. Fariña, G. Navarro, A. Places, and E. Rodríguez. Self-indexing natural language. SPIRE, LNCS 5280, 2008.
14. A. Broder. Identifying and filtering near-duplicate documents. CPM, LNCS 1848, 2000.
15. A. Broder, S. Glassman, M. Manasse, and G. Zweig. Syntactic clustering of the web. Computer Networks, 29(8-13), 1997.
16. G. Buehrer and K. Chellapilla. A scalable pattern mining approach to web graph compression with communities. WSDM, 95-106, 2008.
17. S. Büttcher and C. Clarke. Efficiency vs. Effectiveness in terabyte-scale information retrieval. TREC, 2005.
18. S. Büttcher and C. Clarke. Index compression is good, especially for random access. CIKM, 761-770, 2007.
19. M. Chan and T. Woo. Cache-based compaction: A new technique for optimizing web-transfer. INFOCOM, 1999.
20. F. Chang, J. Dean, S. Ghemawat, W. Hsieh, D. Wallach, M. Burrows, T. Chandra, A. Fikes, and R. Gruber. Bigtable: A distributed storage system for structured data. ACM Trans. Comput. Syst., 26(2), 2008.
21. F. Chierichetti, R. Kumar, S. Lattanzi, M. Mitzenmacher, A. Panconesi, P. Raghavan. On compressing social networks. KDD, 219-228, 2009.
22. F. Claude and G. Navarro. A fast and compact web graph representation. SPIRE, LNCS 4726, 118-129, 2007.
23. D. Cutting. Apache Lucene. http://lucene.apache.org/.
24. T. S. D. Trendafilov, N. Memon. Compressing file collections with a TSP-based approach. TR-CIS-2004-2102, Polytechnic University, 2004.
25. M. Datar, N. Immorlica, P. Indyk, and V. Mirrokni. Locality-sensitive hashing scheme based on p-stable distributions. ACM SoCG, 253-262, 2004.
26. J. Dean and S. Ghemawat. MapReduce: Simplified data processing on large clusters. OSDI, 2004.
27. F. Douglis and A. Iyengar. Application-specific delta-encoding via resemblance detection. USENIX, 113-126, 2003.
28. P. Ferragina, T. Gagie, and G. Manzini. Lightweight Data Indexing and Compression in External Memory. arXiv:0909.4341, 2009.
29. P. Ferragina, R. Giancarlo, and G. Manzini. The engineering of a compression boosting library: Theory vs practice in BWT compression. ESA, LNCS 4168, 756-767, 2006.
30. P. Ferragina, R. Giancarlo, G. Manzini, and M. Sciortino. Boosting textual compressionoptimal linear time. Journal of the ACM, 52:688-713, 2005.
31. P. Ferragina, R. González, G. Navarro, and R. Venturini. Compressed text indexes: From theory to practice. ACM Journal of Experimental Algorithmics, 13, 2008.
32. P. Ferragina and G. Navarro. Pizza&Chili corpus home page. http://pizzachili.di.unipi.it/.
33. P. Ferragina, I. Nitto, and R. Venturini. On optimally partitioning a text to improve its compression. ESA, LNCS 5757, 420-431, 2009.
34. P. Ferragina, I. Nitto, and R. Venturini. On the bit-complexity of Lempel-Ziv compression. ACM-SIAM SODA, 2009.
35. P. Ferragina and R. Venturini. Compressed permuterm index. ACM SIGIR, 535-542, 2007.
36. D. Geer. Reducing the storage burden via data deduplication. Computer, 41(12):15-17, 2008.
37. O.A. Hamid, B. Behzadi, S. Christoph, and M.R. Henzinger. Detecting the origin of text segments efficiently. WWW, 61-70, 2009.
38. T. Haveliwala, A. Gionis, and P. Indyk. Scalable techniques for clustering the web. WebDB, 129-134, 2000.
39. J. Kieffer and E.-H. Yang. Grammar-based codes: A new class of universal lossless source codes. IEEE Trans. Info. Theory, 46(3):737-754, 2000.
40. P. Kulkarni, F. Douglis, J. LaVoie, and J. Tracey. Redundancy elimination within large collections of files. USENIX, 2004.
41. M. Mahoney. Large text compression benchmark. http://www.cs.fit.edu/ mmahoney/compression/text.html.
42. U. Manber. Finding similar files in a large file system. USENIX, 1-10, 1994.
43. F. Menczer. Lexical and semantic clustering by web links. Journal of the American Society for Information Science and Technology, 55(14):1261-1269, 2004.
44. J. Mogul, F. Douglis, A. Feldmann, and B. Krishnamurthy. Potential benefits of delta encoding and data compression for HTTP. ACM SIGCOMM, 181-194, 1997.
45. G. Navarro and V. Mäkinen. Compressed full-text indexes. ACM Computing Surveys, 39(1), 2007.
46. Z. Ouyang, N. Memon, T. Suel, and D. Trendafilov. Cluster-based delta compression of a collection of files. WISE, 257-268, 2002.
47. M. Sahami and T. Heilman. A web-based kernel function for measuring the similarity of short text snippets. WWW, 377-386, 2006.
48. S. Schleimer, D. Wilkerson, and A. Aiken. Winnowing: Local algorithms for document fingerprinting. SIGMOD, 76-85, 2003.
49. D. Shkarin. PPM: One step to practicality. IEEE Data Compression Conference, 202-211, 2002.
50. F. Silvestri. Sorting out the document identifier assignment problem. ECIR, LNCS 4425, 101-112, 2007.
51. T. Suel and N. Memon. Lossless Compression Handbook, chapter "Algorithms for delta compression and remote file synchronization", Academic Press, 2002.
52. T. Suel, P. Noel, and D. Trendafilov. Improved file synchronization techniques for maintaining large replicated collections over slow networks. IEEE ICDE, 153-164, 2004.
53. D. Teodosiu, N. Bjorner, J. Porkka, M. Manasse, and Y. Gurevich. Optimizing File Replication over Limited-Bandwidth Networks using Remote Differential Compression. Microsoft Research TR-2006-2157, 2006.
54. A. Turpin, Y. Tsegay, D. Hawking, and H. Williams. Fast generation of result snippets in web search. ACM SIGIR, 127-134, 2007. (Pubitemid 350164953)
55. S. Vigna. Personal Communication.
56. I. H. Witten, A. Moffat, and T. C. Bell. Managing Gigabytes: Compressing and Indexing Documents and Images, Morgan Kaufmann Publishers, 1999.
57. H. Yan, S. Ding, and T. Suel. Compressing term positions in web indexes. ACM SIGIR, 2009.
58. H. Yan, S. Ding, and T. Suel. Inverted index compression and query processing with optimized document ordering. WWW, pages 401-410, 2009.
59. J. Zhang, X. Long, and T. Suel. Performance of compressed inverted list caching in search engines. WWW, 387-396, 2008.