When indexing equals compression: Experiments with compressing suffix arrays and applications

ACM Transactions on Algorithms

Volumn 2, Issue 4, 2006, Pages 611-639

  Foschini, Luca ^a   Grossi, Roberto ^b   Gupta, Ankur ^c,e   Vitter, Jeffrey Scott ^d,f  

^a SCUOLA SUPERIORE SANT ANNA   (Italy)

^b UNIVERSITY OF PISA   (Italy)

^c DUKE UNIVERSITY   (United States)

^d PURDUE UNIVERSITY   (United States)

^e Duke University   (United States)

^f Purdue University   (United States)

Author keywords

Burrows Wheeler Transform; Entropy; Suffix array; Text indexing

Indexed keywords

BURROWS WHEELER TRANSFORM; COMPRESSING SUFFIX ARRAYS; STATE-OF-THE-ART COMPRESSION; TEXT INDEXING;

ALGORITHMS; ENTROPY; IMAGE CODING; INDEXING (MATERIALS WORKING); MATHEMATICAL TRANSFORMATIONS; TREES (MATHEMATICS);

DATA COMPRESSION;

EID: 33846576963     PISSN: 15496325     EISSN: 15496333     Source Type: Journal    
DOI: 10.1145/1198513.1198521     Document Type: Conference Paper

References (45)

1. ABOUELHODA, M. I. KURTZ, S., AND OHLEBUSCH, E. 2004. Replacing suffix trees with enhanced suffix arrays. J. Disc. Algor. 2, 1, 53-86.
2. ARIMURA, H., ASAKA, H., SAKAMOTO, H., AND ARIKAWA, S. 2001. Efficient discovery of proximity patterns with suffix arrays (extended abstract). In CPM: 12th Symposium on Combinatorial Pattern Matching.
3. BENDER, M. A., AND FARACH-COLTON, M. 2004. The level ancestor problem simplified. Theoret. Comput. Sci. 321, 1, 5-12.
4. BENTLEY, J., SLEATOR, D., TARJAN, R., AND WEI, V. 1986. A locally adaptive data compression scheme. Commun. ACM, 320-330.
5. BRODNIK, A., AND MUNRO, J. I. 1999. Membership in constant time and almost-minimum space. SIAM J. Comput. 28, 5 (Oct.), 1627-1640.
6. THE CANTERBURY CORPUS. 2001. http://corpus.canterbury.ac.nz.
7. HAZELLE, B., AND GUIBAS, L. J. 1986. Fractional cascading: I. A data structuring technique. Algorithmica 1, 2, 133-162.
8. EOROWICZ, S. 2002. Second step algorithms in the Burrows-Wheeler compression algorithm. Softw. Pract. Exper. 32, 99-111.
9. FENWICK, P. 1996. Punctured elias codes for variable-length coding of the integers. The University of Auckland, NZ. TR 137. ISSN 1173-3500.
10. FENWICK, P. 2002. Burrows-wheeler compression with variable-length integer codes. Softw. Pract. Exper. 32, 1307-1316.
11. FERRAGINA, P., GIANCARLO, R., MANZINI, G., AND SCIORTINO, M. 2005. Boosting textual compression in optimal linear time. J. ACM 52, 4, 688-713.
12. FERRAGINA, P., AND MANZINI, G. 2001. An experimental study of an opportunistic index. In Proceedings of the 12th Annual ACM-SIAM Symposium on Discrete Algorithms. ACM, New York, pp. 269-278.
13. FERRAGINA, P., AND MANZINI, G. 2005. Indexing compressed text. J. ACM 52, 4, 552-581.
14. FOSCHINI, L., GROSSI, R., GUPTA, A., AND VITTER, J. S. 2004. Fast compression with a static model in high-order entropy. In Proceedings of the IEEE Data Compression Conference (Snowbird, UT, Mar.)
15. GÖNNET, G. H., BAEZA-YATES, R. A., AND SNIDER, T. 1992. New indices for text: PAT trees and PAT arrays. In Information Retrieval: Data Structures and Algorithms, chap. 5. Prentice-Hall, Englewood Cliffs, NJ, pp. 66-82.
16. GROSSI, R., GUPTA, A., AND VITTER, J. S. 2003. High-order entropy-compressed text indexes. In Proceedings of the 14th Annual ACM-SIAM Symposium on Discrete Algorithms (Jan.), ACM, New York.
17. GROSSI, R., GUPTA, A., AND VITTER, J. S. 2004. When indexing equals compression: Experiments with compressing suffix arrays and applications. In Proceedings of the 15th Annual ACM-SIAM Symposium on Discrete Algorithms, ACM, New York.
18. GROSSI, R., AND VITTER, J. S. 2005. Compressed suffix arrays and suffix trees with applications to text indexing and string matching. SIAM J. Comput. 35, 2, 378-407.
19. GUSFIELD, D. 1997. Algorithms on Strings, Trees and Sequences: Computer Science and Computational Biology. Cambridge University Press, Cambridge, MA.
20. HON, W., LAM, T., TSE, W., WONG, C., AND YIU, S. 2004. Practical aspects of compressed suffix arrays and fm-index in searching dna sequences. In Proceedings of the 6th Workshop on Algorithm Engineering and Experiments (ALENEX).
21. HON, W.-K., SADAKANE, K., AND SUNG, W.-K. 2003. Breaking a time-and-space barrier in constructing full-text indices. In Proceedings of the 44th Annual IEEE Symposium on Foundation of Computer Science. IEEE Computer Society Press, Los Alamitos, CA pp. 251-260.
22. HOWARD, P. G. 1997. Interleaving entropy codes. In Sequences.
23. JACOBSON, G. 1989. Space-efficient static trees and graphs. In Proceedings of the 30th Annual IEEE Symposium on Foundations of Computer Science. IEEE Computer Society Press, Los Alamitos, CA, pp. 549-554.
24. KASAI, T., LEE, G., ARIMURA, H., ARIKAWA, S., AND PARK, K. 2001. Linear-time longest-common-prefix computation in suffix arrays and its applications. In Combinatorial Pattern Matching (CPM). 181-192.
25. KURTZ, S. 1999. Reducing the space requirement of suffix trees. Softw. Pract. Expert. 29, 13, 1149-1171.
26. I, M., AND VITANYI, P. 1997. An Introduction to Kolmogorov Complexity and Its Applications. Springer-Verlag, New York.
27. ANBER, U., AND MYERS, G. 1993. Suffix arrays: A new method for on-line string searches. SIAM J. Comput. 22, 5, 935-948.
28. CCRElGHT, E. M. 1976. A space-economical suffix tree construction algorithm. J. ACM 23, 2, 262-272.
29. OFFAT, A., NEAL, R. M., AND WITTEN, I. H. 1998. Arithmetic coding revisited. ACM Trans. Inf. Syst. (TOIS) 16, 3, 256-294.
30. UNRO, J. I., AND RAMAN, V. 1999. Succinct representation of balanced parentheses, static trees, and planar graphs. SIAM J. Comput. 31, 762-776.
31. UNRO, J. I., RAMAN, V., AND SRINIVASA RAO, S. S. 2001. Space efficient suffix trees. J. Algorithms 39, 205-222.
32. NAVARRO, G., AND MÄKINEN, V. 2006. Compressed full-text indexes. Tech. Rep. TR/DCC-2006-6, University of Chile.
33. NELSON, M. 2003. Run length encoding/RLE. http://www. datacompression.info/RLE.shtml.
34. OKI, M. 2003. http://www.infor.kanazawa-it.ac.jp/~ishii/lhaunix/ .
35. PAGH, R. 2001. Low redundancy in static dictionaries with constant query time. SIAM J. Comput. 31, 353-363.
36. RAMAN, R., RAMAN, V., AND RAO, S. S. 2002. Succinct indexable dictionaries with applications to encoding k-ary trees and multisets. In Proceedings of the ACM-SIAM Symposium on Discrete Algorithms. 233-242.
37. RAO, S. S. 2002. Time-space trade-offs for compressed suffix arrays. IPL 82, 6, 307-311.
38. RlSSANEN, J., AND LANGDON, G. G. 1979. Arithmetic coding. IBM J. Res. Devel. 23, 2 (Mar.), 149-162.
39. SADAKANE, K. 2002. Succinct representations of lcp information and improvements in the compressed suffix arrays. In Proceedings of the 13th Annual ACM-SIAM Symposium on Discrete Algorithms. ACM, New York.
40. SADAKANE, K. 2003. New text indexing functionalities of the compressed suffix arrays. J. Algorithms 48, 2, 294-313.
41. SCHINDLER, M. 1999. http://www.compressconsult.com/rangecoder.
42. SMITH, J. O., III. 2003. http://ccrma-www.stanford.edu/~jos/ mdft/Autocorrelation.html.
43. TREC. TIPSTER 3. 2000. http://trec.nist.gov/data/docs.eng.html.
44. WlRTH, A. I., AND MOFFAT, A. 2001. Can we do without ranks in burrows wheeler transform compression? In Data Compression Conference, pp. 419-428.
45. WITTEN, I. H., MOFFAT, A., AND BELL, T. C. 1999. Managing Gigabytes: Compressing and Indexing Documents and Images. Morgan-Kaufmann, Los Altos, CA.