Data compression for sequencing data

Algorithms for Molecular Biology

Volumn 8, Issue 1, 2013, Pages

  Deorowicz, Sebastian ^a   Grabowski, Szymon ^b  

^a SILESIAN UNIVERSITY OF TECHNOLOGY   (Poland)

^b LODZ UNIVERSITY OF TECHNOLOGY   (Poland)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84888042406     PISSN: None     EISSN: 17487188     Source Type: Journal    
DOI: 10.1186/1748-7188-8-25     Document Type: Review

References (99)

1. Metzker ML. Sequencing technologies-the next generation. Nat Rev Genet 2010, 11:31-46. 10.1038/nrg2626, 19997069.
2. Kahn SD. On the future of genomic data. Science 2011, 331:728-729. 10.1126/science.1197891, 21311016.
3. Roberts JP. Million veterans sequenced. Nat Biotechnol 2013, 31(6):470.
4. Hall N. After the gold rush. Genome Biol 2013, 14(5):115. 10.1186/gb-2013-14-5-115, 3663089, 23657273.
5. National Human Genome Research Institute, DNA Sequencing Costs. [http://www.genome.gov/sequencingcosts/] (accessed February 14, 2013).
6. Steinbiss S, Kurtz S. A new efficient data structure for storage and retrieval of multiple biosequences. IEEE/ACM Trans Comput Biol Bioinformatics 2012, 9(2):345-357.
7. Kodama Y, Shumway M, Leinonen R. The sequence read archive: explosive growth of sequencing data. Nucleic Acids Res 2012, 40(Database issue):54-56.
8. Cochrane G, Cook CE, Birney E. The future of DNA sequence archiving. GigaScience 2012, 1(1). article no. 2.
9. Giancarlo R, Scaturro D, Utro F. Textual data compression in computational biology: A synopsis. Bioinformatics 2009, 25(13):1575-1586. 10.1093/bioinformatics/btp117, 19251772.
10. Giancarlo R, Scaturro D, Utro F. Textual data compression in computational biology: Algorithmic techniques. Comput Sci Rev 2012, 6(1):1-25.
11. Vyverman M, De Baets B, Fack V, Dawyndt P. Prospects and limitations of full-text index structures in genome analysis. Nucleic Acids Res 2012, 40(15):6993-7015. 10.1093/nar/gks408, 3424560, 22584621.
12. Salomon D, Motta G. Handbook of data compression 2010, London: Springer.
13. Huffman D. A method for the construction of minimum-redundancy codes. Proceedings of the Institute of Radio Engineers 1952, 1098-1101.
14. Ziv J, Lempel A. A universal algorithm for sequential data compression. IEEE Trans Inf Theory 1977, IT-23:337-343.
15. Burrows M, Wheeler D. A block sorting lossless data compression algorithm. Technical Report 124, Digital Equipment Corporation 1994, http://www.hpl.hp.com/techreports/Compaq-DEC/SRC-RR-124.pdf.
16. Cock PJA, Fields CJ, Goto N, Heuer ML, Rive PM. The Sanger FASTQ file format for sequences with quality scores, and the Solexa/Illumina FASTQ variants. Nucleic Acids Res 2010, 38(6):1767-1771. 10.1093/nar/gkp1137, 2847217, 20015970.
17. Deorowicz S, Grabowski S. Compression of DNA sequence reads in FASTQ format. Bioinformatics 2011, 27(6):860-862. 10.1093/bioinformatics/btr014, 21252073.
18. Bhola V, Bopardikar AS, Narayanan R, Lee K, Ahn T. No-reference compression of genomic data stored in FASTQ format. Proceedings of the IEEE International Conference on Bioinformatics and Biomedicine 2011, 147-150. Atlanta, USA: IEEE Computer Society, Wu F-X, Zaki M, Morishita S, Pan Y, Wong S, Christianson A, Hu X.
19. Grassi E, Di Gregorio F, Molineris I. KungFQ: A Simple and Powerful Approach to Compress Fastq Files. IEEE/ACM Trans Comput Biol Bioinformatics 2012, 9(6):1837-1842.
20. Yanovsky V. ReCoil-an algorithm for compression of extremely large datasets of DNA data. Algo Mol Biol 2011, 6:23.
21. Cox AJ, Bauer MJ, Jakobi T, Rosone G. Large-scale compression of genomic sequence databases with the Burrows-Wheeler transform. Bioinformatics 2012, 28(11):1415-1419. 10.1093/bioinformatics/bts173, 22556365.
22. Hach F, Numanagić I, Alkan C, Sahinapl SC. SCALCE: boosting Sequence Compression Algorithms using Locally Consistent Encoding. Bioinformatics 2012, 28(23):3051-3057. 10.1093/bioinformatics/bts593, 23047557.
23. Miller JR, Koren S, Sutton G. Assembly algorithms for next-generation sequencing data. Genomics 2010, 95(6):315-327. 10.1016/j.ygeno.2010.03.001, 2874646, 20211242.
24. Wan R, Anh VN, Asai K. Transformations for the compression of FASTQ quality scores of next generation sequencing data. Bioinformatics 2011, 28(5):628-635.
25. Kozanitis C, Saunders C, Kruglyak S, Bafna V, Varghese G. Compressing genomic sequence fragments using SlimGene. J Comput Biol 2011, 18(3):401-413. 10.1089/cmb.2010.0253, 3123913, 21385043.
26. Ochoa I, Asnani H, Bharadia D, Chowdhury M, Weissman T, Yona G. QualComp: a new lossy compressor for quality scores based on rate distortion theory. BMC Bioinformatics 2013, 14:187. 10.1186/1471-2105-14-187, 3698011, 23758828.
27. Illumina Casava v. 1.8.2 Documentation. 2013, [http://support.illumina.com/sequencing/sequencing_software/casava.ilmn], Illumina.
28. Howison M. High-throughput compression of FASTQ data with SeqDB. IEEE/ACM Trans Comput Biol Bioinformatics 2013, 10(1):213-218.
29. Jones DC, Ruzzo WL, Peng X, Katze MG. Compression of next-generation sequencing reads aided by highly efficient de novo assembly. Nucleic Acids Res 2012, 40(22):e171. 10.1093/nar/gks754, 3526293, 22904078.
30. Bonfield JK, Mahoney MV. Compression of FASTQ and SAM format sequencing data. PLoS ONE 2013, 8(3):e59190. 10.1371/journal.pone.0059190, 3606433, 23533605.
31. Tembe W, Lowey J, Suh E. G-SQZ: compact encoding of genomic sequence and quality data. Bioinformatics 2010, 26(17):2192-2194. 10.1093/bioinformatics/btq346, 20605925.
32. Li H, Handsaker B, Wysoker A, Fennell T, Ruan J, Homer N, Marth, Abecasis G, Durbin R, 1000 Genome Project Data Processing Subgroup The sequence alignment/map (SAM) format and SAMtools. Bioinformatics 2009, 25(16):2078-2079. 10.1093/bioinformatics/btp352, 2723002, 19505943, 1000 Genome Project Data Processing Subgroup.
33. Fritz MH-Y, Leinonen R, Cochrane G, Birney E. Efficient storage of high throughput DNA sequencing data using reference-based compression. Genome Res 2011, 21:734-740. 10.1101/gr.114819.110, 3083090, 21245279.
34. Sakib MN, Tang J, Zheng WJ, Huang C-T. Improving transmission efficiency of large sequence alignment/map (SAM) files. PLoS ONE 2011, 6(12):e28251. 10.1371/journal.pone.0028251, 3229529, 22164252.
35. Manzini G, Rastero M. A simple and fast DNA compressor. Softw Pract Exp 2004, 34(14):1397-1411.
36. Pinho AJ, Ferreira PJSG, Neves AJR, Bastos CAC. On the representability of complete genomes by multiple competing finite-context (Markov) models. PLoS ONE 2011, 6(6):e21588. 10.1371/journal.pone.0021588, 3128062, 21738720.
37. Cao MD, Dix TI, Allison L, Mears C. A simple statistical algorithm for biological sequence compression. Proceedings of the Data Compression Conference 2007, 43-52. Washington, DC, USA: IEEE Computer Society Press.
38. Wandelt S, Leser U. Adaptive efficient compression of genomes. Algo Mol Biol 2012, 7:30.
39. Deorowicz S, Grabowski S. Robust relative compression of genomes with random access. Bioinformatics 2011, 27(11):2979-2986.
40. Pinho AJ, Pratas D, Garcia SP. GReEn: a tool for efficient compression of genome resequencing data. Nucleic Acids Res 2012, 40(4):e27. 10.1093/nar/gkr1124, 3287168, 22139935.
41. Wang C, Zhang D. A novel compression tool for efficient storage of genome resequencing data. Nucleic Acids Res 2011, 39(7):e45. 10.1093/nar/gkr009, 3074166, 21266471.
42. Kuruppu S, Puglisi SJ, Zobel J. Optimized relative Lempel-Ziv compression of genomes. Proceedings of the ACSC Australasian Computer Science Conference 2011, 91-98. Sydney, Australia: Australian Computer Society, Inc., Reynolds M.
43. Gusfield D. Algorithms on strings, trees and sequences: Computer science and computational biology 1997, Cambridge, UK: Cambridge University Press.
44. Daily K, Rigor P, Christley S, Hie X, Baldi P. Data structures and compression algorithms for high-throughput sequencing technologies. BMC Bioinformatics 2010, 11:514. 10.1186/1471-2105-11-514, 2964686, 20946637.
45. Popitsch N, von Haeseler A. NGC: lossless and lossy compression of aligned high-throughput sequencing data. Nucleic Acids Res 2013, 41(1):e27. 10.1093/nar/gks939, 3592443, 23066097.
46. Li H. Tabix: fast retrieval of sequence features from generic TAB-delimited files. Bioinformatics 2011, 27(5):718-719. 10.1093/bioinformatics/btq671, 3042176, 21208982.
47. Levy S, Sutton G, Ng PC, Feuk L, Halpern AL, Walenz BP, Axelrod N, Huang J, Kirkness EF, Denisov G, Lin Y, MacDonald JR, Pang AWC, Shago M, Stockwell TB, Tsiamouri A, Bafna V, Bansal V, Kravitz SA, Busam DA, Beeson KY, McIntosh TC, Remington KA, Abril JF, Gill J, Borman J, Rogers YH, Frazier ME, Scherer SW, Strausberg RL, Venter JC. The diploid genome sequence of an individual human. PLoS Biol 2007, 5(10):e254. 10.1371/journal.pbio.0050254, 1964779, 17803354.
48. Christley S, Lu Y, Li C, Xie X. Human genomes as email attachments. Bioinformatics 2009, 25(2):274-275. 10.1093/bioinformatics/btn582, 18996942.
49. Pavlichin D, Weissman T, Yona G. The human genome contracts again. Bioinformatics 2013, 29(17):2199-2202. 10.1093/bioinformatics/btt362, 23793748.
50. Deorowicz S, Danek A, Grabowski S. Genome compression: a novel approach for large collections. Bioinformatics 2013, 29(20):2572-2578. 10.1093/bioinformatics/btt460, 23969136.
51. Chern BG, Ochoa I, Manolakos A, No A, Venkat K, Weissman T. Reference based genome compression. Publicly available preprint arXiv:1204.1912v1 2012.
52. Kuruppu S, Puglisi SJ, Zobel J. Relative Lempel-Ziv compression of genomes for large-scale storage and retrieval. Proceedings of the 17th International Symposium on String Matching and Information Retrieval (SPIRE) 2010, 201-206. Springer-Verlag, Berlin-Heidelberg: Springer, LNCS 6393, Chávez E, Lonardi S.
53. Kreft S, Navarro G. LZ77-like compression with fast random access. Proceedings of the Data Compression Conference 2010, 239-248. Washington, DC, USA: IEEE Computer Society.
54. Ohlebusch E, Fischer J, Gog S. CST++. Proceedings of the 17th International Symposium on String Matching and Information Retrieval (SPIRE) 2010, 322-333. Springer-Verlag, Berlin-Heidelberg: Springer, LNCS 6393, Chávez E, Lonardi S.
55. Compeau PE, Pevzner PA, Tesler G. How to apply de Bruijn graphs to genome assembly. Nat Biotechnol 2011, 29(11):987-991. 10.1038/nbt.2023, 22068540.
56. Conway TC, Bromage AJ. Succinct data structures for assembling large genomes. Bioinformatics 2011, 27(4):479-486. 10.1093/bioinformatics/btq697, 21245053.
57. Bloom BH. Space/time trade-offs in hash coding with allowable errors. Commun ACM 1970, 13(7):422-426.
58. Chikhi R, Rizk G. Space-efficient and exact de Bruijn graph representation based on a Bloom filter. Proceedings of the 12th International Workshop on Algorithms in Bioinformatics (WABI) 2012, 236-248. Springer-Verlag, Berlin-Heidelberg: Springer, LNCS 7534, Raphael BJ, Tang J.
59. Salikhov K, Sacomoto G, Kucherov G. Using cascading Bloom filters to improve the memory usage for de Brujin graphs. Proceedings of the 13th International Workshop on Algorithms in Bioinformatics (WABI) 2013, 364-376. Springer-Verlag, Berlin-Heidelberg: Springer, LNCS 8126, Darling A. E., Stoye J.
60. Ye C, Ma ZS, Cannon CH, Pop M, Yu DW. Exploiting sparseness in de novo genome assembly. BMC Bioinformatics 2012, 13(Suppl 6):S1.
61. Myers EW. The fragment assembly string graph. Bioinformatics 2005, 21(suppl 2):ii79-ii85.
62. Simpson JT, Durbin R. Efficient de novo assembly of large genomes using compressed data structures. Genome Res 2012, 22:549-556. 10.1101/gr.126953.111, 3290790, 22156294.
63. Ferragina P, Manzini G. Opportunistic data structures with applications. Proceedings of the 41st Annual Symposium on Foundations of Computer Science (FOCS) 2000, 390-398. Redondo Beach, California, USA: IEEE Computer Society.
64. Gonnella G, Kurtz S. Readjoiner: a fast and memory efficient string graph-based sequence assembler. BMC Bioinformatics 2012, 13:82. 10.1186/1471-2105-13-82, 3507659, 22559072.
65. Navarro G, Mäkinen V. Compressed full-text indexes. ACM Computing Surv 2007, 39:2.
66. Kreft S, Navarro G. On compressing and indexing repetitive sequences. Theor Comput Sci 2013, 483:115-133.
67. Gagie T, Gawrychowski P, Kärkkäinen J, Nekrich Y, Puglisi SJ. A faster grammar-based self-index. Proceedings of the 6th International Conference on Language and Automata Theory and Applications (LATA) 2012, 240-251. Springer-Verlag, Berlin-Heidelberg: LNCS 7183.
68. Do HH, Jansson J, Sadakane K, Sung W-K. Fast relative Lempel-Ziv self-index for similar sequences. Proceedings of the Joint International Conference on Frontiers in Algorithmics and Algorithmic Aspects in Information and Management (FAW-AAIM) 2012, 291-302. Springer-Verlag, Berlin-Heidelberg: LNCS 7285.
69. Gagie T, Gawrychowski P, Puglisi SJ. Faster approximate pattern matching in compressed repetitive texts. Proceedings of the 22nd International Symposium on Algorithms and Computation (ISAAC) 2011, 653-662. Springer-Verlag, Berlin-Heidelberg: LNCS 7074.
70. Ahmadi A, Behm A, Honnalli N, Li C, Weng L, Xie X. Hobbes: optimized gram-based methods for efficient read alignment. Nucleic Acids Res 2012, 40(6):e41. 10.1093/nar/gkr1246, 3315303, 22199254.
71. Langmead B, Trapnell C, Pop M, Salzberg SL. Ultrafast and memory-efficient alignment of short DNA sequences to the human genome. Genome Biol 2009, 10(3):R25. 10.1186/gb-2009-10-3-r25, 2690996, 19261174.
72. Langmead B, Salzberg SL. Fast gapped-read alignment with Bowtie. Nature Methods 2012, 9:357-359. 10.1038/nmeth.1923, 3322381, 22388286.
73. Li H, Durbin R. Fast and accurate short read alignment with Burrows-Wheeler transform. Bioinformatics 2009, 25(14):1754-1760. 10.1093/bioinformatics/btp324, 2705234, 19451168.
74. Li H, Durbin R. Fast and accurate long-read alignment with Burrows-Wheeler transform. Bioinformatics 2010, 26(5):589-595. 10.1093/bioinformatics/btp698, 2828108, 20080505.
75. Li R, Yu C, Li Y, Lam T-W, Yiu S-M, Kristiansen K, Wang J. SOAP2: an improved ultrafast tool for short read alignment. Bioinformatics 2009, 25(15):1966-1967. 10.1093/bioinformatics/btp336, 19497933.
76. Marco-Sola S, Sammeth M, Guigó R, Ribeca P. The GEM mapper: fast, accurate and versatile alignment by filtration. Nat Methods 2012, 9(12):1185-1188. 10.1038/nmeth.2221, 23103880.
77. Kärkkäinen J. Fast BWT in small space by blockwise suffix sorting. Theor Comput Sci 2007, 387:249-257.
78. Ferragina P, Gagie T, Manzini G. Lightweight data indexing and compression in external memory. Algorithmica 2012, 63(3):707-730.
79. Wang Z, Gerstein M, Snyder M. RNA-Seq: a revolutionary tool for transcriptomics. Nat Rev Genet 2009, 10(1):57-63. 10.1038/nrg2484, 2949280, 19015660.
80. Trapnell C, Pachter L, Salzberg SL. TopHat: discovering splice junctions with RNA-Seq. Bioinformatics 2009, 25(9):1105-1111. 10.1093/bioinformatics/btp120, 2672628, 19289445.
81. Rivals E. CRAC: an integrated approach to the analysis of RNA-seq reads. Genome Biol 2013, 14(3):R30. 10.1186/gb-2013-14-3-r30, 23537109.
82. Alamancos GP, Agirre E, Eyras E. Methods to study splicing from high-throughput RNA Sequencing data. Publicly available preprint arXiv:1304.5952v1.
83. Li H. Exploring single-sample SNP and INDEL calling with whole-genome de novo assembly. Bioinformatics 2012, 28(14):1838-1844. 10.1093/bioinformatics/bts280, 3389770, 22569178.
84. Liu C-M, Wong TKF, Wu E, Luo R, Yiu S-M, Li Y, Wang B, Yu C, Chu X, Zhao K, Li R, Lam TW. SOAP3: ultra-fast GPU-based parallel alignment tool for short reads. Bioinformatics 2012, 28(6):878-879. 10.1093/bioinformatics/bts061, 22285832.
85. Luo R, Wong T, Zhu J, Liu C-M, Zhu X, Wu E, Lee L-K, Lin H, Zhu W, Cheung DW, Ting H-F, Yiu S-M, Peng S, Yu C, Li Y, Li R, Lam TW. SOAP3-dp: Fast, accurate and sensitive GPU-based short read aligner. PLoS ONE 2013, 8(5):e65632. 10.1371/journal.pone.0065632, 3669295, 23741504.
86. Gog S, Petri M. Optimized succinct data structures for massive data. Softw Pract Exp 2013, doi: 10.1002/spe.2198.
87. Loh P-R, Baym M, Berger B. Compressive genomics. Nat Biotechnol 2012, 30(7):627-630. 10.1038/nbt.2241, 22781691.
88. Altschul SF, Gish W, Miller W, Myers EW, Lipman DJ. Basic local alignment search tool. J Mol Biol 1990, 215(3):403-410.
89. Kent WJ. BLAT-the BLAST-like alignment tool. Genome Res 2002, 12(4):656-664. 187518, 11932250.
90. Deorowicz S, Debudaj-Grabysz A, Grabowski S. Disk-based k-mer counting on a PC. BMC Bioinformatics 2013, 14:Article no. 160.
91. Zerbino DR, Birney E. Velvet: algorithms for de novo short read assembly using de Bruijn graphs. Genome Res 2008, 18(5):821-829. 10.1101/gr.074492.107, 2336801, 18349386.
92. Simpson JT, Wong K, Jackman SD, Schein JE, Jones SJM, Birol I. ABySS: A parallel assembler for short read sequence data. Genome Res 2009, 19(6):1117-1123. 10.1101/gr.089532.108, 2694472, 19251739.
93. Cao MD, Dix TI, Allison L. A genome alignment algorithm based on compression. BMC Bioinformatics 2010, 11(1):599. 10.1186/1471-2105-11-599, 3022628, 21159205.
94. Bhaduri A, Qu K, Lee CS, Ungewickell A, Khavari P. Rapid identification of nonhuman sequences in high throughput sequencing data sets. Bioinformatics 2012, 28(8):1174-1175. 10.1093/bioinformatics/bts100, 3324519, 22377895.
95. Ferragina P, Giancarlo R, Greco V, Manzini G, Valiente G. Compression-based classification of biological sequences and structures via the universal similarity metric: experimental assessment. BMC Bioinformatics 2007, 8:252. 10.1186/1471-2105-8-252, 1939857, 17629909.
96. Li M, Chen X, Li X, Ma B, Vitányi PMB. The similarity metric. IEEE Trans Inf Theory 2004, 50(12):3250-3264.
97. Freschi V, Bogliolo A. A lossy compression technique enabling duplication-aware sequence alignment. Evol Bioinformatics 2012, 8:171-180.
98. Illumina HiSeq 2500 system user guide. 2012, [http://supportres.illumina.com/documents/myillumina/223bf628-0b46-409f-aa3d-4f3495fe4f69/hiseq2500_ug_15035786_ a_public.pdf], Illumina.
99. Illumina New algorithms increase computing efficiency for IGN whole-genome analysis. 2013, [http://res.illumina.com/documents/products/technotes/technote_ign_isaac_software.pdf],Illumina.