Verification and validation of bioinformatics software without a gold standard: A case study of BWA and Bowtie

BMC Bioinformatics

Volumn 15, Issue 16, 2014, Pages

  Giannoulatou, Eleni ^a,b   Park, Shin Ho ^a,b   Humphreys, David T ^a,b   Ho, Joshua W K ^a,b  

^a VICTOR CHANG CARDIAC RESEARCH INSTITUTE   (Australia)

^b UNIVERSITY OF NEW SOUTH WALES   (Australia)

Author keywords

Bioinformatics software testing; Genomic medicine; Metamorphic testing; Software quality assurance

Indexed keywords

BIOINFORMATICS; COMPUTER SOFTWARE SELECTION AND EVALUATION; GENES; GOLD; QUALITY ASSURANCE; QUALITY CONTROL; VERIFICATION;

BIOINFORMATICS SOFTWARE; METAMORPHIC TESTING; SEQUENCE ALIGNMENTS; SOFTWARE QUALITY ASSURANCE; SOFTWARE SPECIFICATION; SYSTEMATIC EVALUATION; USER EXPECTATIONS; VERIFICATION-AND-VALIDATION;

SOFTWARE TESTING;

BIOINFORMATICS; COMPUTER PROGRAM; EXPECTATION; GOLD STANDARD; HAPLOTYPE MAP; MEDICINE; SEQUENCE ALIGNMENT; VALIDATION PROCESS; BIOLOGY; DNA SEQUENCE; GENOMICS; HIGH THROUGHPUT SEQUENCING; HUMAN; HUMAN GENOME; PROCEDURES; STANDARDS;

COMPUTATIONAL BIOLOGY; GENOME, HUMAN; GENOMICS; HIGH-THROUGHPUT NUCLEOTIDE SEQUENCING; HUMANS; SEQUENCE ALIGNMENT; SEQUENCE ANALYSIS, DNA; SOFTWARE;

EID: 84961626302     PISSN: None     EISSN: 14712105     Source Type: Journal    
DOI: 10.1186/1471-2105-15-S16-S15     Document Type: Article

References (42)

1. Mardis ER: The impact of next-generation sequencing technology on genetics. Trends in Genetics. 2008, 24 (3): 133-141. 10.1016/j.tig.2007.12.007.
2. Bamshad MJ, Ng SB, Bigham AW, Tabor HK, Emond MJ, Nickerson DA, Shendure J: Exome sequencing as a tool for mendelian disease gene discovery. Nature Reviews Genetics. 2011, 12 (11): 745-755. 10.1038/nrg3031.
3. Kiezun A, Garimella K, Do R, Stitziel NO, Neale BM, McLaren PJ, Gupta N, Sklar P, Sullivan PF, Moran JL: Exome sequencing and the genetic basis of complex traits. Nature Genetics. 2012, 44 (6): 623-630. 10.1038/ng.2303.
4. Boycott KM, Vanstone MR, Bulman DE, MacKenzie AE: Rare-disease genetics in the era of next-generation sequencing: discovery to translation. Nature Reviews Genetics. 2013, 14 (10): 681-691. 10.1038/nrg3555.
5. Feero WG, Guttmacher AE, McDermott U, Downing JR, Stratton MR: Genomics and the continuum of cancer care. New England Journal of Medicine. 2011, 364 (4): 340-350. 10.1056/NEJMra0907178.
6. Berger B, Peng J, Singh M: Computational solutions for omics data. Nature Reviews Genetics. 2013, 14 (5): 333-346. 10.1038/nrg3433.
7. O'Rawe J, Jiang T, Sun G, Wu Y, Wang W, Hu J, Bodily P, Tian L, Hakonarson H, Johnson WE: Low concordance of multiple variant-calling pipelines: practical implications for exome and genome sequencing. Genome Medicine. 2013, 5 (3): 28-10.1186/gm432.
8. McCarthy DJ, Humburg P, Kanapin A, Rivas MA, Gaulton K, Consortium TW, Cazier JB, Donnelly P: Choice of transcripts and software has a large effect on variant annotation. Genome Medicine. 2014, 6: 26-10.1186/gm543.
9. Hatton L, Roberts A: How accurate is scientific software?. IEEE Transactions on Software Engineering. 1994, 20 (10): 785-797. 10.1109/32.328993.
10. Joppa LN, McInerny G, Harper R, Salido L, Takeda K, O'Hara K, Gavaghan D, Emmott S: Troubling trends in scientific software use. Science. 2013, 340 (6134): 814-815. 10.1126/science.1231535.
11. Peng RD: Reproducible research in computational science. Science. 2011, 334 (6060): 1226-10.1126/science.1213847.
12. Hayden EC: Mozilla plan seeks to debug scientific code. Nature. 2013, 501 (7468): 472-472. 10.1038/501472a.
13. Baxter SM, Day SW, Fetrow JS, Reisinger SJ: Scientific software development is not an oxymoron. PLoS Computational Biology. 2006, 2 (9): 87-10.1371/journal.pcbi.0020087.
14. Sandve GK, Nekrutenko A, Taylor J, Hovig E: Ten simple rules for reproducible computational research. PLoS Computational Biology. 2013, 9 (10): 1003285-10.1371/journal.pcbi.1003285.
15. Seemann T: Ten recommendations for creating usable bioinformatics command line software. GigaScience. 2013, 2 (1): 15-10.1186/2047-217X-2-15.
16. Altschul S, Demchak B, Durbin R, Gentleman R, Krzywinski M, Li H, Nekrutenko A, Robinson J, Rasband W, Taylor J: The anatomy of successful computational biology software. Nature Biotechnology. 2013, 31 (10): 894-897. 10.1038/nbt.2721.
17. Wilson G, Aruliah D, Brown CT, Hong NPC, Davis M, Guy RT, Haddock SH, Huff KD, Mitchell IM, Plumbley MD: Best practices for scientific computing. PLoS Biology. 2014, 12 (1): 1001745-10.1371/journal.pbio.1001745.
18. Beizer B: Software Testing Techniques. 1990, Van Nostrand Reinhold
19. Ammann P, Offutt J: Introduction to Software Testing. 2008, Cambridge University Press
20. Bertolino A: Software testing research: Achievements, challenges, dreams. 2007 Future of Software Engineering. 2007, IEEE Computer Society, 85-103.
21. Evans TW, Gillespie CS, Wilkinson DJ: The SBML discrete stochastic models test suite. Bioinformatics. 2008, 24: 285-286. 10.1093/bioinformatics/btm566.
22. Bergmann FT, Sauro HM: Comparing simulation results of SBML capable simulators. Bioinformatics. 2008, 24: 1963-1965. 10.1093/bioinformatics/btn319.
23. Chen TY, Cheung SC, Yiu SM: Metamorphic testing: a new approach for generating next test cases. Technical Report HKUST-CS98-01. 1998, Department of Computer Science, Hong Kong University of Science and Technology, Hong Kong
24. Chen TY, Feng J, Tse TH: Metamorphic testing of programs on partial differential equations: a case study. Proceedings of the 26th Annual International Computer Software and Applications Conference (COMPSAC 2002). 2002, IEEE Computer Society Press, 327-333.
25. Chen TY, Huang DH, Tse TH, Zhou ZQ: Case studies on the selection of useful relations in metamorphic testing. Proceedings of the 4th Ibero-American Symposium on Software Engineering and Knowledge Engineering (JIISIC 2004). 2004, Polytechnic University of Madrid, Madrid, Spain, 569-583.
26. Chen TY, Kuo FC, Zhou ZQ: An effective testing method for end-user programmer. Proceedings of the First Workshop on End-User Software Engineering. 2005
27. Chen TY, Ho JW, Liu H, Xie X: An innovative approach for testing bioinformatics programs using metamorphic testing. BMC Bioinformatics. 2009, 10 (1): 24-10.1186/1471-2105-10-24.
28. Sadi MS, Kuo F, Ho JWK, Charleston MA, Chen TY: Verification of phylogenetic inference programs using metamorphic testing. Journal of Bioinformatics and Computational Biology. 2011, 09 (06): 729-747. 10.1142/S021972001100563X.
29. Xie X, Ho JWK, Murphy C, Kaiser G, Xu B, Chen TY: Testing and validating machine learning classifiers by metamorphic testing. Journal of Systems and Software. 2011, 84 (4): 544-558. 10.1016/j.jss.2010.11.920.
30. Li H, Durbin R: Fast and accurate short read alignment with burrows-wheeler transform. Bioinformatics. 2009, 25 (14): 1754-1760. 10.1093/bioinformatics/btp324.
31. Langmead B, Trapnell C, Pop M, Salzberg SL: Ultrafast and memory-efficient alignment of short dna sequences to the human genome. Genome Biology. 2009, 10 (3): 25-10.1186/gb-2009-10-3-r25.
32. Langmead B, Salzberg SL: Fast gapped-read alignment with bowtie 2. Nature Methods. 2012, 9 (4): 357-359. 10.1038/nmeth.1923.
33. Ruffalo M, LaFramboise T, Koyutürk M: Comparative analysis of algorithms for next-generation sequencing read alignment. Bioinformatics. 2011, 27 (20): 2790-2796. 10.1093/bioinformatics/btr477.
34. Li H, Homer N: A survey of sequence alignment algorithms for next-generation sequencing. Briefings in Bioinformatics. 2010, 11 (5): 473-483. 10.1093/bib/bbq015.
35. Li H, Handsaker B, Wysoker A, Fennell T, Ruan J, Homer N, Marth G, Abecasis G, Durbin R: The sequence alignment/map format and samtools. Bioinformatics. 2009, 25 (16): 2078-2079. 10.1093/bioinformatics/btp352.
36. 1000 Genomes Project Consortium: A map of human genome variation from population-scale sequencing. Nature. 2010, 467 (7319): 1061-1073. 10.1038/nature09534.
37. 1000 Genomes Project Consortium: An integrated map of genetic variation from 1,092 human genomes. Nature. 2012, 491 (7422): 56-65. 10.1038/nature11632.
38. McKenna A, Hanna M, Banks E, Sivachenko A, Cibulskis K, Kernytsky A, Garimella K, Altshuler D, Gabriel S, Daly M: The genome analysis toolkit: a mapreduce framework for analyzing next-generation dna sequencing data. Genome Research. 2010, 20 (9): 1297-1303. 10.1101/gr.107524.110.
39. Li H, Homer N: A survey of sequence alignment algorithms for next-generation sequencing. Briefings in Bioinformatics. 2010, 11 (5): 473-483. 10.1093/bib/bbq015.
40. Bao S, Jiang R, Kwan W, Wang B, Ma X, Song YQ: Evaluation of next-generation sequencing software in mapping and assembly. Journal of Human Genetics. 2011, 56 (6): 406-414. 10.1038/jhg.2011.43.
41. Fonseca NA, Rung J, Brazma A, Marioni JC: Tools for mapping high-throughput sequencing data. Bioinformatics. 2012, 28 (24): 3169-3177. 10.1093/bioinformatics/bts605.
42. Yu X, Guda K, Willis J, Veigl M, Wang Z, Markowitz S, Adams MD, Sun S: How do alignment programs perform on sequencing data with varying qualities and from repetitive regions?. BioData Mining. 2012, 5 (1): 6-10.1186/1756-0381-5-6.