A survey of quality assurance practices in biomedical open source software projects

Journal of Medical Internet Research

Volumn 9, Issue 2, 2007, Pages

  Koru, Günes ^a   El Emam, Khaled ^b   Neisa, Angelica ^b   Umarji, Medha ^a  

^a Baltimore County   (United States)

^b CHILDREN S HOSPITAL OF EASTERN ONTARIO   (Canada)

Author keywords

Code inspections and walkthroughs; Computational biology; Information systems; Medical informatics; Open source software; Software quality assurance; Software reliability; Software program verification

Indexed keywords

BIOINFORMATICS; COMPUTER PROGRAM; MEDICAL INFORMATION SYSTEM; PATIENT SAFETY; QUALITY CONTROL; REVIEW; ARTICLE; INFORMATION PROCESSING; MEDICAL INFORMATICS; PATENT; PEER REVIEW; QUESTIONNAIRE; STANDARD; STATISTICAL MODEL;

DATA COLLECTION; INTELLECTUAL PROPERTY; LOGISTIC MODELS; MEDICAL INFORMATICS; PEER REVIEW; QUALITY CONTROL; QUESTIONNAIRES; SOFTWARE; SOFTWARE DESIGN; SOFTWARE VALIDATION;

EID: 34249951076     PISSN: None     EISSN: 14388871     Source Type: Journal    
DOI: 10.2196/jmir.9.2.e8     Document Type: Review

References (95)

1. Prada G, Roberts G, Vail S, Anderson M, Down E, Fooks G, et al. Understanding Health Care Cost Drivers and Escalators. Ottawa, ON: The Conference Board of Canada; 2004. URL: http://www.health.gov.ab.ca/resources/ publications/conference_board.pdf [accessed 2007 Apr 16]
2. Prada G, Grimes K, McCleery A, Nguyen D, Pomey MP, Reed V, et al. Challenging Health Care System Sustainability: Understanding Health System Performance of Leading Countries. Ottawa, ON: The Conference Board of Canada; 2004. URL: http://www.health.gov.ab.ca/resources/publications/Conference_Board2. pdf [accessed 2007 Apr 16]
3. Irving R. Report on IT in Canadian Hospitals. Canadian Healthcare Technology 2003.
4. Raymond B, Dold C. Clinical Information Systems: Achieving the Vision. Oakland, CA: Kaiser Permanente Institute for Health Policy; 2002. URL: http://www.kpihp.org/publications/docs/clinical_information.pdf [accessed 2007 Apr 16]
5. Bates D, Ebell M, Gotlieb E, Zapp J, Mullins H. A proposal for electronic medical records in US primary care. J Am Med Inform Assoc 2003;10(1):1-10.
6. Goulde M, Brown E. Open Source Software: A Primer for Health Care Leaders. Oakland, CA: California HealthCare Foundation; 2006. URL: http://www.chcf.org/documents/ihealth/OpenSourcePrimer.pdf [accessed 2007 Apr 16]
7. Kantor GS, Wilson WD, Midgley A. Open-source software and the primary care EMR. J Am Med Inform Assoc 2003;10(6):616
8. Giera J. The Costs and Risks of Open Source. Cambridge, MA: Forrester Research Inc; 2004.
9. Hunt F, Probert D, Barratt S. Adopting new technology: the case of open source software at Marconi. The 12th International Conference on Management of Technology (IAMOT May 12-15 2003) Nancy, France 2003.
10. Dal MJ. Open Source Software in Canada. A Collaborative Fact Finding Study. e-cology Corporation. 2003. URL: http://www.e-cology.ca/canfloss/report/ CANfloss_Report.pdf [accessed 2007 Apr 16]
11. Dedrick J, West J. An exploratory study into open source platform adoption. Proceedings of the 37th Hawaii International Conference on System Sciences 2004:10.
12. DiDio L. Linux, Unix and Windows TCO Comparison, Part 1. Boston, MA: The Yankee Group; 2004.
13. Valdes I, Kibbe DC, Tolleson G, Kunik ME, Petersen LA. Barriers to proliferation of electronic medical records. Inform Prim Care 2004;12(1):3-9.
14. Kolata G. In unexpected Medicare benefit, US will offer doctors free electronic records system. New York Times Jul 21 2005:14.
15. Goldstein D, Ponkshe S, Maduro R. Analysis of Open Source Software (OSS) and EHRs: Profile of Increasing Use of OSS in the Federal Government and Healthcare. Atlantis, FL: Medical Alliances Inc; 2004. URL: http://www. medicalalliances.com/downloads/files/Open_Source_Software- Government_and_Healthcare_White_Paper-Medical_Alliances_2.doc [accessed 2007 Apr 16]
16. Elsner C, Egbring M, Kottkamp H, Berger T, Zoller S, Hinricks G. Open source or commercial products for electronic data capture in clinical trials? A scorecard comparison. Comput Cardiol 2003;30:371-373.
17. Erickson BJ, Langer S, Nagy P. The role of open-source software in innovation and standardization in radiology. J Am Coll Radiol 2005 Nov;2(11):927-931.
18. Yang JYH, Gentleman RC, Carey VJ, Bates DM, Bolstad B, Dettling M, Dudoit S, et al. Bioconductor: open software development for computational biology and bioinformatics. Genome Biol 2004;5(10):R80.
19. Stajich J. The Bioperl Project: A Look Ahead. Talk presented at: Bioinformatics Open Source Conference; August 1-2, Edmonton, Canada. 2002. URL: http://www.open-bio.org/bosc2002/slides/2002-08-01-Stajich-Bioperl.pdf [accessed 2007 Apr 16]
20. Pocock M. BioJava Toolkit Progress. Talk presented at: Bioinformatics Open Source Conference; August 1-2,. 2002. URL: http://www.open-bio.org/ bosc2002/talks.html#biojava [accessed 2007 Apr 16]
21. McCormick J, Gage D. Cincinnati Children's Hospital: Shots in the Dark. Baseline. 2004 Aug 1. URL: http://www.baselinemag.com/article2/0,1540,1655082, 00.asp [accessed 2007 Apr 16]
22. Gage D, McCormick J. Why Software Quality Matters: 'We Did Nothing Wrong'. Baseline. 2004 Mar 4. URL: http://www.baselinemag.com/article2/0,1397, 1544403,00.asp [accessed 2007 Apr 16]
23. Basili V, Belady L, Boehm B, Brooks F, Browne J, DeMillo R, et al. Final report: NSF workshop on a software research program for the 21st century. Software Eng Notes 1999;24(3):37-45.
24. Report of the Inquiry into the London Ambulance Service. UK: South West Thames Regional Health Authority; 1993. URL: http://www.cs.ucl.ac.uk/staff/A. Finkelstein/las/lascase0.9.pdf [accessed 2007 Apr 16]
25. van Heusden P. Applying Software Validation Techniques to Bioperl. Talk presented at: Bioinformatics Open Source Conference; July 29-30, 2004;Glasgow, UK. URL
26. Chrissis M-B, Konrad M, Shrum S. CMMI: Guidelines for Process Integration and Product Improvement. Addison-Wesley; 2003.
27. Raymond ES. The Cathedral and the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary. O'Reilly; 1999.
28. Greiner S, Boskovic B, Brest J, Zumer V. Security issues in information systems based on open source technologies. EUROCON 2003:12-15.
29. El EK. Software Inspection Best Practices. In: Agile Project Management Advisory Service, Executive Report. Volume 2. Number 9. Arlington, MA: Cutter Consortium; 2001.
30. Bagchi S, Madeira H. Open source software - a recipe for vulnerable software, or the only way to keep the bugs and the bad guys out? Panel position statement presented at: 14th International Symposium on Software Reliability Engineering (ISSRE) 2003; 1234. 567:347-352.
31. Jorgensen N. Putting it all in the trunk: incremental software development in the FreeBSD open source project. Inform Syst J 2001;11(4):321-336.
32. Stark J. Peer reviews as a quality management technique in open-source software development projects. European Conference on Software Quality 2002:340-350.
33. Maccormack A, Kemerer C, Cusumano M, Crandall B. Trade-offs between productivity and quality in selecting software development practices. IEEE Software 2003;(5):78-84.
34. Cusumano M, MacCormack A, Kemerer C, Randall B. Software development worldwide: the state of the practice. IEEE Software 2003; 1234. 567:28-34.
35. Zhao L, Elbaum S. A survey on quality related activities in open source. Software Eng Notes 2000;25(3):54-57.
36. Porter A, Siy H, Toman C, Votta L. An experiment to assess the cost-benefits of code inspections in large scale software development. IEEE T Software Eng 1997;23(6):329-346.
37. STD I. IEEE Standard for Software Reviews and Audits. ANSI/IEEE STD 1028-1988 1989:.
38. Ackerman AF, Buchwald LS, Lewski FH. Software inspections: an effective verification process. IEEE Software 1989;6(3):31-36.
39. Lau L, Jeffery R, Sauer C. Some Empirical Support for Software Development Technical Reviews. Sydney, Australia: University of New South Wales; 1996.
40. Grady RB. Practical Software Metrics for Project Management and Process Improvement. Englewood Cliffs, NJ: Prentice-Hall; 1992.
41. Laitenberger O, Debaud JM. An encompassing life-cycle centric survey of software inspection. J Syst Software 2000;50(1):5-31.
42. Fagan ME. Design and code inspections to reduce errors in program development. IBM Syst J 1976;15(3):182-211.
43. Weller EF. Lessons from three years of inspection data. IEEE Software 1993;10(5):38-45.
44. Madachy R, Little L, Fan S. Analysis of a successful inspection program. Proceedings of the 18th Annual NASA Software Engineering Laboratory Workshop 1993:176-188.
45. Gilb T, Graham D. Software Inspection. New York: Addison-Wesley; 1993.
46. Strauss S, Ebenau R. Software Inspection Process. New York: McGraw Hill; 1994.
47. Kelly J, Sheriff J, Hops J. An analysis of defect densities found during software inspections. J Syst Software 1992;17(2):111-117.
48. Johnson P. Reengineering inspection. Comm ACM 1998;41(2):49-52.
49. Eick S, Loader C, Long M, Votta L, Weil SV. Estimating software fault content before coding. Proceedings of the 14th International Conference on Software Engineering 1992:59-65.
50. Dunsmore A, Roper M, Wood M. The development and evaluation of three diverse techniques for object-oriented code inspection. IEEE T Software Eng 2003;29(8):677-686.
51. Halloran T, Scherlis WL. High quality and open source software practices. Position paper presented at: The Second Workshop on Open Source Software Engineering, International Conference on Software Engineering; May 19-25,. 2002. URL: http://opensource.ucc.ie/icse2002/HalloranScherlis.pdf [accessed 2007 Apr 16]
52. Gacek C, Arief B. The many meanings of open source. IEEE Software 2004;21(1):34-40.
53. Zhao L, Elbaum S. Quality assurance under the open source development model. J Syst Software 2003;66(1):65-75.
54. Mockus A, Fielding R, Herbsleb J. Two case studies of open source software development: Apache and Mozilla. ACM T Softw Eng Meth 2002;11(3):309-346.
55. Sharma S, Sugumaran V, Rajagoplan B. A framework for creating hybrid open source software communities. Inform Syst J 2002;12(1):7-25.
56. Erenkrantz J. Release management within open source projects. Proceedings of the Third Workshop on Open Source Software Engineering, Portland, OR 2003.
57. Thomas C. Improving verification, validation, and test of the Linux kernel: the Linux stabilization project. In: Proceedings of the Third Workshop on Open Source Software Engineering. Portland, OR; 2003.
58. Iyer M. Analysis of Linux test project's kernel code coverage. Austin, TX: IBM Corporation; 2002. URL: http://ltp.sourceforge.net/documentation/ technical_papers/kernel_coverage.pdf [accessed 2007 Apr 16]
59. Myers G. A controlled experiment in program testing and code walkthroughs/inspections. Comm ACM 1978;21(9):760-768.
60. Basili V, Selby R. Comparing the effectiveness of software testing strategies. IEEE T Software Eng 1987;13(12):1278-1296.
61. Kamsties E, Lott C. An empirical evaluation of three defect-detection techniques. Proceedings of the 5th European Software Engineering Conference, Sitges, Spain, September 25-28, 1995.
62. Wood M, Roper M, Brooks A, Miller J. Comparing and combining software defect-detection techniques. Proceedings of the 6th European Conference on Software Engineering 1997:262-277.
63. Jalote P, Haragopal M. Overcoming the NAH Syndrome for Inspection Deployment. Proceedings of the International Conference on Software Engineering 1998:371-378.
64. Hissam S, Weinstock C, Plakosh D, Asundi J. Perspectives on Open Source Software. CMU/SEI-2001-TR-019. Pittsburg, PA: Carnegie Mellon Software Engineering Institute; 2001. URL: http://www.sei.cmu.edu/pub/documents/01. reports/pdf/01tr019.pdf [accessed 2007 Apr 16]
65. Mockus A, Fielding RT, Herbsleb J. The Apache server.. Proceedings of the 22nd International Conference on Software Engineering A case study of open source software development; 2000:263-272.
66. Miller B, Fredriksen L, So B. An Empirical Study of the Reliability of Unix Utilities. Comm ACM 1990;33(12):32-44.
67. Capiluppi A, Lago P, Morisio M. Characteristics of open source projects. Proceedings of the Seventh European Conference on Software Maintenance and Engineering; 2003;:317.
68. Hunt F, Johnson P. On the Pareto Distribution of Sourceforge Projects. Cambridge, UK: Cambridge University; 2003. URL: http://www.ifm.eng.cam.ac.uk/ people/fhh10/Sourceforge/Sourceforge%20paper.pdf [accessed 2007 Apr 16]
69. Dinh-Trong T, Bieman J. The FreeBSD project: a replication case study of open source development. IEEE T Software Eng 2005;31(6):481-494.
70. Stark JE. Peer Reviews in Open-Source Software Development. Australia: Griffith University; 2001. URL: http://ecommerce.cit.gu.edu.au/ict/docs/theses/ JStark_Dissertation_OSS.pdf [accessed 2007 Apr 16]
71. SourceForge Home page. sourceforge.net. URL: http://sourceforge.net/ [accessed 2007 Apr 16]
72. FreshMeat Home page. freshmeat.net. URL: http://freshmeat.net/ [accessed 2007 Apr 16]
73. BioMed Central Home page. biomedcentral.com. URL: http://www. biomedcentral.com/ [accessed 2007 Apr 16]
74. Koru AG, Tian J. Defect Handling in Medium and Large Open Source Projects. IEEE Software 2004;21(4):54-61.
75. Kreft I, Yoon B. Are multilevel techniques necessary? An attempt at demystification. Annual Meeting of the American Educational Research Association; April 4-8,. New Orleans, LA; 1994. URL: http://eric.ed.gov/ ERICDocs/data/ericdocs2/content_storage_01/0000000b/80/26/c1/54.pdf [accessed 2007 Apr 16]
76. Maas C, Hox J. Sufficient Sample Sizes for Multilevel Modeling. Methodology 2005;1(3):86-92.
77. Hosmer D, Lemeshow S. Applied Logistic Regression. New York: John Wiley & Sons; 1989.
78. Schonlau M, Fricker RD, Elliott MN. Conducting Research Surveys via E-mail and the Web. Santa Monica, CA: RAND Corporation; 2002.
79. Hars A, Ou S. Working for free? Motivations for participating in open source projects. Int J Electron Comm 2002;6(3):25-39.
80. Hars A, Ou S. Working for free? Motivations of participating in open source projects. Proceedings of the 34th Hawaii International Conference on System Sciences 2001;7:7014.
81. Lakhani K, Wolf R. Why hackers do what they do: understanding motivation and effort in free/open source software projects. In: Feller J, Fitzgerald B, Hissam S, Lakhani K, editors. Perspectives on Free and Open Source Software. Cambridge, MA; The MIT Press; 2005.
82. Lindner JR, Murphy TH, Briers GE. Handling nonresponse in social science research. J Agr Educ 2001;42(4):43-53.
83. Siegel S, Castellan J. Nonparametric Statistics for the Behavioral Sciences. New York: McGraw Hill; 1988.
84. Eysenbach G. Improving the quality of Web surveys: the Checklist for Reporting Results of Internet E-Surveys (CHERRIES). J Med Internet Res 2004 Sep 29;6(3):e34
85. Serlin R, Levin J. Teaching how to derive directly interpretable coding schemes for multiple regression analysis. J Educ Stat 1985;10(3):223-238.
86. Wendorf C. Primer on multiple regression coding: Common forms and the additional case of repeated contrasts. Understand Stat 2004;3(1):47-57.
87. Robles G, Scheider H, Tretkowski I, Weber N. Who is doing it? A research on Libre Software developers. Berlin, Germany: Technical University of Berlin; 2001. URL: http://widi.berlios.de/paper/study.pdf [accessed 2007 Apr 16]
88. Lakhani K, Wolf B, Bates J, DiBona C. The Boston Consulting Group hacker survey. O'Reilly Open Source Conference; July 22-26,. 2002. URL: http://www.ostg.com/bcg/BCGHACKERSURVEY.pdf [accessed 2007 Apr 16]
89. Hertel G, Niedner S, Herrmann S. Motivation of software developers in open source projects: an Internet-based survey of contributors to the Linux kernel. Res Pol 2003;32(7):1159-1177.
90. Briand L, El Emam K, Laitenberger O, Fussbroich T. Using simulation to build inspection efficiency benchmarks for development projects. Proceedings of the 20th International Conference on Software Engineering 1998:340-349.
91. Wohlin C, Aurum A, Petersson H, Shull F, Ciolkowski M. Software inspection benchmarking - a qualitative and quantitative comparative opportunity. METRICS '02 Proceedings of the 8th International Symposium on Software Metrics 2002:118-127.
92. Runeson P, Wohlin C. An experimental evaluation of an experience-based capture-recapture method in software code inspections. Empir Softw Eng 1998;3(4):381-406.
93. Land LPW, Sauer C, Jeffery R. Validating the defect detection performance advantage of group designs for software reviews: report of a laboratory experiment using program code. In: ESEC '97/FSE-5: Proceedings of the 6th European conference held jointly with the 5th ACM SIGSOFT international symposium on Foundations of software engineering 1997:294-309-294-309.
94. Porter A, Siy H, Toman C, Votta L. An experiment to assess the cost-benefits of code inspections in large scale software development. Proceedings of the 3rd ACM SIGSOFT Symposium on Foundations of Software Engineering 1995;:92-103.
95. The Good Automated Manufacturing Practice (GAMP) Guide for Validation of Automated Systems in Pharmaceutical Manufacture. Tampa, FL: International Society for Pharmaceutical Engineering; 2002.