On fault representativeness of software fault injection

IEEE Transactions on Software Engineering

Volumn 39, Issue 1, 2013, Pages 80-96

  Natella, Roberto ^a   Cotroneo, Domenico ^a   Duraes, Joao A ^b   Madeira, Henrique S ^c  

^a UNIVERSITY OF NAPLES FEDERICO II   (Italy)

^b POLYTECHNIC INSTITUTE OF COIMBRA   (Portugal)

^c UNIVERSITY OF COIMBRA   (Portugal)

Author keywords

experimental dependability evaluation; fault tolerant systems; Software fault injection; software reliability

Indexed keywords

CLASSIFICATION ALGORITHM; COMPLEX SOFTWARE; DEPENDABILITY EVALUATION; DIFFERENT DISTRIBUTIONS; EXPERIMENTAL STUDIES; FAULT-TOLERANT SYSTEMS; HARDWARE FAULTS; REAL SYSTEMS; REGRESSION TESTS; SOFTWARE COMPONENT; SOFTWARE FAULT; STATE-OF-THE-ART APPROACH;

SOFTWARE RELIABILITY;

FAULT TOLERANCE;

EID: 84872039034     PISSN: 00985589     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSE.2011.124     Document Type: Article

References (62)

1. J. Gray, "Why Do Computers Stop and What Can Be Done About It?" Tandem TR85.7, June 1985.
2. R. Chillarege, I.S. Bhandari, J.K. Chaar, M.J. Halliday, D.S. Moebus, B.K. Ray, and M.-Y. Wong, "Orthogonal Defect Classification-A Concept for In-Process Measurements," IEEE Trans. Software Eng., vol. 18, no. 11, pp. 943-956, Nov. 1992. (Pubitemid 23608410)
3. J. Musa, Software Reliability Engineering. McGraw-Hill, 1996.
4. E.J. Weyuker, "Testing Component-Based Software: A Cautionary Tale," IEEE Software, vol. 15, no. 5, pp. 54-59, Sept./Oct. 1998. (Pubitemid 128610815)
5. J.C. Knight, "Safety Critical Systems: Challenges and Directions" Proc. 24th Int'l Conf. Software Eng., pp. 547-550, 2002. (Pubitemid 35009140)
6. NASA Software Safety Guidebook, NASA-GB-8719.13, 2004.
7. Int'l Organization for Standardization, "Product Development: Software Level," ISO 26262-6, 2011.
8. A. Avizienis, "The N-Version Approach to Fault-Tolerant Software," IEEE Trans. Software Eng., vol. 11, no. 12, pp. 1491-1501, Dec. 1985.
9. M.R. Lyu, Software Fault Tolerance. John Wiley & Sons, 1995.
10. F. Cristian, "Exception Handling and Software Fault Tolerance," IEEE Trans. Computers, vol. 31, no. 6, pp. 531-540, June 1982. (Pubitemid 12525324)
11. J. Arlat, M. Aguera, L. Amat, Y. Crouzet, J.C. Fabre, J.C. Laprie, E. Martin, and D. Powell, "Fault Injection for Dependability Validation: A Methodology and Some Applications," IEEE Trans. Software Eng., vol. 16, no. 2, pp. 166-182, Feb. 1990. (Pubitemid 20666028)
12. J. Voas, F. Charron, G. McGraw, K. Miller, and M. Friedman, "Predicting How Badly 'Good' Software Can Behave," IEEE Software, vol. 14, no. 4, pp. 73-83, July/Aug. 1997. (Pubitemid 127551621)
13. J. Christmansson and R. Chillarege, "Generation of an Error Set That Emulates Software Faults Based on Field Data," Proc. 26th IEEE Fault Tolerant Computing Symp., pp. 304-313, 1996.
14. R.G. Hamlet, "Testing Programs with the Aid of a Compiler," IEEE Trans. Software Eng., vol. 3, no. 4, pp. 279-290, July 1977.
15. R.A. DeMillo, R.J. Lipton, and F.G. Sayward, "Hints on Test Data Selection: Help for the Practicing Programmer," Computer, vol. 11, no. 4, pp. 34-41, Apr. 1978.
16. K.N. King and A.J. Offutt, "A Fortran Language System for Mutation-Based Software Testing," Software: Practice and Experience, vol. 21, no. 7, pp. 685-718, July 1991. (Pubitemid 21666424)
17. J.A. Duraes and H. Madeira, "Emulation of Software Faults: A Field Data Study and a Practical Approach," IEEE Trans. Software Eng., vol. 32, no. 11, pp. 849-867, Nov. 2006. (Pubitemid 46437706)
18. N.E. Fenton and N. Ohlsson, "Quantitative Analysis of Faults and Failures in a Complex Software System," IEEE Trans. Software Eng., vol. 26, no. 8, pp. 797-814, Aug. 2000.
19. D. Powell, E. Martins, J. Arlat, and Y. Crouzet, "Estimators for Fault Tolerance Coverage Evaluation," IEEE Trans. Computers, vol. 44, no. 2, pp. 261-274, Feb. 1995.
20. M. Cukier, D. Powell, and J. Arlat, "Coverage Estimation Methods for Stratified Fault-Injection," IEEE Trans. Computers, vol. 48, no. 7, pp. 707-723, July 1999.
21. J.H. Andrews, L.C. Briand, and Y. Labiche, "Is Mutation an Appropriate Tool for Testing Experiments?" Proc. 27th Int'l Conf. Software Eng., pp. 402-411, 2005.
22. H. Do and G. Rothermel, "On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques," IEEE Trans. Software Eng., vol. 32, no. 9, pp. 733-752, Sept. 2006. (Pubitemid 44604891)
23. R. Geist, A.J. Offutt, and F.C. Harris Jr., "Estimation and Enhancement of Real-Time Software Reliability through Mutation Analysis," IEEE Trans. Computers, vol. 41, no. 5, pp. 550-558, May 1992.
24. Y. Jia and M. Harman, "An Analysis and Survey of the Development of Mutation Testing," IEEE Trans. Software Eng., vol. 37, no. 5, pp. 649-678, Sept./Oct. 2011.
25. A.J. Offutt, G. Rothermel, and C. Zapf, "An Experimental Evaluation of Selective Mutation," Proc. 15th Int'l Conf. Software Eng., pp. 100-107, 1993.
26. A.J. Offutt, A. Lee, G. Rothermel, R.H. Untch, and C. Zapf, "An Experimental Determination of Sufficient Mutant Operators," ACM Trans. Software Eng. and Methodology, vol. 5, no. 2, pp. 99-118, Apr. 1996. (Pubitemid 126418670)
27. W.E. Wong and A.P. Mathur, "Reducing the Cost of Mutation Testing: An Empirical Study," J. Systems and Software, vol. 31, no. 3, pp. 185-196, Dec. 1995.
28. M. Sridharan and A.S. Namin, "Prioritizing Mutation Operators Based on Importance Sampling," Proc. 21st Int'l Symp. Software Reliability Eng., pp. 378-387, 2010.
29. W.T. Ng and P.M. Chen, "The Design and Verification of the Rio File Cache," IEEE Trans. Computers, vol. 50, no. 4, pp. 322-337, Apr. 2001. (Pubitemid 32454186)
30. S. Chandra and P.M. Chen, "How Fail-Stop Are Faulty Programs?" Proc. IEEE 28th Fault Tolerant Computing Symp., 1998.
31. J.J. Hudak, B.H. Suh, D.P. Siewiorek, and Z. Segall, "Evaluation and Comparison of Fault-Tolerant Software Techniques," IEEE Trans. Reliability, vol. 42, no. 2, pp. 190-204, June 1993. (Pubitemid 23703395)
32. M. Vieira and H. Madeira, "A Dependability Benchmark for OLTP Application Environments," Proc. 29th Int'l Conf. Very Large Data Bases, pp. 742-753, 2003.
33. A. Avizienis, J.-C. Laprie, B. Randell, and C. Landwehr, "Basic Concepts and Taxonomy of Dependable and Secure Computing," IEEE Trans. Dependable and Secure Computing, vol. 1, no. 1, pp. 11-33, Jan.-Mar. 2004. (Pubitemid 40134576)
34. D. Stott, B. Floering, D. Burke, Z. Kalbarczyk, and R. Iyer, "NFTAPE: A Framework for Assessing Dependability in Distributed Systems with Lightweight Fault Injectors," Proc. Int'l Conf. Computer Performance and Dependability Symp., pp. 91-100, 2000. (Pubitemid 30578193)
35. J. Aidemark, J. Vinter, P. Folkesson, and J. Karlsson, "GOOFI: Generic Object-Oriented Fault Injection Tool," Proc. Int'l Conf. Dependable Systems and Networks, pp. 83-88, 2001.
36. Z. Segall, D. Vrsalovic, D. Siewiorek, J. Kownacki, J. Barton, R. Dancey, A. Robinson, and T. Lin, "FIAT-Fault Injection Based Automated Testing Environment" Proc. IEEE 18th Fault Tolerant Computing Symp., pp. 102-107, 1988.
37. M.-C. Hsueh, T.K. Tsai, and R.K. Iyer, "Fault Injection Techniques and Tools," Computer, vol. 30, no. 4, pp. 75-82, Apr. 1997. (Pubitemid 127546072)
38. J. Carreira, H. Madeira, and J.G. Silva, "Xception: A Technique for the Experimental Evaluation of Dependability in Modern Computers," IEEE Trans. Software Eng., vol. 24, no. 2, pp. 125-136, Feb. 1998. (Pubitemid 128740270)
39. R. Moraes, R. Barbosa, J.A. Duraes, N. Mendes, E. Martins, and H. Madeira, "Injection of Faults at Component Interfaces and Inside the Component Code: Are They Equivalent?" Proc. Sixth European Dependable Computing Conf., pp. 53-64, 2006.
40. P. Koopman and J. DeVale, "The Exception Handling Effectiveness of POSIX Operating Systems," IEEE Trans. Software Eng., vol. 26, no. 9, pp. 837-848, Sept. 2000.
41. A.K. Ghosh, M. Schmid, and V. Shah, "Testing the Robustness of Windows NT Software," Proc. Ninth IEEE Int'l Symp. Software Reliability Eng., pp. 231-236, 1998.
42. J. Arlat, J.-C. Fabre, M. Rodriguez, and F. Salles, "Dependability of COTS Microkernel-Based Systems," IEEE Trans. Computers, vol. 51, no. 2, pp. 138-163, Feb. 2002. (Pubitemid 34198183)
43. E. Martins, C.M.F. Rubira, and N.G.M. Leme, "Jaca: A Reflective Fault Injection Tool Based on Patterns" Proc. IEEE Int'l Conf. Dependable Systems and Networks, pp. 483-487, 2002.
44. W.-L. Kao, R.K. Iyer, and D. Tang, "FINE: A Fault Injection and Monitoring Environment for Tracing the UNIX System Behavior under Faults," IEEE Trans. Software Eng., vol. 19, no. 11, pp. 1105-1118, Nov. 1993.
45. H. Madeira, D. Costa, and M. Vieira, "On the Emulation of Software Faults by Software Fault Injection," Proc. IEEE Int'l Conf. Dependable Systems and Networks, pp. 417-426, 2000.
46. T. Jarboui, J. Arlat, Y. Crouzet, K. Kanoun, and T. Marteau, "Analysis of the Effects of Real and Injected Software Faults: Linux as a Case Study," Proc. Pacific Rim Int'l Symp. Dependable Computing, pp. 51-58, 2002.
47. M.E. Delamaro and J.C. Maldonado, "Proteum-A Tool for the Assessment of Test Adequacy for C Programs," Proc. Conf. Performability in Computer Systems, pp. 79-95, 1996.
48. MySQL Market Share, http://www.mysql.com/why-mysql/marketshare/, Mar. 2011.
49. EnterpriseDB's Postgres Plus Users by Application Type, http://www.enterprisedb.com/customer-success/customers-byapplication-workload, Mar. 2011.
50. J. Rufino, S. Filipe, M. Coutinho, S. Santos, and J. Windsor, "ARINC 653 Interface in RTEMS," Proc. Data Systems in Aerospace Conf., 2007.
51. GCC Documentation, http://gcc.gnu.org/onlinedocs/gcc/, 2012.
52. QEMU Virtualization Software, http://qemu.org, 2012.
53. SAFE Software Fault Emulation Tool, http://www.mobilab.unina.it/SFI.htm, 2012.
54. TPC-C OLTP Benchmark, http://www.tpc.org/tpcc/, 2012.
55. D. Sheskin, Handbook of Parametric and Nonparametric Statistical Procedures. Chapman and Hall/CRC, 2004.
56. N.E. Fenton and S.L. Pfleeger, Software Metrics: A Rigorous and Practical Approach. PWS Publishing Co., 1998.
57. V.R. Basili and B.T. Perricone, "Software Errors and Complexity: An Empirical Investigation," Comm. ACM, vol. 27, no. 1, pp. 42-52, Jan. 1984.
58. S. Henry and D. Kafura, "Software Structure Metrics Based on Information Flow," IEEE Trans. Software Eng., vol. 7, no. 5, pp. 510-518, Sept. 1981. (Pubitemid 12439180)
59. SciTools Understand Tool, http://www.scitools.com, 2012.
60. T. Menzies, J. Greenwald, and A. Frank, "Data Mining Static Code Attributes to Learn Defect Predictors," IEEE Trans. Software Eng., vol. 33, no. 1, pp. 2-13, Jan. 2007. (Pubitemid 46002165)
61. K. Srinivasan and D. Fisher, "Machine Learning Approaches to Estimating Software Development Effort," IEEE Trans. Software Eng., vol. 21, no. 2, pp. 126-137, Feb. 1995.
62. I. Witten and E. Frank, Data Mining: Practical Machine Learning Tools and Techniques. Morgan Kaufmann, 2005.