Random testing: Theoretical results and practical implications

IEEE Transactions on Software Engineering

Volumn 38, Issue 2, 2012, Pages 258-277

  Arcuri, Andrea ^a   Iqbal, Muhammad Zohaib ^a   Briand, Lionel ^a  

^a SIMULA RESEARCH LABORATORY   (Norway)

Author keywords

adaptive random testing; Coupon collector; partition testing; predictability; random testing; Schur function; theory

Indexed keywords

ADAPTIVE RANDOM TESTING; COUPON COLLECTOR; PARTITION TESTING; PREDICTABILITY; RANDOM TESTING; SCHUR FUNCTION; THEORY;

TESTING;

SOFTWARE TESTING;

EID: 84859716089     PISSN: 00985589     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSE.2011.121     Document Type: Article

References (52)

1. J.H. Andrews, A. Groce, M. Weston, and R.G. Xu, "Random Test Run Length and Effectiveness," Proc. IEEE/ACM Int'l Conf. Automated Software Eng., pp. 19-28, 2008
2. A. Arcuri, "Theoretical Analysis of Local Search in Software Testing," Proc. Symp. Stochastic Algorithms, Foundations and Applications, pp. 156-168, 2009
3. A. Arcuri, "A Theoretical and Empirical Analysis of the Role of Test Sequence Length in Software Testing for Structural Coverage," IEEE Trans. Software Eng., http://doi.ieeecomputersociety. org/10.1109/TSE.2011.44, 2011
4. A. Arcuri and L. Briand, "Adaptive Random Testing: An Illusion of Effectiveness?" Proc. ACM Int'l Symp. Software Testing and Analysis, pp. 265-275, 2011
5. A. Arcuri and L. Briand, "Formal Analysis of The Probability of Interaction Fault Detection using Random Testing," IEEE Trans. Software Eng., doi:10.1109/TSE.2011.85, 2011
6. A. Arcuri and L. Briand, "A Practical Guide for Using Statistical Tests to Assess Randomized Algorithms in Software Engineering," Proc. ACM/IEEE Int'l Conf. Software Eng., pp. 1-10, 2011
7. A. Arcuri, M.Z. Iqbal, and L. Briand, "Black-Box System Testing of Real-Time Embedded Systems Using Random and Search-Based Testing," Proc. IFIP Int'l Conf. Testing Software and Systems, pp. 95-110, 2010
8. A. Arcuri, M.Z. Iqbal, and L. Briand, "Formal Analysis of the Effectiveness and Predictability of Random Testing," Proc. ACM Int'l Symp. Software Testing and Analysis, pp. 219-229, 2010
9. P.J. Boland, H. Singh, and B. Cukic, "Comparing Partition and Random Testing via Majorization and Schur Functions," IEEE Trans. Software Eng., vol. 29, no. 1, pp. 88-94, Jan. 2003
10. T.Y. Chen, F.C. Kuo, and R. Merkel, "On the Statistical Properties of Testing Effectiveness Measures," J. Systems and Software, vol. 79, no. 5, pp. 591-601, 2006
11. T.Y. Chen and Y.T. Yu, "On the Relationship between Partition and Random Testing," IEEE Trans. Software Eng., vol. 20, no. 12, pp. 877-980, Dec. 1994
12. T.Y. Chen and Y.T. Yu, "On the Expected Number of Failures Detected by Subdomain Testing and Random Testing," IEEE Trans. Software Eng., vol. 22, no. 2, pp. 109-119, Feb. 1996 (Pubitemid 126771661)
13. I. Ciupa, A. Pretschner, M. Oriol, A. Leitner, and B. Meyer, "On the Number and Nature of Faults Found by Random Testing," Software Testing, Verification and Reliability, vol. 21, pp. 3-28, 2009
14. T.H. Cormen, C.E. Leiserson, R.L. Rivest, and C. Stein, Introduction to Algorithms, second ed. MIT Press and McGraw-Hill, 2001
15. R.A. DeMillo, R.J. Lipton, and F. Sayward, "Hints on Test Data Selection: Help for the Practicing Programmer," Computer, vol. 11, no. 4, pp. 34-41, Apr. 1978 (Pubitemid 8631038)
16. J.W. Duran and S.C. Ntafos, "An Evaluation of Random Testing," IEEE Trans. Software Eng., vol. 10, no. 4, pp. 438-444, July 1984
17. W. Feller, An Introduction to Probability Theory and Its Applications, third ed., vol. 1. Wiley, 1968
18. G. Fraser and A. Arcuri, "Evolutionary Generation of Whole Test Suites," Proc. Int'l Conf. Quality Software, pp. 31-40, 2011
19. P. Godefroid, N. Klarlund, and K. Sen, "Dart: Directed Automated Random Testing," Proc. ACM Conf. Programming Language Design and Implementation, pp. 213-223, 2005
20. A. Groce, G. Holzmann, and R. Joshi, "Randomized Differential Testing as a Prelude to Formal Verification," Proc. ACM/IEEE Int'l Conf. Software Eng., pp. 621-631, 2007
21. W.J. Gutjahr, "Optimal Test Distributions for Software Failure Cost Estimation," IEEE Trans. Software Eng., vol. 21, no. 3, pp. 219-228, Mar. 1995
22. W.J. Gutjahr, "Partition Testing vs. Random Testing: The Influence of Uncertainty," IEEE Trans. Software Eng., vol. 25, no. 5, pp. 661-674, Sept./Oct. 1999 (Pubitemid 30541839)
23. D. Hamlet and R. Taylor, "Partition Testing Does Not Inspire Confidence," IEEE Trans. Software Eng., vol. 16, no. 12, pp. 1402-1411, Dec. 1990
24. R. Hamlet, "Random Testing," Encyclopedia of Software Eng,. pp. 970-978, Wiley, 1994
25. M. Harman, "The Current State and Future of Search Based Software Engineering," Proc. Future of Software Eng., pp. 342-357, 2007 (Pubitemid 47484936)
26. M. Harman, F. Islam, T. Xie, and S. Wappler, "Automated Test Data Generation for Aspect-Oriented Programs," Proc. Eighth ACM Int'l Conf. Aspect-Oriented Software Development, pp. 185-196, 2009
27. M. Harman, S.G. Kim, K. Lakhotia, P. McMinn, and S. Yoo, "Optimizing for the Number of Tests Generated in Search Based Test Data Generation with an Application to the Oracle Cost Problem," Proc. Int'l Workshop Search-Based Software Testing, 2010
28. M. Harman and P. McMinn, "A Theoretical and Empirical Study of Search Based Testing: Local, Global and Hybrid Search," IEEE Trans. Software Eng., vol. 36, no. 2, pp. 226-247, Mar./Apr. 2010
29. L. Holst, "Extreme Value Distributions for Random Coupon Collector and Birthday Problems," Extremes, vol. 4, no. 2, pp. 129-145, 2001
30. M.Z. Iqbal, A. Arcuri, and L. Briand, "Environment Modeling with UML/MARTE to Support Black-Box System Testing for Real-Time Embedded Systems: Methodology and Industrial Case Studies," Proc. ACM/IEEE Int'l Conf. Model Driven Eng. Languages and Systems, pp. 286-300, 2010
31. D. Knuth, The Art of Computer Programming, vol. 3. Addison Wesley 1973
32. M. Lyu, Handbook of Software Reliability Engineering. McGraw-Hill, Inc., 1996
33. M. Matsumoto and T. Nishimura, "Mersenne Twister: A 623-Dimensionally Equidistributed Uniform Pseudo-Random Number Generator," ACM Trans. Modeling and Computer Simulation, vol. 8, no. 1, pp. 23-30, 1998
34. P. McMinn, "Search-Based Software Test Data Generation: A Survey," Software Testing, Verification and Reliability, vol. 14, no. 2, pp. 105-156, 2004
35. M. Motwani and P. Raghavan, Randomized Algorithms. Cambridge Univ. Press, 1995
36. G. Myers, The Art of Software Testing. Wiley, 1979
37. C. Pacheco, S.K. Lahiri, M.D. Ernst, and T. Ball, "Feedback-Directed Random Test Generation," Proc. ACM/IEEE Int'l Conf. Software Eng., pp. 75-84, 2007 (Pubitemid 47430643)
38. R Development Core Team, R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, 2008
39. J.A. Rice, Mathematical Statistics and Data Analysis, second ed. Duxbury Press, 1994
40. B. Rosén, "Asymptotic Normality in a Coupon Collector's Problem," Probability Theory and Related Fields, vol. 13, no. 3, pp. 256-279, 1969
41. R. Sharma, M. Gligoric, A. Arcuri, G. Fraser, and D. Marinov, "Testing Container Classes: Random or Systematic?" Proc. Fundamental Approaches to Software Eng., 2011
42. S. Shioda, "Some Upper and Lower Bounds on the Coupon Collector Problem," J. Computational and Applied Math., vol. 200, no. 1, pp. 154-167, 2007 (Pubitemid 44738300)
43. P. Thévenod-Fosse and H. Waeselynck, "An Investigation of Statistical Software Testing," Software Testing, Verification and Reliability, vol. 1, no. 2, pp. 15-25, 1991
44. N. Tillmann and N.J. de Halleux, "Pex-White Box Test Generation for .NET," Proc. Int'l Conf. Tests And Proofs, pp. 134-253, 2008
45. P. Tonella, "Evolutionary Testing of Classes," Proc. ACM Int'l Symp. Software Testing and Analysis, pp. 119-128, 2004 (Pubitemid 41121383)
46. M.Z. Tsoukalas, J.W. Duran, and S.C. Ntafos, "On Some Reliability Estimation Problems in Random and Partition Testing," IEEE Trans. Software Eng., vol. 19, no. 7, pp. 687-697, July 1993
47. T. Vos, A. Baars, F. Lindlar, P. Kruse, A. Windisch, and J. Wegener, "Industrial Scaled Automated Structural Testing with the Evolutionary Testing Tool," Proc. IEEE Int'l Conf. Software Testing, Verification and Validation, pp. 175-184, 2010
48. J. Wegener, A. Baresel, and H. Sthamer, "Evolutionary Test Environment for Automatic Structural Testing," Information and Software Technology, vol. 43, no. 14, pp. 841-854, 2001 (Pubitemid 33050964)
49. E.J. Weyuker and B. Jeng, "Analyzing Partition Testing Strategies," IEEE Trans. Software Eng., vol. 17, no. 7, pp. 703-711, July 1991 (Pubitemid 21674681)
50. E.J. Weyuker and T.J. Ostrand, "Theories of Program Testing and the Application of Revealing Subdomains," IEEE Trans. Software Eng., vol. 6, no. 3, pp. 236-246, May 1980
51. D. White, A. Arcuri, and J. Clark, "Evolutionary Improvement of Programs," IEEE Trans. Evolutionary Computation, vol. 15, no. 4, pp. 515-538, Aug. 2011
52. Y. Zhan and J.A. Clark, "A Search-Based Framework for Automatic Testing of Matlab/Simulink Models," J. Systems and Software, vol. 81, no. 2, pp. 262-285, 2008. (Pubitemid 350235613)