Adaptive random testing: An illusion of effectiveness?

2011 International Symposium on Software Testing and Analysis, ISSTA 2011 - Proceedings

Volumn , Issue , 2011, Pages 265-275

  Arcuri, Andrea ^a   Briand, Lionel ^a,b  

^a SIMULA RESEARCH LABORATORY   (Norway)

^b UNIVERSITY OF OSLO   (Norway)

Author keywords

distance; diversity; F measure; faulty region; P measure; random testing; shape; similarity

Indexed keywords

DISTANCE; DIVERSITY; F-MEASURE; FAULTY REGION; P-MEASURE; RANDOM TESTING; SHAPE; SIMILARITY;

COMPUTER SOFTWARE SELECTION AND EVALUATION; TESTING;

SOFTWARE TESTING;

EID: 80051947344     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2001420.2001452     Document Type: Conference Paper

References (44)

1. P. E. Ammann and J. C. Knight. Data diversity: An approach to software fault tolerance. IEEE Transations on Computers, 37(4):418-425, 1988.
2. J. H. Andrews, L. C. Briand, Y. Labiche, and A. S. Namin. Using mutation analysis for assessing and comparing testing coverage criteria. IEEE Transactions on Software Engineering (TSE), 32(8):608-624, 2006.
3. A. Arcuri and L. Briand. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In ACM/IEEE International Conference on Software Engineering (ICSE), 2011.
4. A. Arcuri, M. Z. Iqbal, and L. Briand. Black-box system testing of real-time embedded systems using random and search-based testing. In IFIP International Conference on Testing Software and Systems (ICTSS), pages 95-110, 2010.
5. A. Arcuri, M. Z. Iqbal, and L. Briand. Formal analysis of the effectiveness and predictability of random testing. In ACM International Symposium on Software Testing and Analysis (ISSTA), pages 219-229, 2010.
6. T. Chen, F. Kuo, and H. Liu. Adaptive random testing based on distribution metrics. Journal of Systems and Software (JSS), 82(9):1419-1433, 2009.
7. T. Y. Chen, D. Huang, F. Kuo, R. Merkel, and J. Mayer. Enhanced lattice-based adaptive random testing. In Proceedings of the ACM symposium on Applied Computing, pages 422-429, 2009.
8. T. Y. Chen, F. Kuo, and Z. Zhou. On favourable conditions for adaptive random testing. International Journal of Software Engineering and Knowledge Engineering, 17(6):805-825, 2007.
9. T. Y. Chen, F. C. Kuo, and H. Liu. Distributing test cases more evenly in adaptive random testing. Journal of Systems and Software (JSS), 81(12):2146-2162, 2008.
10. T. Y. Chen, F. C. Kuo, and H. Liu. Application of a failure driven test profile in random testing. IEEE Transactions on Reliability, 58(1):179-192, 2009.
11. T. Y. Chen, F. C. Kuo, R. G. Merkel, and S. Ng. Mirror adaptive random testing. Information and Software Technology (IST), 46(15):1001-1010, 2004.
12. T. Y. Chen, F. Kuoa, R. G. Merkela, and T. Tseb. Adaptive random testing: The art of test case diversity. Journal of Systems and Software (JSS), 2010. (in press).
13. T. Y. Chen, H. Leung, and I. K. Mak. Adaptive random testing. In Advances in Computer Science, pages 320-329, 2004.
14. T. Y. Chen and R. Merkel. An upper bound on software testing effectiveness. 17(3):1-27, 2008.
15. I. Ciupa, A. Leitner, M. Oriol, and B. Meyer. Artoo: adaptive random testing for object-oriented software. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 71-80, 2008.
16. T. H. Cormen, C. E. Leiserson, R. L. Rivest, and C. Stein. Introduction to Algorithms. MIT Press and McGraw-Hill, second edition, 2001.
17. J. W. Duran and S. C. Ntafos. An evaluation of random testing. IEEE Transactions on Software Engineering (TSE), 10(4):438-444, 1984.
18. W. Feller. An Introduction to Probability Theory and Its Applications, Vol. 1. Wiley, 3 edition, 1968.
19. G. B. Finelli. Nasa software failure characterization experiments. Reliability Engineering & System Safety, 32:155-169, 1991.
20. P. Godefroid, N. Klarlund, and K. Sen. Dart: directed automated random testing. In ACM Conference on Programming language design and implementation (PLDI), pages 213-223, 2005.
21. D. E. Goldberg. Genetic Algorithms in Search and Optimization. Addison-wesley, 1989.
22. R. Grissom and J. Kim. Effect sizes for research: A broad practical approach. Lawrence Erlbaum, 2005.
23. M. Harman, S. A. Mansouri, and Y. Zhang. Search based software engineering: A comprehensive analysis and review of trends techniques and applications. Technical Report TR-09-03, King's College, 2009.
24. H. Hemmati, A. Arcuri, and L. Briand. Reducing the cost of model-based testing through test case diversity. In IFIP International Conference on Testing Software and Systems (ICTSS), 2010.
25. B. Jiang, Z. Zhang, W. Chan, and T. Tse. Adaptive random test case prioritization. In IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 233-244, 2009.
26. F. C. Kuo. An indepth study of mirror adaptive random testing. In International Conference on Quality Software, pages 51-58, 2009.
27. F. C. Kuo, T. Y. Chen, H. Liu, and W. K. Chan. Enhancing adaptive random testing for programs with high dimensional input domains or failure-unrelated parameters. Software Quality Journal, 16(3):303-327, 2008.
28. Z. Li, M. Harman, and R. M. Hierons. Meta-heuristic search algorithms for regression test case prioritization. IEEE Transactions on Software Engineering (TSE), 33(4):225-237, 2007.
29. Y. Lin, X. Tang, Y. Chen, and J. Zhao. A divergence-oriented approach to adaptive random testing of java programs. In IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 221-232, 2009.
30. Y. S. Ma, J. Offutt, and Y. R. Kwon. Mujava: an automated class mutation system. Software Testing, Verification and Reliability, 15(2):97-133, 2005. (Pubitemid 40858060)
31. N. Mansour, R. Bahsoon, and G. Baradhi. Empirical comparison of regression test selection algorithms. Journal of Systems and Software, 57(1):79-90, 2001.
32. J. Mayer. Lattice-based adaptive random testing. In IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 333-336, 2005.
33. J. Mayer and C. Schneckenburger. An empirical analysis and comparison of random testing techniques. In ACM/IEEE International symposium on Empirical Software Engineering, pages 105-114, 2006.
34. G. Myers. The Art of Software Testing. Wiley, New York, 1979.
35. C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball. Feedback-directed random test generation. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 75-84, 2007.
36. W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery. Numerical recipes: the art of scientific computing. Cambridge University Press, 2007.
37. R Development Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2008. ISBN 3-900051-07-0.
38. C. Schneckenburger and J. Mayer. Towards the determination of typical failure patterns. In International workshop on Software quality assurance, pages 90-93, 2007.
39. S. Shin, S. Park, K. Choi, and K. Jung. Normalized Adaptive Random Test for Integration Tests. In IEEE International Workshop on Software Test Automation, 2010.
40. A. F. Tappenden and J. Miller. A novel evolutionary approach for adaptive random testing. IEEE Transactions On Reliability, 58(4):619-633, 2009.
41. T. A. Thayer, M. Lipow, and E. C. Nelson. Software Reliability. North Holland, Amsterdam, 1978.
42. L. J. White and E. I. Cohen. A domain strategy for computer program testing. IEEE Transactions on Software Engineering (TSE), 6(3):247-257, 1980.
43. S. H. Wu, S. Jandhyala., Y. K. Malaiya, and A. P. Jayasumana. Antirandom testing: a distance-based approach. VLSI Design, 10:1-9, 2008.
44. Z. Zhou. Using Coverage Information to Guide Test Case Selection in Adaptive Random Testing. In IEEE International Workshop on Software Test Automation, 2010.