The effects of time constraints on test case prioritization: A series of controlled experiments

IEEE Transactions on Software Engineering

Volumn 36, Issue 5, 2010, Pages 593-617

  Do, Hyunsook ^a   Mirarab, Siavash ^b   Tahvildari, Ladan ^c   Rothermel, Gregg ^d  

^a North Dakota State University   (Canada)

^b IBM CANADA LTD   (Canada)

^c UNIVERSITY OF WATERLOO   (Canada)

^d UNIVERSITY OF NEBRASKA LINCOLN   (United States)

Author keywords

Bayesian networks; cost benefits; empirical studies; Regression testing; test case prioritization

Indexed keywords

CONTROLLED EXPERIMENT; COSTS AND BENEFITS; EMPIRICAL STUDIES; MODIFIED SOFTWARE; PRIORITIZATION; REGRESSION TESTING; REGRESSION TESTS; SOFTWARE DEVELOPMENT PROCESS; SOFTWARE SYSTEMS; TEST CASE; TEST CASE PRIORITIZATION; TEST ENGINEERS; TESTING PROCESS; TIME CONSTRAINTS;

BAYESIAN NETWORKS; COMPUTER SOFTWARE MAINTENANCE; COST EFFECTIVENESS; COSTS; DISTRIBUTED PARAMETER NETWORKS; INFERENCE ENGINES; INTELLIGENT NETWORKS; REGRESSION ANALYSIS; SOFTWARE DESIGN; SOFTWARE TESTING; STATISTICAL TESTS;

EXPERIMENTS;

EID: 77957752029     PISSN: 00985589     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSE.2010.58     Document Type: Article

References (62)

1. S. Alspaugh, K.R. Walcott, M. Belanich, G.M. Kapfhammer, and M.L. Soffa, "Efficient Time-Aware Prioritization with Knapsack Solvers," Proc. ACM Int'l Workshop Empirical Assessment of Software Eng. Languages and Technologies, pp. 17-31, Nov. 2007.
2. J.H. Andrews, L.C. Briand, and Y. Labiche, "Is Mutation an Appropriate Tool for Testing Experiments?" Proc. Int'l Conf. Software Eng., pp. 402-411, May 2005.
3. R. Bell, T. Ostrand, and E. Weyuker, "Looking for Bugs in All the Right Places," Proc. Int'l Symp. Software Testing and Analysis, pp. 61-72, July 2006.
4. B. Boehm, "Value-Based Software Engineering," ACM SIGSOFT Software Eng. Notes, vol. 28, no. 2, p. 4, 2003.
5. B. Boehm, C. Abts, A.W. Brown, S. Chulani, E. Horowitz, B.K. Clark, R. Madachy, D. Reifer, and B. Steece, Software Cost Estimation with COCOMO II. Prentice-Hall, 2000.
6. S.R. Chidamber and C.F. Kemerer, "Towards a Metrics Suite for Object Oriented Design," ACM SIGPLAN Notes, vol. 26, no. 11, pp. 197-211, Nov. 1991.
7. C. Collberg, G. Myles, and M. Stepp, "An Empirical Study of Java Bytecode Programs," Technical Report TR04-11, Dept. of Computer Science, Univ. of Arizona, 2004.
8. Culpepper and Assoc., Inc., "Culpepper Compensation and Benefit Surveys," http://www.culpepper.com, 2010.
9. H. Do, S. Elbaum, and G. Rothermel, "Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and Its Potential Impact," Empirical Software Eng.: An Int'l J., vol. 10, no. 4, pp. 405-435, 2005.
10. H. Do, S. Mirarab, L. Tahvildari, and G. Rothermel, "An Empirical Study of the Effect of Time Constraints on the Cost-Benefits of Regression Testing," Proc. ACM SIGSOFT, pp. 71-82, Nov. 2008.
11. H. Do and G. Rothermel, "An Empirical Study of Regression Testing Techniques Incorporating Context and Lifetime Factors and Improved Cost-Benefit Models," Proc. ACM SIGSOFT, pp. 141-151, Nov. 2006.
12. H. Do and G. Rothermel, "On the Use of Mutation Faults in Empirical Assessments of Test Case Prioritization Techniques," IEEE Trans. Software Eng., vol. 32, no. 9, pp. 733-752, Sept. 2006.
13. H. Do and G. Rothermel, "Using Sensitivity Analysis to Create Simplified Economic Models for Regression Testing," Proc. Int'l Symp. Software Testing and Analysis, pp. 51-61, July 2008.
14. H. Do, G. Rothermel, and A. Kinneer, "Empirical Studies of Test Case Prioritization in a JUnit Testing Environment," Proc. Int'l Symp. Software Reliability Eng., pp. 113-124, Nov. 2004.
15. H. Do, G. Rothermel, and A. Kinneer, "Prioritizing JUnit Test Cases: An Empirical Assessment and Cost-Benefits Analysis," Empirical Software Eng.: An Int'l J., vol. 11, no. 1, pp. 33-70, 2006.
16. S. Elbaum, A. Malishevsky, and G. Rothermel, "Prioritizing Test Cases for Regression Testing," Proc. Int'l Symp. Software Testing and Analysis, pp. 102-112, Aug. 2000.
17. S. Elbaum, A. Malishevsky, and G. Rothermel, "Incorporating Varying Test Costs and Fault Severities into Test Case Prioritization," Proc. Int'l Conf. Software Eng., pp. 329-338, May 2001.
18. S. Elbaum, A.G. Malishevsky, and G. Rothermel, "Test Case Prioritization: A Family of Empirical Studies," IEEE Trans. Software Eng., vol. 28, no. 2, pp. 159-182, Feb. 2002.
19. B. Freimut, L.C. Briand, and F. Vollei, "Determining Inspection Cost-Effectiveness by Combining Project Data and Expert Opinion," IEEE Trans. Software Eng., vol. 31, no. 12, pp. 1074-1092, Dec. 2005.
20. M.J. Harrold, D. Rosenblum, G. Rothermel, and E. Weyuker, "Empirical Studies of a Prediction Model for Regression Test Selection," IEEE Trans. Software Eng., vol. 27, no. 3, pp. 248-263, Mar. 2001. (Pubitemid 32297404)
21. S. Hou, L. Zhang, T. Xie, and J. Sun, "Quota-Constrained Test Case Prioritization for Regression Testing of Service-Centric Systems," Proc. Int'l Conf. Software Maintenance, pp. 257-266, Sept. 2008.
22. D. Jeffrey and N. Gupta, "Test Case Prioritization Using Relevant Slices," Proc. Ann. Int'l Computer Software and Applications Conf., pp. 411-420, Sept. 2006.
23. R. Johnson, Elementary Statistics, sixth ed. Duxbury Press, 1992.
24. C. Jones, Applied Software Measurement: Assuring Productivity and Quality. McGraw-Hill, 1997.
25. J. Jones and M.J. Harrold, "Test Suite Reduction and Prioritization for Modified Condition/Decision Coverage," IEEE Trans. Software Eng., vol. 29, no. 3, pp. 193-209, Mar. 2003.
26. J. Kim and A. Porter, "A History-Based Test Prioritization Technique for Regression Testing in Resource Constrained Environments," Proc. Int'l Conf. Software Eng., pp. 119-129, May 2002.
27. A. Kinneer, M. Dwyer, and G. Rothermel, "Sofya: A Flexible Framework for Development of Dynamic Program Analysis for Java Software," Technical Report TR-UNL-CSE-2006-0006, Univ. of Nebraska - Lincoln, Apr. 2006.
28. B. Korel, G. Koutsogiannakis, and L. Tahat, "Application of System Models in Regression Test Suite Prioritization," Proc. Int'l Conf. Software Maintenance, pp. 247-256, Sept. 2008.
29. B. Korel, L. Tahat, and M. Harman, "Test Prioritization Using System Models," Proc. Int'l Conf. Software Maintenance, pp. 559-568, Sept. 2005.
30. S. Kusumoto, K. Matsumoto, T. Kikuno, and K. Torii, "A New Metric for Cost-Effectiveness of Software Reviews," IEICE Trans. Information Systems, vol. E75-D no. 5, pp. 674-680, 1992.
31. D. Leon and A. Podgurski, "A Comparison of Coverage-Based and Distribution-Based Techniques for Filtering and Prioritizing Test Cases," Proc. Int'l Symp. Software Reliability Eng., pp. 442-453, Nov. 2003.
32. H.K.N. Leung and L.J. White, "A Cost Model to Compare Regression Test Strategies," Proc. Int'l Conf. Software Maintenance, pp. 201-208, Oct. 1991.
33. Z. Li, M. Harman, and R.M. Hierons, "Search Algorithms for Regression Test Case Prioritization," IEEE Trans. Software Eng., vol. 33, no. 4, pp. 225-237, Apr. 2007.
34. A. Malishevsky, G. Rothermel, and S. Elbaum, "Modeling the Cost-Benefits Tradeoffs for Regression Testing Techniques," Proc. Int'l Conf. Software Maintenance, pp. 204-213, Oct. 2002.
35. S. Mirarab, "A Bayesian Framework for Software Regression Testing," master's thesis, Dept. of Electrical and Computer Eng., Univ. of Waterloo, Aug. 2008.
36. S. Mirarab and L. Tahvildari, "A Prioritization Approach for Software Test Cases on Bayesian Networks," Proc. Int'l Conf. Fundamental Approaches to Software Eng., pp. 276-290, Mar. 2007.
37. S. Mirarab and L. Tahvildari, "An Empirical Study on Bayesian Network-Based Approach for Test Case Prioritization," Proc. Int'l Conf. Software Testing, Verification, and Validation, pp. 278-287, Apr. 2008.
38. M.M. Muller and F. Padberg, "About the Return on Investment of Test-Driven Development," Proc. Int'l Workshop Economics-Driven Software Eng. Research, pp. 2631-2636, May 2003.
39. K. Onoma, W.-T. Tsai, M. Poonawala, and H. Suganuma, "Regression Testing in an Industrial Environment," Comm. ACM, vol. 41, no. 5, pp. 81-86, 1988.
40. T.J. Ostrand and M.J. Balcer, "The Category-Partition Method for Specifying and Generating Functional Tests," Comm. ACM, vol. 31, no. 6, pp. 676-688, 1988.
41. H. Park, J. Ryu, and J. Baik, "Historical Value-Based Approach for Cost-Cognizant Test Case Prioritization to Improve the Effectiveness of Regression Testing," Proc. Int'l Conf. Secure System Integration and Reliability Improvement, pp. 39-46, July 2008.
42. D.E. Perry and C.S. Stieg, "Software Faults in Evolving a Large, Real-Time System: A Case Study," Proc. European Software Eng. Conf., pp. 48-67, Sept. 1993.
43. J.J. Phillips, Return on Investment in Training and Performance Improvement Programs. Gulf Publishing Company, 1997.
44. X. Qu, M.B. Cohen, and K.M. Woolf, "Combinatorial Interaction Regression Testing: A Study of Test Case Generation and Prioritization," Proc. Int'l Conf. Software Maintenance, pp. 255-264, Oct. 2007.
45. F.L. Ramsey and D.W. Schafer, The Statistical Sleuth. Duxbury Press, 1997.
46. G. Rothermel and M.J. Harrold, "Analyzing Regression Test Selection Techniques," IEEE Trans. Software Eng., vol. 22, no. 8, pp. 529-551, Aug. 1996.
47. G. Rothermel and M.J. Harrold, "A Safe, Efficient Regression Test Selection Technique," ACM Trans. Software Eng. and Methodology, vol. 6, no. 2, pp. 173-210, 1997. (Pubitemid 127684922)
48. G. Rothermel, R. Untch, C. Chu, and M.J. Harrold, "Test Case Prioritization," IEEE Trans. Software Eng., vol. 27, no. 10, pp. 929-948, Oct. 2001.
49. G. Rothermel, R.H. Untch, C. Chu, and M.J. Harrold, "Test Case Prioritization: An Empirical Study," Proc. Int'l Conf. Software Maintenance, pp. 179-188, Aug. 1999.
50. A. Saltelli, S. Tarantola, F. Campolongo, and M. Ratto, Sensitivity Analysis in Practice. John Wiley, 2004.
51. S. Sampath, R. Bryce, G. Viswanath, V. Kandimalla, and A. Koru, "Prioritizing User-Session-Based Test Cases for Web Applications Testing," Proc. Int'l Conf. Software Testing, Verification, and Validation, pp. 141-150, Apr. 2008.
52. M. Sherriff, M. Lake, and L. Williams, "Prioritization of Regression Tests Using Singular Value Decomposition with Empirical Change Records," Proc. Int'l Symp. Software Reliability Eng., pp. 81-90, Nov. 2007.
53. F. Shull, V. Basili, B. Boehm, A.W. Brown, P. Costa, M. Lindvall, D. Port, I. Rus, R. Tesoriero, and M. Zelkowitz, "What We Have Learned about Fighting Defects," Proc. Int'l Software Metrics Symp., pp. 249-258, June 2002.
54. R. Solingen, "Measuring the ROI of Software Process Improvement," IEEE Software, vol. 21, no. 3, pp. 32-38 May/June 2004.
55. A. Srivastava and J. Thiagarajan, "Effectively Prioritizing Tests in Development Environmentm," ACM SIGSOFT Software Eng. Notes, vol. 27, no. 4, pp. 97-106, 2002.
56. T. Ostrand, E. Weyuker, and R. Bell, "Predicting the Location and Number of Faults in Large Software Systems," IEEE Trans. Software Eng., vol. 31, no. 4, pp. 340-355, Apr. 2005.
57. S. Wagner, "A Model and Sensitivity Analysis of the Quality Economic of Defect-Detection Techniques," Proc. Int'l Symp. Software Testing and Analysis, pp. 73-84, July 2006.
58. S. Wagner, "An Approach to Global Sensitivity Analysis: FAST on COCOMO," Proc. Int'l Symp. Empirical Software Eng. and Measurement, pp. 440-442, Sept. 2007.
59. A. Walcott, M.L. Soffa, G.M. Kapfhammer, and R.S. Roos, "Time-Aware Test Suite Prioritization," Proc. Int'l Symp. Software Testing and Analysis, pp. 1-12, July 2006.
60. W.E. Wong, J.R. Horgan, S. London, and H. Agrawal, "A Study of Effective Regression Testing in Practice," Proc. Int'l Symp. Software Reliability Eng., pp. 264-274, Nov. 1997.
61. S. Yoo, M. Harman, P. Tonella, and A. Susi, "Clustering Test Cases to Achieve Effective and Scalable Prioritisation Incorporating Expert Knowledge," Proc. Int'l Symp. Software Testing and Analysis, pp. 201-212, July 2009.
62. L. Zhang, S.-S. Hou, C. Guo, T. Xie, and H. Mei, "Time-Aware Test-Case Prioritization Using Integer Linear Programming," Proc. Int'l Symp. Software Testing and Analysis, pp. 213-224, July 2009.