A practical guide for using statistical tests to assess randomized algorithms in software engineering

Proceedings - International Conference on Software Engineering

Volumn , Issue , 2011, Pages 1-10

  Arcuri, Andrea ^a   Briand, Lionel ^b  

^a SIMULA RESEARCH LABORATORY   (Norway)

^b UNIVERSITY OF OSLO   (Norway)

Author keywords

bonferroni adjustment; confidence interval; effect size; non parametric test; parametric test; statistical difference; survey; systematic review

Indexed keywords

BONFERRONI ADJUSTMENT; CONFIDENCE INTERVAL; EFFECT SIZE; NON-PARAMETRIC TEST; PARAMETRIC TEST; STATISTICAL DIFFERENCES; SYSTEMATIC REVIEW;

ALGORITHMS; ENGINEERING; PARAMETER ESTIMATION; RANDOM PROCESSES; SOFTWARE ENGINEERING; STATISTICAL TESTS; SURVEYS;

SOFTWARE TESTING;

EID: 79959871222     PISSN: 02705257     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1985793.1985795     Document Type: Conference Paper

References (60)

1. R. Abraham and M. Erwig. Mutation Operators for Spreadsheets. IEEE Transactions on Software Engineering (TSE), 35(1), 2009.
2. J. Aguilar-Ruiz, I. Ramos, J. C. Riquelme, and M. Toro. An evolutionary approach to estimating software development projects. Information and Software Technology, 43:875-882, 2001. (Pubitemid 33050967)
3. S. Ali, L. Briand, H. Hemmati, and R. Panesar-Walawege. A systematic review of the application and empirical investigation of search-based test-case generation. IEEE Transactions on Software Engineering (TSE), 2009.
4. A. Arcuri. Full theoretical runtime analysis of alternating variable method on the triangle classification problem. In International Symposium on Search Based Software Engineering (SSBSE), pages 113-121, 2009.
5. A. Arcuri, M. Z. Iqbal, and L. Briand. Black-box system testing of real-time embedded systems using random and search-based testing. In IFIP International Conference on Testing Software and Systems (ICTSS), pages 95-110, 2010.
6. A. Arcuri, M. Z. Iqbal, and L. Briand. Formal analysis of the effectiveness and predictability of random testing. In ACM International Symposium on Software Testing and Analysis (ISSTA), pages 219-229, 2010.
7. A. Arcuri and X. Yao. A novel co-evolutionary approach to automatic software bug fixing. In IEEE Congress on Evolutionary Computation (CEC), pages 162-168, 2008.
8. A. J. Bagnall, V. J. Rayward-Smith, and I. M. Whittley. The next release problem. Information and Software Technology, 43(14):883-890, 2001.
9. G. Canfora, M. D. Penta, R. Esposito, and M. L. Villani. An approach for qos-aware service composition based on genetic algorithms. In Genetic and Evolutionary Computation Conference (GECCO), pages 1069-1075, 2005.
10. J. Cohen. Statistical power analysis for the behavioral sciences, 1988.
11. K. D. Cooper, P. J. Schielke, and D. Subramanian. Optimizing for reduced code space using genetic algorithms. In Proceedings of the ACM SIGPLAN workshop on Languages, compilers, and tools for embedded systems, pages 1-9, 1999.
12. M. Cowles and C. Davis. On the origins of the .05 level of statistical significance. American Psychologist, 37(5):553-558, 1982.
13. J. W. Duran and S. C. Ntafos. An evaluation of random testing. IEEE Transactions on Software Engineering (TSE), 10(4):438-444, 1984.
14. J. Durillo, Y. Zhang, E. Alba, and A. Nebro. A Study of the Multi-objective Next Release Problem. In International Symposium on Search Based Software Engineering (SSBSE), pages 49-58, 2009.
15. T. Dybå, V. Kampenes, and D. Sjøberg. A systematic review of statistical power in software engineering experiments. Information and Software Technology (IST), 48(8):745-755, 2006.
16. M. Fay and M. Proschan. Wilcoxon-Mann-Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules. Statistics Surveys, 4:1-39, 2010.
17. W. Feller. An Introduction to Probability Theory and Its Applications, Vol. 1. Wiley, 3 edition, 1968.
18. V. Ganesh, T. Leek, and M. Rinard. Taint-based directed whitebox fuzzing. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 474-484, 2009.
19. L. García. Escaping the Bonferroni iron claw in ecological studies. Oikos, 105(3):657-663, 2004. (Pubitemid 38826088)
20. B. Garvin, M. Cohen, and M. Dwyer. An improved meta-heuristic search for constrained interaction testing. In International Symposium on Search Based Software Engineering (SSBSE), pages 13-22, 2009.
21. K. Ghani, J. Clark, and Y. Heslington. Widening the Goal Posts: Program Stretching to Aid Search Based Software Testing. In International Symposium on Search Based Software Engineering (SSBSE), pages 122-131, 2009.
22. P. Godefroid, N. Klarlund, and K. Sen. Dart: directed automated random testing. In ACM Conference on Programming language design and implementation (PLDI), pages 213-223, 2005.
23. S. Goodman. Toward evidence-based medical statistics. 1: The P value fallacy. Annals of Internal Medicine, 130(12):995-1004, 1999.
24. R. Grissom and J. Kim. Effect sizes for research: A broad practical approach. Lawrence Erlbaum, 2005.
25. M. Harman, S. A. Mansouri, and Y. Zhang. Search based software engineering: A comprehensive analysis and review of trends techniques and applications. Technical Report TR-09-03, King's College, 2009.
26. M. Harman and P. McMinn. A theoretical and empirical study of search based testing: Local, global and hybrid search. IEEE Transactions on Software Engineering (TSE), 36(2):226-247, 2010.
27. H. Hsu and A. Orso. MINTS: A general framework and tool for supporting test-suite minimization. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 419-429, 2009.
28. V. Kampenes, T. Dybå, J. Hannay, and D. Sjøberg. A systematic review of effect size in software engineering experiments. Information and Software Technology (IST), 49(11-12):1073-1086, 2007. (Pubitemid 47464910)
29. M. Katz. Multivariable analysis: a practical guide for clinicians. Cambridge Univ Pr, 2006.
30. K. Khan, R. Kunz, J. Kleijnen, and G. Antes. Systematic reviews to support evidence-based medicine: how to review and apply findings of healthcare research. RSM Press, 2004.
31. U. Khan and I. Bate. WCET analysis of modern processors using multi-criteria optimisation. In International Symposium on Search Based Software Engineering (SSBSE), pages 103-112, 2009.
32. T. Khoshgoftaar, L. Yi, and N. Seliya. A multiobjective module-order model for software quality enhancement. IEEE Transactions on Evolutionary Computation (TEC), 8(6):593-608, 2004. (Pubitemid 40085397)
33. A. Kieyzun, P. Guo, K. Jayaraman, and M. Ernst. Automatic creation of SQL injection and cross-site scripting attacks. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 199-209, 2009.
34. D. Kim and S. Park. Dynamic Architectural Selection: A Genetic Algorithm Based Approach. In International Symposium on Search Based Software Engineering (SSBSE), pages 59-68, 2009.
35. B. Kitchenham, O. Pearl Brereton, D. Budgen, M. Turner, J. Bailey, and S. Linkman. Systematic literature reviews in software engineering-A systematic literature review. Information and Software Technology (IST), 51(1):7-15, 2009.
36. J. Klein and M. Moeschberger. Survival analysis: techniques for censored and truncated data. Springer Verlag, 2003.
37. S. Kpodjedo, F. Ricca, G. Antoniol, and P. Galinier. Evolution and Search Based Metrics to Improve Defects Prediction. In International Symposium on Search Based Software Engineering (SSBSE), pages 23-32, 2009.
38. N. Leech and A. Onwuegbuzie. A Call for Greater Use of Nonparametric Statistics. Technical report, US Dept. Education, 2002.
39. A. Marchetto and P. Tonella. Search-based testing of Ajax web applications. In International Symposium on Search Based Software Engineering (SSBSE), pages 3-12, 2009.
40. A. Masood, R. Bhatti, A. Ghafoor, and A. Mathur. Scalable and Effective Test Generation for Role-Based Access Control Systems. IEEE Transactions on Software Engineering (TSE), pages 654-668, 2009.
41. P. McMinn. Search-based software test data generation: A survey. Software Testing, Verification and Reliability, 14(2):105-156, 2004.
42. T. Menzies, S. Williams, B. Boehm, and J. Hihn. How to avoid drastic software process change (using stochastic stability). In ACM/IEEE International Conference on Software Engineering (ICSE), pages 540-550, 2009.
43. B. S. Mitchell and S. Mancoridis. On the automatic modularization of software systems using the bunch tool. IEEE Transactions on Software Engineering (TSE), 32(3):193-208, 2006.
44. M. Motwani and P. Raghavan. Randomized Algorithms. Cambridge University Press, 1995.
45. S. Nakagawa. A farewell to Bonferroni: the problems of low statistical power and publication bias. Behavioral Ecology, 15(6):1044-1045, 2004. (Pubitemid 39471217)
46. S. Nakagawa and I. Cuthill. Effect size, confidence interval and statistical significance: a practical guide for biologists. Biological Reviews, 82(4):591-605, 2007. (Pubitemid 47609922)
47. A. Ngo-The and G. Ruhe. Optimized Resource Allocation for Software Release Planning. IEEE Transactions on Software Engineering (TSE), 35(1):109-123, 2009.
48. S. Nijssen and T. Back. An analysis of the behavior of simplified evolutionary algorithms on trap functions. IEEE Transactions on Evolutionary Computation (TEC), 7(1):11-22, 2003.
49. T. Perneger. What's wrong with Bonferroni adjustments. British Medical Journal, 316:1236-1238, 1998. (Pubitemid 28171344)
50. S. Poulding and J. Clark. Efficient Software Verification: Statistical Testing Using Automated Search. IEEE Transactions on Software Engineering (TSE), 2010.
51. R Development Core Team. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2008. ISBN 3-900051-07-0.
52. J. A. Rice. Mathematical Statistics and Data Analysis. Duxbury Press, 2 edition, 1994.
53. G. Rudolph. Convergence analysis of canonical genetic algorithms. IEEE transactions on Neural Networks, 5(1):96-101, 1994.
54. G. Ruxton. The unequal variance t-test is an underused alternative to Student's t-test and the Mann-Whitney U test. Behavioral Ecology, 17(4):688-690, 2006. (Pubitemid 43992440)
55. S. Sawilowsky and R. Blair. A more realistic look at the robustness and type II error properties of the t test to departures from population normality. Psychological Bulletin, 111(2):352-360, 1992.
56. M. Shevertalov, J. Kothari, E. Stehle, and S. Mancoridis. On the Use of Discretized Source Code Metrics for Author Identification. In International Symposium on Search Based Software Engineering (SSBSE), pages 69-78, 2009.
57. T. Thum, D. Batory, and C. Kastner. Reasoning about edits to feature models. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 254-264, 2009.
58. A. Vargha and H. D. Delaney. A critique and improvement of the CL common language effect size statistics of McGraw and Wong. Journal of Educational and Behavioral Statistics, 25(2):101-132, 2000.
59. W. Weimer, T. Nguyen, C. L. Goues, and S. Forrest. Automatically finding patches using genetic programming. In ACM/IEEE International Conference on Software Engineering (ICSE), pages 364-374, 2009.
60. R. Wilcox. Fundamentals of modern statistical methods: Substantially improving power and accuracy. Springer Verlag, 2001.