Replicating software engineering experiments: A poisoned chalice or the Holy Grail

Information and Software Technology

Volumn 47, Issue 4, 2005, Pages 233-244

  Miller, James ^a  

^a UNIVERSITY OF ALBERTA   (Canada)

Author keywords

Empirical values; Real world settings; Software engineer

Indexed keywords

COMPUTATION THEORY; COMPUTER SIMULATION; KNOWLEDGE ACQUISITION; PARAMETER ESTIMATION; PROBLEM SOLVING; PROCESS CONTROL; STATISTICAL METHODS;

EMPIRICAL EVALUATIONS; EMPIRICAL VALUES; REAL WORLD SETTING; SOFTWARE ENGINEER;

SOFTWARE ENGINEERING;

EID: 13144250220     PISSN: 09505849     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.infsof.2004.08.005     Document Type: Article

References (47)

1. I. Amato Pons and Fleischmann Redux Science 260 1993 895
2. V.R. Basili, R.W. Selby, and D.H. Hutchens Experimentation in software engineering IEEE Transactions in Software Engineering 12 7 1986 733 743
3. V.R. Basili, S. Green, O. Laitenberger, F. Lanubile, F. Shull, S. Sorumgard, and M. Zelkowitz The empirical investigation of perspective-based reading Empirical Software Engineering 1 1996 133 1264
4. J. Brewer, and A. Hunter Multi-method Research A Synthesis of Styles 1989 Sage Publications Beverly Hills, CA
5. W. Broad, and N. Wade Betrayers of the Truth 1986 Oxford University Press Oxford
6. R.E. Brooks Studying programmer behaviour experimentally: the problems of proper methodology Communications of the ACM 23 4 1980 207 213
7. D.T. Campbell, and J.C. Stanley Experimental and Quasi-experimental Designs for Research 1963 Houghton Mifflin Company Boston, MA
8. F. Close TOO HOT TO HANDLE: The Story of the Race for Cold Fussion 1990 W.H. Allen Publishing
9. J. Cohen Statistical Power Analysis for the Behavioural Sciences 1977 Academic Press London
10. H.M. Collins CHANGING ORDER: Replication and Induction in Scientific Practice 1985 SAGE Publications Beverly Hills, CA
11. B. Curtis Measurement and experimentation in software engineering Proceedings of the IEEE 68 9 1990 1144 1157
12. J. Daly, Replication and a Multi-Method Approach to Empirical Software Engineering research, PhD Thesis, Department of Computer Science, University of Strathclyde, 1996.
13. J. Daly, A. Brooks, J. Miller, M. Roper, and M. Wood Verification of results in software maintenance through external replication Proceedings of the IEEE International Conference on Software Maintenance 1994 pp. 50-57
14. R. DerSimonian, and N. Laird Meta-analysis in Clinical Trials Controlled Clinical Trials 7 1986 177 188
15. W. Dumouchel Bayesian meta-analysis D. Berry Statistical Methodology in Pharmaceutical Science 1990 Marcel-Dekker New York 509 529
16. A. Dunsmore, M. Roper, and M. Wood The development and evaluation of three diverse techniques for OO code inspection IEEE Transactions on Software Engineering 2003 677 686
17. P. Fusaro, F. Lanubile, and G. Visaggio A replicated experiment to assess requirements Inspection Techniques 2 1997 39 57
18. J.A. Hall, L. Tickle-Degnen, R. Rosenthal, and F. Mosteller Hypotheses and problems in research synthesis H.M. Cooper L.V. Hedges The Handbook of Research Synthesis 1994 Sage Publications Beverly Hills, CA 17 28
19. W.W. Hauck, and S. Anderson A survey regarding the reporting of simulation studies The American Statistician 38 3 1984 214 216
20. W. Hayes Research synthesis in software engineering: a case for meta-analysis Sixth IEEE International Symposium on Software Metrics 1999
21. L.V. Hedges, and I. Olkin Statistical Methods for Meta-analysis 1985 Academic Press New York
22. D.C. Hoaglin, and D.F. Andrews The reporting of computation-based results in statistics The American Statistician 29 3 1975 122 126
23. J.N. Hooker Needed: an empirical science of algorithms Operations Research 42 2 1994 201 212
24. J. Hunter, and F.L. Schmidt Cumulative research knowledge and social policy formulation: the critical role of meta-analysis Psychology, Public Policy and Law 2 1997 324 347
25. E. Kamsties, C.M. Lott, An empirical evaluation of three defect detection techniques, International Software Engineering Network Technical Report, ISERN-95-02, 1995.
26. T.D. Korson, An empirical study of the effects of modularity on program modifiability, PhD Thesis, College of Business Administration, Georgia State University, 1986.
27. S. Kramer, and R. Rosenthal Effect sizes and significance levels in small sample research R. Hoyle Statistical Strategies for Small Sample Research 1999 Sage Publications Beverly Hills, CA
28. O. Laitenberger, C. Atkinson, M. Schlich, and K. El Eman An experimental comparison of reading techniques for defect detection in UML design documents The Journal of Systems and Software 53 2000 183 204
29. O. Laitenberger, K. El Eman, and T.G. Harbich An internally replicated Quasi-Experimental Comparison of Checklist and Perspective based Reading of Code Documents IEEE Transactions on Software Engineering 2001 387 421
30. R. Lindsay, and A. Ehrenberg The design of replicated studies American Statistician 47 3 1993 217 228
31. C.M. Lott, and H.D. Rombach Repeatable software engineering experiments for comparing defect-detection techniques Journal of Empirical Software Engineering 1 1996 241 277
32. J. McGrath, J. Martin, and J. Kulka Judgment Calls in Research 1982 Sage Publications Beverly Hills, CA
33. J. Miller Can results from software engineering experiments be safety combined? Sixth IEEE International Symposium on Software Metrics 1999
34. J. Miller Statistical significance testing - a Panacea for software technology experiments? Journal of Systems and Software 73 9 2004 183 191
35. J. Miller, M. Wood, and M. Roper Further experiences with scenarios and checklists Empirical Software Engineering 3 1998 37 64
36. M.J. Nye Science in the Provinces 1986 University of California Press Los Angles
37. D.L. Parnas, and D.M. Weiss Active design reviews: principles and practices Journal of Systems and Software 7 4 1987 259 265
38. A. Porter, and L.G. Votta Comparing detection methods for software requirements specification: a replication using professional subjects Empirical Software Engineering 3 1998 355 379
39. A. Porter, L.G. Votta, and V.R. Basili Comparing detection methods for software requirements inspections: a replicated experiment IEEE Transactions on Software Engineering 21 6 1995 563 575
40. R. Rosenthal Meta-analytic Procedures for Social Research 1984 Sage Publications Beverly Hills, CA
41. R. Rosenthal Replication in behavioural research J.W. Neuliep Replication Research in the Social Sciences 1991 Sage Publications Beverly Hills, CA
42. D.A. Scanlan Structured flowcharts outperform pseudocode: an experimental comparison IEEE Software 6 5 1989 28 36
43. B. Shneiderman, R. Mayer, D. McKay, and P. Heller Experimental investigations of the utility of detailed flowcharts in programming Communications of the ACM 20 6 1977 373 381
44. T. Thelin, P. Runeson, and C. Wohlin An experimental comparison of usage-based reading and checklist-based reading IEEE Transactions of Software Engineering 29 8 2003 687 704
45. C. Wohlin, H. Petersson, and A. Aurum Combining data from reading experiments in software inspections: a feasibility study N. Juristo A.M. Moreno in: Lecture Notes on Empirical Software Engineering 2003 85 132
46. F.M. Wolf Meta-Analysis: Quantitative Methods for Research Synthesis 1986 Sage Publications Beverly Hills, CA
47. M. Wood, J. Daly, J. Miller, and M. Roper Multi-method research: an empirical investigation of object-oriented technology Journal of Systems and Software 48 1999 13 26