A systematic review of statistical power in software engineering experiments

Information and Software Technology

Volumn 48, Issue 8, 2006, Pages 745-755

  Dybå, Tore ^a,b   Kampenes, Vigdis By ^a   Sjøberg, Dag I K ^a  

^a SIMULA RESEARCH LABORATORY   (Norway)

^b SINTEF ICT   (Norway)

Author keywords

Controlled experiment; Effect size; Empirical software engineering; Statistical power; Systematic review

Indexed keywords

COMPUTER SOFTWARE; INFORMATION SCIENCE; RESEARCH AND DEVELOPMENT MANAGEMENT; STATISTICAL METHODS;

CONTROLLED EXPERIMENT; EFFECT SIZE; EMPIRICAL SOFTWARE ENGINEERING; STATISTICAL POWER; SYSTEMATIC REVIEW;

SOFTWARE ENGINEERING;

EID: 33747126100     PISSN: 09505849     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.infsof.2005.08.009     Document Type: Review

References (44)

1. Baroudi J., and Orlikowski W. The problem of statistical power in MIS research. MIS Quarterly 13 1 (1989) 87-106
2. Borokowski S.C., Welsh M.J., and Zhang Q. An analysis of statistical power in behavioral accounting research. Behavioral Research in Accounting 13 (2001) 63-84
3. Brewer J.K. On the power of statistical tests in the American educational research journal. American Educational Research Journal 9 (1972) 391-401
4. Brock J.K.U. The 'Power' of International Business Research. Journal of International Business Studies 34 1 (2003) 90-99
5. Cashen L.H., and Geiger S.W. Statistical power and the testing of null hypotheses: a review of contemporary management research and recommendations for future studies. Organizational Research Methods 7 2 (2004) 151-167
6. Chase L.J., and Chase R.B. A statistical power analysis of applied psychology research. Journal of Applied Psychology 6 2 (1976) 234-237
7. Chase L.J., and Tucker R.K. A power-analytic examination of contemporary communica-tion research. Speech Monographs 42 1 (1975) 29-41
8. Clark-Carter D. The account taken of statistical power in research published in the british journal of psychology. British Journal of Psychology 88 1 (1997) 71-83
9. Cohen B.H. Explaining Psychological Statistics. second ed. (2001), Wiley, New york
10. Cohen J. The statistical power of abnormal-social psychological research: a review. Journal of Abnormal and Social Psychology 65 3 (1962) 145-153
11. Cohen J. Statistical Power Analysis for the Behavioral Sciences (1977), Academic Press, New York Revised Edition
12. Cohen J. Statistical Power Analysis for the Behavioral Sciences. second ed. (1988), Laurence Erlbaum, Hillsdale, New Jersey
13. Cohen J. A power prime. Psychological Bulletin 112 1 (1992) 155-159
14. Cook T.D., and Campbell D.T. Quasi-Experimentation: Design & Analysis Issues for Field Settings (1979), Houghton Mifflin Company, Boston
15. Dybå T. An instrument for measuring the key factors of success in software process improvement. Empirical Software Engineering 5 4 (2000) 357-390
16. Ferguson T.D., and Ketchen Jr. D.J. Organizational configurations and performance: The role of statistical power in extant research. Strategic Management Journal 20 (1999) 385-395
17. Glass R.L., Vessey I., and Ramesh V. Research in software engineering: an analysis of the literature. Information and Software Technology 44 8 (2002) 491-506
18. Hays W.L. Statistics. fifth ed. (1994), Harcourt Brace, New York
19. B.A. Kitchenham, Procedures for performing systematic reviews, Keele University Technical Report TR/SE-0401 and NICTA Technical Report 0400011T.1, 2004.
20. Kitchenham B.A., Pfleeger S.L., Pickard L.M., Jones P.W., Hoaglin D.C., El Emam K., and Rosenberg J. Preliminary guidelines for empirical research in software engineering. IEEE Transactions on Software Engineering 28 8 (2002) 721-734
21. Kraemer H.C., and Thiemann S. How Many Subjects? Statistical Power Analysis in Research (1987), Sage, Beverly Hills
22. Lindsay R.M., and Ehrenberg A.S.C. The design of replicated studies. The American Statistician 47 3 (1993) 217-228
23. Lipsey M.W. Design Sensitivity: Statistical Power for Experimental Research (1990), Sage, Newbury Park, CA
24. Maxwell S.E. The persistence of underpowered studies in psychological research: causes, consequences, and remedies. Psychological Methods 9 2 (2004) 147-163
25. Mazen A.M., Graf L.A., Kellogg C.E., and Hemmasi M. Statistical power in contemporary management research. Academy of Management Journal 30 2 (1987) 369-380
26. McClelland G.H. Increasing statistical power without increasing sample size. American Psychologist 55 8 (2000) 963-964
27. Miller J. Applying meta-analytic procedures to software engineering experiments. Journal of Systems and Software 54 (2000) 29-39
28. Miller J. Statistical significance testing-a panacea for software technology experiments?. Journal of Systems and Software 73 (2004) 183-192
29. Miller J., Daly J., Wood M., Roper M., and Brooks A. Statistical power and its subcomponents-missing and misunderstood concepts in empirical software engineering research. Information and Software Technology 39 4 (1997) 285-295
30. Mone M.A., Mueller G.C., and Mauland W. The perceptions and usage of statistical power in applied psychology and management research. Personnel Psychology 49 1 (1996) 103-120
31. Neyman J., and Pearson E.S. On the use and interpretation of certain test criteria for purposes of statistical inference. Biometrika 20A (1928) 175-240 263-294
32. Neyman J., and Pearson E.S. On the problem of the most efficient tests of statistical hypotheses. Transactions of the Royal Society of London Series A 231 (1933) 289-337
33. Nunnally J.C., and Bernstein I.A. Psychometric Theory. third ed. (1994), McGraw-Hill, New York
34. Ottenbacher K.J. The power of replications and the replications of power. The American Statistician 50 3 (1996) 271-275
35. Pickard L.M., Kitchenham B.A., and Jones P.W. Combining empirical results in software engineering. Information and Software Technology 40 14 (1998) 811-821
36. Rademacher R.A. Statistical power in information systems research: application and impact on the discipline. Journal of Computer Information Systems 39 4 (1999) 1-7
37. Sawyer A.G., and Ball A.D. Statistical power and effect size in marketing research. Journal of Marketing Research 18 3 (1981) 275-290
38. Sedlmeier P., and Gigerenzer G. Do studies of statistical power have an effect on the power of studies?. Psychological Bulletin 105 (1989) 309-316
39. Shadish W.R., Cook T.D., and Campbell D.T. Experimental and Quasi-Experimental Designs for Generalized Causal Inference (2002), Houghton Mifflin Company, Boston
40. Sjøberg D.I.K., Hannay J.E., Hansen O., Kampenes V.B., Karahasanović A., Liborg N.-K., and Rekdal A.C. A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering 31(9) (2005) 733-753. (2005)
41. Spector P. Summated rating scale construction: an introduction. Sage University Paper series on Quantitative Applications in the Social Sciences 07-082 (1992), Sage, Newbury Park, California
42. Wilkinson L. The task force on statistical inference, Statistical methods in psychology journals: guidelines and explanations. American Psychologist 54 8 (1999) 594-604
43. Yin R.K. Case Study Research: Design and Methods (2003), Sage, Thousand Oaks, CA
44. Zelkowitz M.V., and Wallace D. Experimental validation in software engineering. Information and Software Technology 39 11 (1997) 735-743