Statistical significance testing - A panacea for software technology experiments?

Journal of Systems and Software

Volumn 73, Issue 2, 2004, Pages 183-192

  Miller, James ^a  

^a UNIVERSITY OF ALBERTA   (Canada)

Author keywords

Empirical; Hypothesis; Replication

Indexed keywords

CALIBRATION; COSTS; OBJECT ORIENTED PROGRAMMING; PROBABILITY; PROBLEM SOLVING; STATISTICAL TESTS;

EMPIRICAL; HYPOTHESIS; REPLICATION; STATISTICAL SIGNIFICANCE TESTING;

COMPUTER AIDED SOFTWARE ENGINEERING;

EID: 3242678951     PISSN: 01641212     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jss.2003.12.019     Document Type: Article

References (43)

1. Basili V., Reiter R. Jr. A controlled experiment quantitative comparing software development approaches. IEEE Transactions on Software Engineering. 7:1981;299-320.
2. Basili V., Shull F., Lanubile F. Building knowledge through families of experiments. IEEE Transactions on Software Engineering. 25(4):1999;456-473.
3. Bechhofer R.E. A single-sample multiple decision procedure for ranking means of normal populations with known variances. Annals of Mathematical Statistics. 25:1954;16-39.
4. Berger J.O., Berry D.A. The relevance of stopping rules in statistical inference. Gupta S., Berger J.O. Statistical Decision Theory and Related Topics 4. vol. 1:1988;29-47.
5. Berger J.O., Delampady M. Testing precise hypothesis. Statistical Science. 2:1987;317-352.
6. Berger J.O., Selke T. Testing a point null hypothesis: the irreconcilability of. P -values and evidence Journal of American Statistical Association. 33:1987;112-122.
7. Berry, S.M., Viele, K., 1995. Adjusting the α -level for sample size, Technical Report 635, Carnegie Mellon University, Dept. of Statistics.
8. Brewer J., Hunter A. Mutlimethod Research: A Synthesis of Styles. 1989;Sage Publications.
9. Briand L., Wust J. The impact of design properties on development cost in object-oriented systems. IEEE Transaction on Software Engineering. 27(11):2001;963-986.
10. Brooks R. Studying programmer behavior experimentally: The problems of proper methodology. Communications of the ACM. 23(4):1980;207-213.
11. Carver R.P. The case against statistical significance testing. Harvard Educational Review. 48:1978;378-399.
12. Cohen J. Statistical Power Analysis for the Behavioral Sciences. 1977;Academic Press.
13. Cornfield J., Haenszel W., Hammond W., Lilienfeld E., Shimkin A., Synder E. Smoking and lung cancer: recent evidence and a discussion of some questions. Journal of the National Cancer Institute. 22:1959;173-201.
14. Courtney R., Gustafson D. Shotgun correlations in software measures. Software Engineering Journal. 8(1):1993;5-13.
15. Curtis B. Measurement and experimentation in Software Engineering. Proceedings of the IEEE. 69(3):1980;1144-1157.
16. Gardner M.J., Altman D.G. Confidence intervals rather than. P -values: Estimation rather than hypothesis testing British Medical Journal. 292:1986;746-750.
17. Hilborn R., Stearns S. On inference in ecology and evolutionary biology; the problem of multiple causes. Acta Biotheoretica. 31:1982;145-164.
18. Honorton, C., 1985. How to evaluate and improve the replicability of parapsychological effects. In: Shapin, B., Coly, L. (Eds.), The Repeatability Problem in Parapsychology. Parapsychology foundation, pp. 238-255.
19. Iman R. A Data-based Approach to Statistics. 1994;Duxbury Press.
20. Jeffreys H. Theory of Probability. 1961;Oxford University Press.
21. Jeffreys W.H. Bayesian analysis of random event generator data. Journal of Scientific Exploration. 4:1990;153-169.
22. Johnson D.H. The insignificance of statistical significance testing. Journal of Wildlife Management. 63(3):1999;763-772.
23. Lindley D.V. A statistical paradox. Biometrika. 44:1957;187-192.
24. Lindley D.V. Bayesian Statistics: A Review. 1972;SIAM.
25. Matthews R. Faith, hope and Statistics. New Scientist. 156:1997;36-39.
26. Miller J., Daly J., Wood M., Brooks A., Roper M. Statistical power and its sub-components - missing and misunderstood concepts in empirical software engineering research. Information and Software Technology. 39(2):1997;285-295.
27. Miller J. Applying meta-analytical procedures to software engineering experiments. Journal of Systems and Software. 54:2000;29-39.
28. Nester M.R. An applied statistician's creed. Applied Statistics. 45:1996;401-410.
29. Nunnally J.C. The place of statistics in psychology. Educational and Psychological Measurement. 20:1960;641-650.
30. Popper, K.R., 1934. The Logic of Scientific Discovery, Routledge, 1977; First English Edition, Hutchinson, 1959; First published as Logik Der Forschung in Vienna: Springer.
31. Quinn J.F., Dunham A.E. On hypothesis testing in ecology and evolution. American Naturalist. 122:1983;602-617.
32. Rao, K.R., 1985. Replication in conventional and controversial sciences. In: Shapin, B., Coly, L. (Eds.), The Repeatability Problem in Parapsychology. Parapsychology foundation, pp. 22-41.
33. Rozeboom W.W. The fallacy of the null hypothesis significance test. Psychological Bulletin. 57:1960;416-428.
34. Samuels M. Simpson's paradox and related phenomena. Journal of American Statistical Association. 1993;81.
35. Schmidt F.L. Statistical significance testing and cumulative knowledge in psychology: Implications for training of researchers. Psychological Methods. 1:1996;115-129.
36. Seindenfeld T., Schervish M.J., Kadene J.B. Sieg W. Decisions Without Ordering, Acting and Reflecting. 1990;143-170 Kluwer Academic Publishers, Dordrecht.
37. Selke, T., Bayarri, M.J., Berger, J.O., 1999. Calibration of P -values for Testing Precise Null Hypotheses, Technical Report 99-13, Duke University, Institute of Statistics and Decision Sciences.
38. Selvin H.C., Stuart A. Data-dredging procedures in survey analysis. American Statistician. 320:1966;20-23.
39. Shaver J.P. What statistical significance testing is, and what it is not. Journal of Experimental Education. 61:1993;293-316.
40. Shull F., Lanubile F., Basili V. Investigating reading techniques for object-oriented framework learning. IEEE Transactions on Software Engineering. 26(11):2000;1101-1118.
41. Smith G. Statistical Reasoning. 1985;Allyn and Bacon.
42. Succi G., Pedrycz W., Stefanovic M., Miller J. Practical assessment of the models for identification of defect-prone classes in object-oriented commercial systems using design metrics. Journal of Systems and Software. 65:2003;1-12.
43. Wood M., Daly J., Miller J., Roper M. Multi-method research: An empirical investigation of object-oriented technology. Journal of Systems and Software. 48:1999;13-26.