How reliable are systematic reviews in empirical software engineering?

IEEE Transactions on Software Engineering

Volumn 36, Issue 5, 2010, Pages 676-687

  MacDonell, Stephen ^a   Shepperd, Martin ^b   Kitchenham, Barbara ^c   Mendes, Emilia ^d  

^a AUCKLAND UNIVERSITY OF TECHNOLOGY   (New Zealand)

^b BRUNEL UNIVERSITY   (United Kingdom)

^c KEELE UNIVERSITY   (United Kingdom)

^d UNIVERSITY OF AUCKLAND   (New Zealand)

Author keywords

cost estimation; Empirical software engineering; meta analysis; systematic review

Indexed keywords

COST ESTIMATIONS; EMPIRICAL SOFTWARE ENGINEERING; META-ANALYSIS; RESEARCH INSTRUMENTS; RESEARCH METHODS; RESEARCH QUESTIONS; SYSTEMATIC REVIEW;

COST BENEFIT ANALYSIS; SOFTWARE ENGINEERING;

RESEARCH;

EID: 77957749322     PISSN: 00985589     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSE.2010.28     Document Type: Review

References (26)

1. V. Basili and K. Freburger, "Programming Measurement and Estimation in the Software Engineering Laboratory," J. Systems and Software, Vol. 2, pp. 47-57, 1981.
2. Experimental Software Engineering Issues: A Critical Assessment and Future Directions, H. Rombach, V. Basili, and R. Selby, eds., Springer, 1993.
3. B. Kitchenham, T. Dybå, and M. Jørgensen, "Evidence-Based Software Engineering," Proc. 26th IEEE Int'l Conf. Software Eng., 2004.
4. T. Dybå, B. Kitchenham, and M. Jørgensen, "Evidence-Based Software Engineering for Practitioners," IEEE Software, vol. 22, no. 1, pp. 58-65, Jan./Feb. 2005.
5. B. Kitchenham, E. Mendes, and G. Travassos, "A Systematic Review of Crossvs. Within-Company Cost Estimation Studies," Proc. 10th Int'l Conf. Empirical Assessment in Software Eng., 2006.
6. S. MacDonell and M. Shepperd, "Comparing Local and Global Software Effort Estimation Models Reflections on a Systematic Review," Proc. First Int'l Symp. Empirical Software Eng. and Measurement, 2007.
7. A. Oakley, D. Gough, S. Oliver, and J. Thomas, "The Policy of Evidence and Methodology: Lessons from the EPPI-Centre," Evidence and Policy, vol. 1, no. 1, pp. 5-31, 2005.
8. S. Oliver, G. Peersman, A. Harden, and A. Oakley, "Discrepancies in Findings from Effectiveness Reviews: The Case of Health Promotion for Older People in Accident and Injury Prevention," Health Education J., Vol. 58, pp. 66-77, 1999.
9. A. Oakley, "Social Science and Evidence-Based Everything: The Case of Education," Educational Rev., Vol. 54, pp. 277-286, 2002.
10. A. Oakley and D. Fullerton, A Systematic Review of Smoking Prevention Programmes for Young People, EPPI Centre, Inst. for Education, 1995.
11. L. Pauling, How to Live Longer and Feel Better. Atlantic Books, 1986.
12. P. Knipschild, "Some Examples of Systematic Reviews," British Medical J., Vol. 309, pp. 719-721, 1994.
13. W. Shadish, "Author Judgements about Work They Cite: Three Studies from Psychological Journals," Social Studies of Science, Vol. 25, pp. 477-498, 1995.
14. A. Davis, O. Dieste, A. Hickey, N. Juristo, and A. Moreno, "Effectiveness of Requirements Elicitation Techniques: Empirical Results Derived from a Systematic Review," Proc. 14th IEEE Int'l Requirements Eng. Conf., pp. 179-188, 2006.
15. M. Blettner, W. Sauerbrei, B. Schlehofer, T. Scheuchenpflug, and C. Friedenreich, "Traditional Reviews, Meta-Analyses and Pooled Analyses in Epidemiology," Int'l J. Epidemiology, vol. 28, no. 1, pp. 1-9, 1999.
16. V.R. Basili, F. Shull, and F. Lanubile, "Building Knowledge through Families of Experiments," IEEE Trans. Software Eng., vol. 25, no. 4, pp. 456-473, July/Aug. 1999.
17. W. Hayes, "Research Synthesis in Software Engineering: A Case for Meta-Analysis," Proc. Sixth IEEE Int'l Software Metrics Symp., pp. 143-151, 1999.
18. L. Pickard, B. Kitchenham, and P. Jones, "Combining Empirical Results in Software Engineering," Information and Software Technology, vol. 40, no. 14, pp. 811-821, 1998.
19. J. Miller, "Can Results from Software Engineering Experiments be Safely Combined?" Proc. IEEE Sixth Int'l Software Metrics Symp., L. Briand, ed., 1999.
20. B. Kitchenham, P. Brereton, D. Budgen, M. Turner, J. Bailey, and S. Linkman, "Systematic Literature Reviews in Software Engineering - A Systematic Literature Review," Information and Software Technology, vol. 51, no. 1, pp. 7-15, 2009.
21. DARE, "What are the Criteria for the Inclusion of Reviews on DARE?" technical report, Centre for Rev. and Dissemination, 2007.
22. M. Petticrew, "Why Certain Systematic Reviews Reach Uncertain Conclusions," British Medical J., vol. 326, no. 7392, pp. 756-758, 2003.
23. J. Higgins, S. Thompson, J. Deeks, and D. Altman, "Measuring Inconsistency in Meta-Analysis," British Medical J., Vol. 327, pp. 557-560, 2003.
24. P. Katrak, A.E. Bialocerkowski, N. Massy-Westropp, S. Kumar, and K.A. Grimmer, "A Systematic Review of the Content of Critical Appraisal Tools," BMC Medical Research Methodology (electronic resource), vol. 4, no. 1, p. 22, 2004.
25. B. Kitchenham, E. Mendes, and G. Travassos, "Cross versus Within-Company Cost Estimation Studies: A Systematic Review," IEEE Trans. Software Eng., vol. 33, no. 5, pp. 316-329, May 2007.
26. B. Kitchenham and S. Charters, "Guidelines for Performing Systematic Literature Reviews in Software Engineering," Version 2.3, EBSE Technical Report EBSE-2007-01, Keele Univ., 2007.