Strength of Evidence in Systematic Reviews in Software Engineering

ESEM'08: Proceedings of the 2008 ACM-IEEE International Symposium on Empirical Software Engineering and Measurement

Volumn , Issue , 2008, Pages 178-187

  Dybå, Tore ^a   Dingsøyr, Torgeir ^a  

^a SINTEF ICT   (Norway)

Author keywords

Quality assessment; Strength of evidence; Systematic review

Indexed keywords

SOFTWARE ENGINEERING;

AGILE SOFTWARE DEVELOPMENT; EMPIRICAL STUDIES; KNOW HOWS; OBSERVATIONAL STUDIES; QUALITY ASSESSMENT; STRENGTH OF EVIDENCE; SYSTEMATIC REVIEW;

COMPUTER SOFTWARE SELECTION AND EVALUATION;

EID: 62949200408     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/1414004.1414034     Document Type: Conference Paper

References (50)

1. Arisholm, E., Gallis, H., Dybå, T., and Sjøberg, D. (2007) Evaluating Pair Programming with Respect to System Complexity and Programmer Expertise, IEEE Transactions on Software Engineering, 33(2): 65-86
2. Bailey, J., Zhang, C., Budget, D., and Turner, M., "Search Engine Overlaps: Do They Agree or Disagree?," Second International Workshop on Realising Evidence-Based Software Engineering (REBSE'07), 2007.
3. Bhandari, M., Busse, J.W., Jackowski, D., Montori, V.M., Schünemann, H., Sprague, S., Mears, D., Schemitsch, E.H., Heels-Ansdell, D., and Devereaux, P.J. (2004) Association between Industry Funding and Statistically Significant Pro-Industry Findings in Medical and Surgical Randomized Trials, CMAJ, 170: 477-480
4. Biolchini, J., Mian, P.G., Natali, A.C.C., and Travassos, G.H. (2005) Systematic Review in Software Engineering, Univ. Rio de Janeiro, TR, ES 679/05.
5. Brehmer, B. (1989) In One Word: Not from Experience, Acta Psychologica, 45(1-3): 223-241.
6. Brereton, P., Kitchenham, B. A., Budgen, D., Turner, M., and Khalil, M., "Lessons from Applying the Systematic Literature Review Process within the Software Engineering Domain," Journal of Systems and Software, no. 4, vol. 80, pp. 571-583, 2007.
7. Campbell, D.T. and Stanley, J.C. (1963) Experimental and Quasi-Experimental Designs for Research, Boston: Houghton Mifflin Company
8. Cook, T.D. and Campbell, D.T. (1979) Quasi-Experimentation: Design & Analysis Issues for Field Settings, Boston: Houghton Mifflin Company
9. Cooper, H. (1998) Synthesizing Research (3rd Ed.), Thousand Oaks, CA: Sage.
10. Cooper, H. and Hedges, L.V. (Eds.) (1994) Handbook of Research Synthesis, New York: Russell Sage Foundation.
11. Davies, A., Dieste, O., Hickey, A., Juristo, N., and Moreno, A.M. (2006) Effectiveness of Requirements Elicitation Techniques: Empirical Results Derived from a Systematic Review, Proceedings 14th IEEE International Requirements Engineering Conference (RE'06), IEEE Computer Society, pp. 179-188.
12. Dieste, O. and Padua, A.G. (2007) Developing Search Strategies for Detecting Relevant Experiments for Systematic Reviews, Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement (ESEM'07), Madrid, Spain, 20-21 Sept., IEEE Computer Society, pp. 215-224.
13. Dittrich, Y., John, M., Singer, J., and Tessem, B. (2007) For the Special issue on Qualitative Software Engineering Research, Information and Software Technology, 6(49): 531-539.
14. Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., and Sutton, A. (2005) Synthesising Qualitative and Quantitative Evidence: A Review of Possible Methods, J. of Health Services Research &; Policy, 10(1): 45-53.
15. Dybå, T. and Dingsøyr, T. (2008) Empirical Studies of Agile Software Development: A Systematic Review, Information and Software Technology, 50(9-10): 833-859.
16. Dybå, T., Dingsøyr, T., and Hanssen, G.K. (2007) Applying Systematic Reviews to Diverse Study Types: An Experience Report, Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement (ESEM'07), Madrid, Spain, 20-21 Sept., IEEE Computer Society, pp. 225-234.
17. Dybå, T., Kampenes, V.B. and Sjøberg, D.I.K. (2006) A Systematic Review of Statistical Power in Software Engineering Experiments, Information and Software Technology, 48(8):745-755.
18. Dybå, T., Kitchenham, B.A., and Jørgensen, M. (2005) Evidence-Based Software Engineering for Practitioners, IEEE Software, 22(1): 58-65.
19. Egger, M., Smith, G.D., and Altman, D.G. (2001) Systematic Reviews in Health Care: Meta-analysis in Context (2nd Ed.), London: BMJ Publishing Group.
20. GRADE Working Group (2004) Grading Quality of Evidence and Strength of Recommendations," BMJ, 328: 1490.
21. Greenhalgh, T. (2006) How to Read a Paper: The Basics of Evidence-Based Medicine (3rd Ed.), London: BMJ Publishing Group.
22. Hannay, J., Sjøberg, D.I.K., and Dybå, T. (2007) A Systematic Review of Theory Use in SE Experiments, IEEE Transactions on Software Engineering, 33(2): 87-107.
23. Higgins J.P.T. and Green, S. (Eds.) (2008), Cochrane Handbook for Systematic Reviews of Interventions, Version 5.0.0 (updated February 2008), The Cochrane Collaboration, available from www.cochrane-handbook.org.
24. Höst, M. and Runeson, P. (2007) Checklists for Software Engineering Case Study Research, Proceedings of the First International Symposium on Empirical Software Engineering and Measurement (ESEM'07), Madrid, Spain, 20-21 Sept., IEEE Computer Society, pp. 479-481
25. Jadad, A.R., Moore, R.A., Carroll, D., Jenkinson, C., Reynolds, D.J., Gavaghan, D.J., and McQuay, H.J. (1996) Assessing the Quality of Reports of Randomized Clinical Trials: Is Blinding Necessary?, Controlled Clinical Trials, 17(1): 1-12.
26. Jedlitschka, A. and Pfahl, D. (2005) Reporting Guidelines for Controlled Experiments in Software Engineering, Proceedings of the 4th International Symposium on Empirical Software Engineering (ISESE'05), Noosa Heads, Australia, 17-18 Nov, IEEE Computer Society, pp. 95-104.
27. Jørgensen, M. and Shepperd, M. (2007) A Systematic Review of Software Development Cost Estimation Studies, IEEE Transactions on Software Engineering, 33(1): 33-53.
28. Kampenes, V.B., Dybå, T., Hannay, J.E., and Sjøberg, D.I.K. (2007) A Systematic Review of Effect Size in Software Engineering Experiments, Information and Software Technology, 49(11-12): 1073-1086.
29. Kampenes, V.B., Dybå, T., Hannay, J.E., and Sjøberg, D.I.K. (In Press) A Systematic Review of Quasi-Experiments in Software Engineering, Accepted to Information and Software Technology.
30. Khan, K.S. ter Riet, G., Glanville, J., Sowden, A.J., and Kleijnen, J. (Eds.) (2001) Undertaking Systematic Review of Research on Effectiveness, CRD's Guidance for those Carrying Out or Commissioning Reviews, CRD Report Number 4 (2nd Ed.), NHS Centre for Reviews and Dissemination, University of York.
31. Kitchenham, B.A. (2007) Empirical Paradigm - The Role of Experiments, in V.R. Basili et al. (Eds.), Empirical Software Engineering Issues: Critical Assessment and Future Directions, Proceedings from Int. Workshop, Dagstuhl Castle, June 26-30, 2006, Lecture Notes in Compute Science 4336, Springer, pp. 25-32.
32. Kitchenham, B.A. and Charters, S. (2007) Guidelines for performing Systematic Literature Reviews in Software Engineering, Version 2.3, Keele University, EBSE Technical Report, EBSE-2007-01.
33. Kitchenham, B.A., Al-Khilidar, H., Babar, M.A., Berry, M., Cox, K., Keung, J., Kurniawati, F., Staples, M., Zhang, H., and Zhu, L. (2008) Evaluating Guidelines for Reporting Empirical Software Engineering Studies, Empirical Software Engineering, 13(1): 97-121.
34. Kitchenham, B.A., Dybå, T., and Jørgensen, M. (2004) Evidence-Based Software Engineering, Proceedings of the 26th International Conference on Software Engineering (ICSE 2004), IEEE CS Press, pp. 273-281.
35. Kitchenham, B.A., Pfleeger, S.L., Pickard, L.M., Jones, P.W., Hoaglin, D.C., El Emam, K., and Rosenberg, J. (2002) Preliminary Guidelines for Empirical Research in Software Engineering, IEEE Transactions on Software Engineering, 28(8): 721-734.
36. Klein, H.K. and Myers, M.D. (1999) A Set of Principles for Conducting and Evaluating Interpretive Field Studies in Information Systems, MIS Quarterly, 23(1): 67-93.
37. Laitenberger, O. and Rombach, D. (2003) (Quasi-) Experimental Studies in Industrial Settings, in N. Juristo and A.M. Moreno (Eds.), Lecture Notes on Empirical Software Engineering, Singapore: World Scientific (Series on Software Engineering and Knowledge Engineering 12), pp. 167-227.
38. Lee, A.S. (1989) A Scientific Methodology for MIS Case Studies, MIS Quarterly, 13(1): 33-50.
39. Mcbreen, P. (2003) Questioning Extreme Programming, Boston, MA, USA: Pearson Education.
40. Mendes, E. (2005) A Systematic Review of Web Engineering Research, Proceedings of the 4th International Symposium on Empirical Software Engineering (ISESE'05), Noosa Heads, Australia, 17-18 Nov, IEEE Computer Society, pp. 481-490.
41. Moher D, Schultz KF, Altman D (2001) The CONSORT Statement: Revised Recommendations for Improving the Quality of Reports of Parallel-Group Randomized Trials, Lancet, 357:1191-1194, April 14.
42. Moher D., Cook, D.J., Eastwood, S., Olkin, I., Rennie, D., and Stroup, D.F. (1999) Improving the Quality of Reports of Meta-Analyses of Randomised Controlled Trials: The QUOROM Statement, Lancet, 354: 1896-1900.
43. Mulrow, C. and Cook, D. (Eds.) (1998) Systematic Reviews: Synthesis of Best Evidence for Health Care Decisions, Philadelphia: Am. College of Physicians.
44. Noblit, G.W. and Hare, R.D. (1988) Meta-Ethnography: Synthesizing Qualitative Studies, Thousand Oaks: Sage.
45. Petticrew, M. and Roberts, H. (2006) Systematic Reviews in the Social Sciences: A Practical Guide, Oxford, UK: Blackwell.
46. Shadish, W.R., Cook, T.D. and Campbell, D.T. (2002), Experimental and Quasi-Experimental Designs for Generalized Causal Inference, Boston: Houghton Mifflin.
47. Sjøberg, D.I.K., Dybå, T., and Jørgensen, M. (2007) The Future of Empirical Methods in Software Engineering Research, 29th International Conference on Software Engineering (ICSE'07), Future of Software Engineering (FOSE'07), Minneapolis, Minnesota, USA, 20-26 May, IEEE Computer Society Press IEEE Computer Society, pp. 358-378.
48. Sjøberg, D.I.K., Hannay, J.E., Hansen, O., Kampenes, V.B., Karahasanović, A., Liborg, N.-K., and Rekdal, A.C. (2005) A Survey of Controlled Experiments in Software Engineering, IEEE Transactions on Software Engineering, 31(9):733-753,
49. Staples, M. and Niazi, M. (2007) Experiences Using Systematic Review Guidelines, Journal of Systems and Software, 80(9): 1425-1437.
50. Stroup, D.F., Berlin, J.A., Morton, S.C., Olkin, I., Williamson, G.D., Rennie, D., Moher, D., Becker, B.J., Sipe, T.A., and Thacker, S.B. (2000) Meta-analysis of Observational Studies in Epidemiology: A Proposal for Reporting, JAMA, 283(15): 2008-2012.