Replication of empirical studies in software engineering: Preliminary findings from a systematic mapping study

Proceedings - 2011 2nd International Workshop on Replication in Empirical Software Engineering Research, RESER 2011

Volumn , Issue , 2011, Pages 61-70

  Da Silva, Fabio Q B ^a   Suassuna, Marcos ^a   Lopes, Rodrigo F ^a   Gouveia, Tatiana B ^a   França, A César A ^a   De Oliveira, João Paulo N ^a   De Oliveira, Leonardo F M ^a   Santos, André L M ^a  

^a FEDERAL UNIVERSITY OF PERNAMBUCO   (Brazil)

Author keywords

Empirical studies; Experiments; Literature review; Mapping study; Replications; Software engineering

Indexed keywords

COMPUTER AIDED SOFTWARE ENGINEERING; EXPERIMENTS; MAPPING; SOFTWARE DESIGN; SOFTWARE ENGINEERING; SOFTWARE TESTING;

CONFIGURATION MANAGEMENT; EMPIRICAL STUDIES; EXTERNAL REPLICATION; LITERATURE REVIEWS; MAPPING STUDIES; REPLICATIONS; SOFTWARE CONSTRUCTION; SYSTEMATIC MAPPING STUDIES;

ENGINEERING RESEARCH;

EID: 84858064613     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/RESER.2011.14     Document Type: Conference Paper

References (38)

1. Abran, A.; Moore, J.; Bourque, P.; Dupuis, T. (Eds.), Guide to software engineering body of knowledge, IEEE Computer Society, 2004.
2. Almqvist, J.P.F. Replication of controlled experiments in empirical software engineering- a survey. Master's thesis, Department of Computer Science, Faculty of Science, Lund University, Sweden, 2006.
3. Arksey, H.; ÓMalley, L. Scoping studies: towards a methodological framework, International Journal of Social Research Methodology, 8(1), 19-32, 2005. (Pubitemid 40322704)
4. Basili V, et al. (1999) Building knowledge through families of experiments. IEEE Trans Softw Eng 25 (4):456-473. (Pubitemid 129304112)
5. Brooks, A. et al. Replication of Experimental Results in Software Engineering. Technical Report, EFoCS-17-95 [RR/95/193], Dept. of Computer Science, Univ. of Strathclyde, 1995.
6. Brooks, A. et al. Replication's role in software engineering. In F. Shull, J. Singer, and D. I. K. Sjøberg, editors, Guide to Advanced Empirical Software Engineering, pages 365-379. Springer, 2007.
7. Carver, Jeffrey C. Towards Reporting Guidelines for Experimental Replications: A Proposal. In RESEŔ2010: Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research, Cape Town, South Africa, 2009.
8. da Silva, F.Q.B. et al., Six years of systematic literature reviews in software engineering: An updated tertiary study, Inform. Softw. Technol. (2011), doi:10.1016/j.infsof.2011.04.004.
9. Davidsen, M. K.; Krogstie, J. A longitudinal study of development and maintenance. Information and Software Technology, 52(7), pp. 707-719, 2010.
10. Daly, J.; Brooks, A.; Miller, J.; Roper, M.; Wood, M. Verification of Results in Software Maintenance Through External Replication. IEEE International Conference on Software Maintenance, pp. 50-57, 1994.
11. Easterbrook, S. M.; et, al. Selecting Empirical Methods for Software Engineering Research. In F. Shull, J. Singer and D. Sjøberg(eds) Guide to Advanced Empirical Software Engineering, Springer, 2007.
12. França, A. César C. et al. The Effect of Reasoning Strategies on Success in Early Learning of Programming: Lessons Learned from an External Experiment Replication. In EASE'2010: 14th International Conference on Evaluation and Assessment in Software Engineering, Keele University, UK, 2010.
13. Gómez G., Omar S.; Juristo, N.; Vegas, N. Replication, Reproduction and Re-analysis: Three ways for verifying experimental findings. In RESEŔ2010: Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research, Cape Town, South Africa, 2009.
14. Gómez G., Omar, S.; Juristo, N.; Vegas, N. Replications Types in Experimental Disciplines. ESEM '10, September 16-17, 2010, Bolzano-Bozen, Italy.
15. Gould, J., & Kolb, W. L. (Eds.). A dictionary of the social sciences. London: Tavistock Publications, 1964.
16. Holgeid, K. K.; Krogstie, J.; Sjøberg, D. I. K. A study of development and maintenance in Norway: assessing the efficiency of information systems support using functional maintenance. Information and Software Technology, 42, pp. 687-700, 2000.
17. Juristo, N.; Vegas, S. Using differences among replications of software engineering experiments to gain knowledge. In ESEM'09: Proceedings of the 2009 3rd International Symposium on Empirical Software Engineering and Measurement. IEEE Computer Society, Washington, DC, USA, pp 356-366, 2009.
18. Kitchenham, B.; Dybå, T.; Jørgensen, M. Evidence-based Software Engineering. Proceedings of the 26th International Conference on Software Engineering, (ICSE '04), IEEE, Washington DC, USA, pp 273 - 281, 2004.
19. Kitchenham, B.; Charters, S. Guidelines for performing systematic literature reviews in software engineering, Technical Report EBSE-2007-01, School of Computer Science and Mathematics, Keele University, 2007.
20. Kitchenham, B. A.; Pfleeger, S. L. Personal Opinion Surveys. In F. Shull, J. Singer and D. Sjøberg(eds) Guide to Advanced Empirical Software Engineering, Chapter 3, Springer, 2007.
21. Kitchenham, B. The role of replications in empirical software engineering-a word of warning. Empirical Software Engineering Journal. 13:219-221. 2008.
22. B. Kitchenham et al., Systematic literature reviews in software engineering- a systematic literature review, Information and Software Technology 51, pp. 7-15, 2009.
23. B. Kitchenham, et al., Literature reviews in software engineering- a tertiary study, Information and Software Technology 52 (8), pp. 792- 805, 2010.
24. Koru, A. G. et al. Theory of relative defect proneness-Replicated studies on the functional form of the size-defect relationship. Empirical Software Engineering, vol. 13, pp. 473-498, 2008.
25. Krein, Jonathan L. and Knutson, Charles D. A Case for Replication: Synthesizing Research Methodologies in Software Engineering. In RESEŔ2010: Proceedings of the 1st International Workshop on Replication in Empirical Software Engineering Research, Cape Town, South Africa, 2009.
26. Krogstie, J.; Sølvberg, A. Software Maintenance in Norway: a survey investigation. Proceedings International Conference on Software Maintenance ICSM-94, pp. 304-313, 1994.
27. Krogstie, J.; Jahr, A.; Sjøberg, D. I. K. A longitudinal study of development and maintenance in Norway : Report from the 2003 investigation. Information and Software Technology, 48, pp. 993- 1005, 2006. (Pubitemid 44485997)
28. La Sorte, Michael A. Replication as a Verification Technique in Survey Research: A Paradigm. The Sociological Quarterly, Vol. 13, No. 2, Spring, pp. 218-227, 1972.
29. Lindsay, R.M.; Ehrenberg, A. The Design of Replicated Studies. American Statistician. 47(3): 217-228, 1993.
30. Lung, J. et al. On the difficulty of replicating human subjects studies in software engineering. In Proceedings of the 13th international conference on Software engineering - ICSE '08, New York, USA: ACM Press, p. 191, 2008.
31. Petticrew, M.; Roberts, H. Systematic Reviews in the Social Sciences. Blackwell Publishing, 2006.
32. Popper, K.. The Logic of Scientific Discovery. Hutchinson & Co, 1959.
33. Schmidt, S. Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology, 13(2):90-100, 2009. DOI: 10.1037/a0015108.
34. Sjøberg, D.; et al. A survey of controlled experiments in software engineering. IEEE Transactions on Software Engineering 31 (9), pp. 733-753, 2005. (Pubitemid 41555415)
35. Shull, F.; Basili, V.; Carver, J.; Maldonado, J. C.; Travassos, G. H.; Mendonça, M.; and Fabbri, S. Replicating software engineering experiments: Addressing the tacit knowledge problem. In ISESE'02: Proc. Int. Symp. on Empirical Softw. Eng., Washington, DC, USA, 2002. IEEE Computer Society.
36. Shull, F.; Carver, J.; Vegas, S.; Juristo, N. The Role of Replications in Empirical Software Engineering. Empirical Software Engineering Journal. 13:211-218. 2008.
37. Vegas, S.; et al. Analysis of the Influence of Communication between Researchers on Experiment Replication. Proceedings of the International Symposium on Empirical Software Engineering (ISESE'06). Pp 28-37. September 20-21, 2006, Rio de Janeiro, Brazil. (Pubitemid 46634438)
38. Yin, R. K. Case study research: Design and methods, 4th Edition. London, Sage Publications, 2009.