Would wider adoption of reproducible research be beneficial for empirical software engineering research?

Journal of Intelligent and Fuzzy Systems

Volumn 32, Issue 2, 2017, Pages 1509-1521

  Madeyski, Lech ^a   Kitchenham, Barbara ^b  

^a WROCLAW UNIVERSITY OF TECHNOLOGY   (Poland)

^b KEELE UNIVERSITY   (United Kingdom)

Author keywords

Empirical software engineering; Reproducible research; Scientific practice

Indexed keywords

SOFTWARE ENGINEERING; SOFTWARE RELIABILITY;

COMPUTATIONAL ENVIRONMENTS; EMPIRICAL SOFTWARE ENGINEERING; GOOD SCIENTIFIC PRACTICES; REPRODUCIBLE RESEARCH; RESEARCH OUTCOME; SCIENTIFIC PRACTICE; SUMMARY STATISTIC;

ENGINEERING RESEARCH;

EID: 85011341240     PISSN: 10641246     EISSN: 18758967     Source Type: Journal    
DOI: 10.3233/JIFS-169146     Document Type: Conference Paper

References (48)

1. E. Barr, C. Bird, E. Hyatt, T. Menzies and G. Robles, On the shoulders of giants. In Proceedings of the FSE/SDP Workshop on Future of Software Engineering Research, FoSER '10, ACM, 2010, pp. 23-28.
2. V. R. Basili, F. Shull and F. Lanubile, Building knowledge through families of experiments, Transactions on Software Engineering 25(4) (1999), 456-473.
3. V. R. Basili, D. Rombach, K. Schneider, B. Kitchenham, D. Pfhal, and R. Selby, Empirical Software Engineering Issues: Critical Assessment and Future Directions, LNCS 4336, Springer, 2006.
4. V. R. Basili, M. V. Zelkowitz, D. I. Sjøberg, P. Johnson and A. J. Cowling, Protocols in the use of empirical software engineering artifacts, Empirical Software Engineering 12(1) (2007), 107-119.
5. D. Bowes, T. Hall and S. Beecham, Slurp: A tool to help large complex literature reviews. In Proceedings of the 2nd International workshop on Evidential Assessment of Software Technologies, EAST, 2012, pp. 33-36.
6. J. C. Carver, N. Juristo, M. T. Baldassarre and S. Vegas, Replications of software engineering experiments, Empirical Software Engineering 19(2) (2014), 267-276.
7. Open Science Collaboration. Estimating the reproducibility of psychological science, Science 349(6251) (2015).
8. Open Source Collaboration. An open, large-scale, collaborative effort to estimate the reproducibility of psychological science, Perspectives on Psychological Science 7(6) (2012), 657-660.
9. D. B. Dahl, xtable: Export tables to LaTeX or HTML, 2016. R package version 1. 8-2.
10. B. Dit, E. Moritz, M. Linares-Vásquez and D. Poshyvanyk, Supporting and accelerating reproducible research in software maintenance using tracelab component library. In Software Maintenance (ICSM), 2013 29th IEEE International Conference on, 2013, pp. 330-339.
11. A. Dusa and A. Thiem, Qualitative Comparative Analysis, 2014. R Package Version 1. 1-4.
12. C. Gandrud, Reproducible Research with R and R Studio, CRC Press, 2013.
13. R. Gentleman and D. Temple Lang, Statistical analyses and reproducible research, Journal of Computational and Graphical Statistics 16(1) (2007), 1-23.
14. O. S. Gómez, N. Juristo and S. Vegas, Understanding replication of experiments in software engineering: A classification, Information and Software Technology 56 (2014), 1033-1048.
15. J. M. González-Barahona and G. Robles, On the reproducibility of empirical software engineering studies based on data retrieved from development repositories, Empirical Software Engineering 17 (2012), 75-89.
16. R. Huang, RQDA: R-based Qualitative Data Analysis, 2014. R package version 0. 2-7.
17. J. P. A. Ioannidis, Why most published research findings are false, PLoS Medicine 2(8) (2005), 696-701.
18. J. P. A. Ioannidis, D. B. Allison, C. A. Ball, I. Coulibaly, X. Cui, A. C. Culhane, M. Falchi, C. Furlanello, L. Game, G. Jurman, J. Mangion, T. Mehta, M. Nitzberg, G. P. Page, E. Petretto and V. van Noort, Repeatability of published microarray gene expression analyses, Nature Genetics 41 (2009), 149-155.
19. M. Jørgensen, T. Dybå, K. Liestøl and D. I. K. Sjøberg, Incorrect results in software engineering experiments: How to improve research practices, Journal of Systems and Software 116 (2016), 133-145.
20. M. Jureczko and L. Madeyski, Cross-project defect prediction with respect to code ownership model: An empirical study, e-Informatica Software Engineering Journal 9(1) (2015), 21-35.
21. B. Kitchenham and E. Mendes, Why comparative effort prediction studies may be invalid. In Proceedings of the 5th International Conference on Predictor Models in Software Engineering, PROMISE '09, ACM, 2009, pp. 4: 1-4: 5.
22. B. A. Kitchenham, L. Madeyski, D. Budgen, J. Keung, P. Brereton, S. Charters, S. Gibbs and A. Pohthong, Robust statistical methods for empirical software engineering, Empirical Software Engineering (2016). DOI: 10. 1007/s10664-016-9437-5 (in press).
23. B. A. Kitchenham and L. Madeyski, Meta-analysis. In B. A. Kitchenham, D. Budgen and P. Brereton, editors, Evidence-Based Software Engineering and Systematic Reviews, chapter 11, CRC Press, 2016, pp. 133-154.
24. D. E. Knuth, Literate programming, The Computer Journal 27(2) (1984), 97-111.
25. M. Kuhn, Cran task view: Reproducible research. http: // cran. r-project. org/web/views/ReproducibleResearch. html
26. F. Leisch, Sweave, Part 1: Mixing R and LaTeX: A short introduction to the Sweave file format and corresponding R functions, R News 2/3 (2002), 28-31.
27. L. Madeyski, Reproducer: Reproduce Statistical Analyses and Meta-Analyses, 2016. R package version 0. 1. 5 (http: //CRAN. R-project. org/package=reproducer).
28. L. Madeyski and M. Jureczko, Which process metrics can significantly improve defect prediction models? An empirical study, Software Quality Journal 23(3) (2015), 393-422.
29. L. Madeyski and M. Majchrzak, Software measurement and defect prediction with dePress extensible framework, Foundations of Computing and Decision Sciences 39(4) (2014), 249-270.
30. M. McCracken, A. Maxwell and C. Hofman, BibDesk. http: //bibdesk. sourceforge. net/, 2015.
31. B. A. Nosek, J. R. Spies, M. Cohn, E. Bartmess, D. Lakens, D. Holman, J. Cohoon, M. Lewis, S. Gordon-McKeon, H. IJzerman, J. Grahe, M. Brandt, J. M. Carp and R. Giner-Sorolla, Open science collaboration, 2015.
32. L. Osherovich, Hedging against academic risk, Science-Business eXchange 4(15) (2011).
33. H. A. Piwowar, R. S. Day and D. B. Fridsma, Sharing detailed research data is associated with increased citation rate, PLoS ONE 2(3) (2007).
34. R Core Team, R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria, 2015.
35. G. Robles, Replicating MSR: A study of the potential replicability of papers published in the Mining Software Repositories proceedings. In Mining Software Repositories (MSR), 2010 7th IEEE Working Conference on, 2010, pp. 171-180.
36. G. Robles and D. M. Germán, Beyond replication: An example of the potential benefits of replicability in the mining of software repositories community. In Proceedings 1st International Workshop on Replication in Empirical Software Engineering Research (RESER 2010), 2010.
37. P. Runeson, A. Stefik and A. Andrews, Variation factors in the design and analysis of replicated controlled experiments. Three (dis)similar studies on inspections versus unit testing, Empirical Software Engineering 19 (2014), 1781-1808.
38. M. Schwab, M. Karrenbach and J. Claerbout, Making scientific computations reproducible, Computing in Science and Engineering 2(6) (2000), 61-67.
39. M. Shepperd, D. Bowes and T. Hall, Researcher bias: The use of machine learning in software defect prediction, IEEE Transactions on Software Engineering 40(6) (2014), 603-616.
40. F. Shull, V. Basili, J. Carver, J. C. Maldonado, G. H. Travassos, M. Mendonça and S. Fabbri, Replicating software engineering experiments: Addressing the tacit knowledge problem. In Proceedings of International Symposium on Empirical Software Engineering ISESE, 2002, pp. 7-16.
41. V. Stodden, The legal framework for reproducible scientific research licensing and copyright, Computing in Science and Engineering 11(1) (2009), 35-40.
42. V. Stodden, F. Leisch and R. D. Peng, editors. Implementing Reproducible Research. CRC Press, 2014.
43. K. Ushey, J. McPherson, J. Cheng and J. J. Allaire, Packrat: A Dependency Management System for Projects and their R Package Dependencies, 2015. R package version 0. 4. 3.
44. R. Vaidyanathan, slidify: Generate reproducible html5 slides from R markdown, 2012. R package version 0. 4. 5.
45. P. Vandewalle, J. Kovacevic and M. Vetterli, Reproducible research in signal processing, IEEE Signal Processing Magazine 26(3) (2009), 37-47.
46. S. Vegas, C. Apa and N. Juristo, Crossover designs in software engineering experiments: Benefits and perils, IEEE Transactions on Software Engineering 42(2) (2016), 120-135.
47. P. A. Whigham, C. Owen and S. MacDonell, A baseline model for software effort estimation, ACM Transactions on Software Engineering and Methodology 24(3) (2015), 20: 1-20: 11.
48. Y. Xie, knitr: A General-Purpose Package for Dynamic Report Generation in R, 2016. R package version 1. 12. 3.