On the reproducibility of empirical software engineering studies based on data retrieved from development repositories

Empirical Software Engineering

Volumn 17, Issue 1-2, 2012, Pages 75-89

  González Barahona, Jesús M ^a   Robles, Gregorio ^a  

^a UNIVERSIDAD REY JUAN CARLOS   (Spain)

Author keywords

Mining software repositories; Repeatable results; Reproducibility

Indexed keywords

DATA SETS; EMPIRICAL SOFTWARE ENGINEERING; MINING SOFTWARE REPOSITORIES; REPEATABLE RESULTS; REPRODUCIBILITIES; SOURCE CODES; SYSTEMATIC ANALYSIS;

COMMUNICATION SYSTEMS; SOFTWARE ENGINEERING;

INFORMATION MANAGEMENT;

EID: 84857372534     PISSN: 13823256     EISSN: 15737616     Source Type: Journal    
DOI: 10.1007/s10664-011-9181-9     Document Type: Article

References (26)

1. Barr ET, Bird C, Hyatt E, Menzies T, Robles G (2010) On the shoulders of giants. In: FoSER, pp 23-28
2. VR Basili F Shull F Lanubile 1999 Building knowledge through families of experiments IEEE Trans Softw Eng 25 4 456 473 10.1109/32.799939 (Pubitemid 129304112)
3. Boetticher G, Menzies T, Ostrand T (2007) PROMISE repository of empirical software engineering data. Department of Computer Science, West Virginia University. http://promisedata.org/
4. de Leeuw J (2001) Reproducible research. The bottom line. Technical report, UC Los Angeles: Department of Statistics, UCLA. http://escholarship.org/ uc/item/9050x4r4
5. DL Donoho A Maleki IU Rahman M Shahram V Stodden 2009 Reproducible research in computational harmonic analysis Comput Sci Eng 11 8 18 10.1109/MCSE.2009.15
6. Fayyad U, Piatetsky-Shapiro G, Smyth P (1996) Knowledge discovery and data mining: towards a unifying framework. In: Proceedings of the 2nd international conference on knowledge discovery and data mining, KDD-96, Portland (Oregon, USA). AAAI Press, Menlo Park, pp 82-88
7. S Fomel JF Claerbout 2009 Guest editors' introduction: reproducible research Comput Sci Eng 11 5 7 10.1109/MCSE.2009.14
8. R Gentleman DT Lang 2007 Statistical analyses and reproducible research J Comput Graph Stat 16 1 1 23 10.1198/106186007X178663
9. Germán DM (2004) Mining CVS repositories, the softChange experience. In: Proceedings of the international workshop on mining software repositories, Edinburghh, UK
10. Gomez OS, Juristo N, Vegas S (2010) Replication, reproduction and re-analysis: three ways for verifying experimental findings. In: Proceedings of the 1st international workshop on replication in empirical software engineering research (RESER 2010), Cape Town, South Africa
11. JM González-Barahona G Robles M Michlmayr JJ Amor DM Germán 2009 Macro-level software evolution: a case study of a large software compilation Empir Software Eng 14 3 262 285 10.1007/s10664-008-9100-x
12. Hayes JH, Dekhtyar A, Sundaram S (2005) Text mining for software engineering: how analyst feedback impacts final results. In: Proceedings of the second international workshop on mining software repositories, St. Louis, USA
13. Herraiz I, Izquierdo-Cortazar D, Rivas-Hernández F (2009) FLOSSMetrics: Free/Libre/Open Source Software Metrics. In: CSMR, pp 281-284
14. Hothorn T, Leisch F (2011) Case studies in reproducibility. Brief Bioinform
15. J Howison M Conklin K Crowston 2006 FLOSSmole: a collaborative repository for FLOSS research data and analyses IJITWE 1 3 17 26
16. CD Knutson JL Krein L Prechelt N Juristo 2010 Report from the 1st international workshop on replication in empirical software engineering research (RESER 2010) SIGSOFT Softw Eng Notes 35 42 44 10.1145/1838687.1838698
17. R Koenker A Zeileis 2009 On reproducible econometric research J Appl Econ 24 5 833 847 2750993 10.1002/jae.1083
18. J Miller 2005 Replicating software engineering experiments: a poisoned chalice or the holy grail Inf Softw Technol 47 233 244 10.1016/j.infsof.2004.08. 005 (Pubitemid 40178269)
19. Panjer LD (2007) Predicting eclipse bug lifetimes. In: Proceedings of the fourth international workshop on mining software repositories, MSR '07, p 29
20. Robles G (2010) Replicating MSR: A study of the potential replicability of papers published in the mining software repositories proceedings. In: 2010 7th IEEE working conference on mining software repositories (MSR), pp 171-180
21. Robles G, Germán DM (2010) Beyond replication: an example of the potential benefits of replicability in the mining of software repositories community. In: Proceedings of the 1st international workshop on replication in empirical software engineering sesearch (RESER 2010)
22. G Robles JM González-Barahona JJ Merelo-Guervós 2006 Beyond source code: the importance of other artifacts in software development (a case study) J Syst Softw 79 9 1233 1248 10.1016/j.jss.2006.02.048 (Pubitemid 44314120)
23. F Shull MG Mendonça VR Basili J Carver JC Maldonado SCPF Fabbri GH Travassos MCF de Oliveira 2004 Knowledge-sharing issues in experimental software engineering Empir Software Eng 9 1-2 111 137 10.1023/B:EMSE.0000013516. 80487.33
24. FJ Shull JC Carver S Vegas N Juristo 2008 The role of replications in empirical software engineering Empir Software Eng 13 2 211 218 10.1007/s10664-008-9060-1
25. Vandewalle P, Barrenexea G, Jovanovic I, Ridolfi A, Vetterli M (2007) Experiences with reproducible research in various facets of signal processing research. In: Proceedings of the international conference on acoustics, speech and signal processing. ICASSP 2007, vol 4, pp IV-1253-IV-1256 (Pubitemid 47178604)
26. Vegas S, Juristo N, Moreno A, Solari M, Letelier P (2006) Analysis of the influence of communication between researchers on experiment replication. In: ISESE '06: Proceedings of the 2006 ACM/IEEE international symposium on empirical software engineering, pp 28-37 (Pubitemid 46634438)