Refining the systematic literature review process-two participant-observer case studies

Empirical Software Engineering

Volumn 15, Issue 6, 2010, Pages 618-653

  Kitchenham, Barbara A ^a   Brereton, Pearl ^a   Turner, Mark ^a   Niazi, Mahmood K ^a   Linkman, Stephen ^a   Pretorius, Rialette ^b   Budgen, David ^b  

^a KEELE UNIVERSITY   (United Kingdom)

^b DURHAM UNIVERSITY   (United Kingdom)

Author keywords

Automated search; Broad search; Case study; Manual search; Mapping studies; Quality evaluation process; Systematic literature review; Targeted search

Indexed keywords

AUTOMATED SEARCHES; BROAD SEARCH; CASE STUDY; MANUAL SEARCH; MAPPING STUDIES; QUALITY EVALUATION; SYSTEMATIC LITERATURE REVIEW; TARGETED SEARCH;

AUTOMATION; REFINING; RESEARCH; SOFTWARE ENGINEERING;

QUALITY CONTROL;

EID: 77956766628     PISSN: 13823256     EISSN: 15737616     Source Type: Journal    
DOI: 10.1007/s10664-010-9134-8     Document Type: Article

References (73)

1. Bailey J, Budgen D, Turner M, Kitchenham B, Brereton P, Linkman S (2007) Evidence relating to Object-Oriented software design: A survey, Proceedings of Empirical Software Engineering and Measurement, IEEE Computer Society Press pp. 482-484
2. Barcelos RF, Travassos GH (2006) Evaluation approaches for Software Architectural Documents: A systematic Review, Ibero-American Workshop on Requirements Engineering and Software Environments (IDEAS). La Plata, Argentina
3. S Beecham N Baddoo T Hall H Robinson H Sharp 2008 Motivation in Software Engineering: A systematic literature review IST 50 860 878
4. CG Bellini RDCDF Pereira JL Becker 2008 Measurement in Software Engineering from the Roadmap to the Crossroads Int J Softw Eng Knowl 18 1 37 64 10.1142/S021819400800357X (Pubitemid 351709433)
5. Bornmann L, Mutz R, Daniel D-D (2010)A reliability-generalization study of journal peer reviews-a multi-level analysis of inter-rater reliability and its determinants. (Submitted)
6. Brereton OP, Kitchenham BA (2007) The Scope of EPIC Case Studies. EPIC technical Report EPIC-2007-04
7. OP Brereton BA Kitchenham D Budgen M Turner MA Khalil 2007 Lessons from applying the Systematic Literature Review process within the Software Engineering domain J Syst & Softw 80 4 571 583 10.1016/j.jss.2006.07.009 (Pubitemid 46329126)
8. Brereton P, Kitchenham B, Budgen D, Li Z (2008) Using a Protocol Template for Case Study Planning. Proceedings of EASE 2008, BCS-eWIC
9. Centre for Reviews and Dissemination (2007) What are the criteria for the inclusion of reviews on DARE? Available at http://www.york.ac.uk/inst/crd/faq4. htm (accessed 24 July 2007)
10. Davis A, Dieste O Hickey A, Juristo N, Moreno AM (2006) Effectiveness of Requirements Elicitation Techniques: Empirical Results Derived from a Systematic Review, 14th IEEE International Requirements Engineering Conference (RE06), pp. 179-188
11. Davis A, Hickey A, Dieste O, Juristo N, Moreno AM (2007) A Quantitative Assessment of Requirements Engineering Publications-1963-2006, LNCS 4542/2007. Requirements Engineering, Foundation for Software Quality, pp 129-143
12. de Boer RC, Farenhorst R (2008) In search of Architectural Knowledge, SHARK 08: Proceedings of the 3 rd international workshop on Sharing and reusing architectural knowledge, May, pp 71-78
13. T Dybå T Dingsøyr 2008 Empirical studies of agile software development: A systematic review IST 50 833 859
14. T Dybå B Kitchenham M Jørgensen 2005 Evidence-based Software Engineering for Practitioners IEEE Softw 22 1 58 65 10.1109/MS.2005.6 (Pubitemid 40714956)
15. Feller J, Finnegan P, Kelly D, MacNamara M (2006) Developing Open Source Software: A Community-Based Analysis of Research, IFIP International Federation for Information Processing, 208/2006, Social Inclusion: Societal and Organizational Implications for Information Systems, pp 261-278
16. Fink A (2005) Conducting Research Literature Reviews. Conducting Research Literature Reviews. From the Internet to Paper. 2nd Edition Sage Publications Ltd.
17. Freire AP, Goularte R, Fortes RPM (2007) Techniques for developing more accessible web applications: a survey towards a process classification, SIGDOC 07: Proceedings of the 25th annual ACM international conference on Design of communication, October, pp 162-169
18. RL Glass I Vessey V Ramesh 2002 Research in software engineering: an analysis of the literature Inf Softw Technol 44 491 506 10.1016/S0950-5849(02) 00049-6 (Pubitemid 34704253)
19. O Gómez H Oktaba M Piattini F García 2008 A Systematic Review Measurement in Software Engineering: State-of-the-Art in Measures ICSOFT 2006, CCIS 10 Lect Notes Comput Sci 5007 165 176
20. Greenhalgh Trisha (2000) How to read a paper: The Basics of Evidence-Based Medicine. BMJ Books
21. Grimstad S, Jorgensen M, Møløkken-∅stvold K (2005) The Clients Impact on Effort Estimation Accuracy in Software Development Projects, 11th IEEE International Software Metrics Symposium (METRICS05), pp 3
22. J Hannay M Jørgensen 2008 The Role of Deliberate Artificial Design Elements in Software Engineering Experiments IEEE Trans Softw Eng 34 2 242 259 10.1109/TSE.2008.13 (Pubitemid 351595962)
23. Hanssen GK, Bjørnson FO, Westerheim H (2007) Tailoring and Introduction of the Rational Unified Process. EuroSPI 2007, LNCS 4764, pp 7-18
24. L Harjumaa J Markkula M Oivo 2008 How does a Measurement Programme Evolve in Software Organizations? PROFES 2008 LNCS 5089 230 243
25. Höfer A, Tichy WF (2007) Status of Empirical Research in Software Engineering, in V. Basili et al., (Eds) Empirical Software Engineering Issues, Springer-Verlag LNCS 4336, pp 10-19
26. Hosbond JH, Nielsen PA (2005) Mobile Systems Development-A literature review, Proceedings of IFIP 8.2 Annual Conference
27. Höst M, Wohlin C, Thelin T (2005) Experimental context classification: incentives and experience of subjects, ICSE05, Proceedings of the 27th international conference on Software engineering, ACM
28. Jefferies C, Brereton P, Turner M (2008) A Systematic Literature review to investigate Reengineering Existing Systems for Multi-Chanel Access, Conference on Software Maintenance and Reengineering (CSMR). April, Athens, pp 258-262
29. M Jørgensen 2004 A review of studies on expert estimation of software development effort J Syst Softw 70 1-2 37 60 10.1016/S0164-1212(02) 00156-5
30. M Jørgensen 2005 Evidence-Based Guidelines for Assessment of Software Development Cost Uncertainty IEEE Trans Softw Eng 2005 942 954 10.1109/TSE.2005.128 (Pubitemid 46395335)
31. M Jørgensen 2007 Estimation of Software Development Work Effort: Evidence on Expert Judgement and Formal Models Int J Forecasting 3 3 449 462 10.1016/j.ijforecast.2007.05.008
32. M Jørgensen M Shepperd 2007 A Systematic Review of Software Development Cost Estimation Studies IEEE Trans SE 33 1 33 53 10.1109/TSE.2007.256943 (Pubitemid 46002167)
33. Jørgensen M, Dybå T, Kitchenham BA (2005) Teaching Evidence-Based Software Engineering to University Students, 11th IEEE International Software Metrics Symposium (METRICS05), p. 24
34. N Juristo AM Moreno S Vegas M Solari 2006 In Search of What We Experimentally Know about Unit Testing IEEE Softw 23 6 72 80 10.1109/MS.2006.166 (Pubitemid 44824160)
35. H Kagdi ML Collard JI Maletic 2006 A survey and taxonomy of approaches for mining software repositories in the context of software evolution J Softw Maintenance Evol Res Pract 19 2 77 131 10.1002/smr.344
36. VB Kampenes T Dybå JE Hannay DIK Sjøberg 2007 A Systematic Review of Effect Size in Software Engineering Experiments Inf Softw Technol 49 11-12 1073 1086 10.1016/j.infsof.2007.02.015 (Pubitemid 47464910)
37. Khan Khalid S, Kunz Regina, Kleijnen Jos, Antes Gerd (2003) Systematic Reviews to Support Evidence-based Medicine, The Royal Society of Medicine Press Ltd
38. Kitchenham BA (2004) Procedures for Undertaking Systematic Reviews, Joint Technical Report, Computer Science Department, Keele University (TR/SE-0401) and National ICT Australia Ltd (0400011 T.1)
39. Kitchenham BA, Charters S (2007) Guidelines for performing Systematic Literature Reviews in Software Engineering Technical Report EBSE/EPIC-2007-01, 2007
40. Kitchenham B, Dybå T, Jørgensen M (2004) Evidence-based Software Engineering. Proceedings of the 26th International Conference on Software Engineering, (ICSE 04), IEEE Computer Society, Washington DC, USA, pp 273-281
41. B Kitchenham E Mendes GH Travassos 2007 A Systematic Review of Cross- vs. Within-Company Cost Estimation Studies IEEE Trans SE 33 5 316 329 10.1109/TSE.2007.1001 (Pubitemid 46681200)
42. Kitchenham BA, Brereton OP, Turner M (2008a) EPIC Case Study 2-Extension of a Tertiary Study. EPIC Technical Report, EPIC-2009-007
43. Kitchenham BA, Brereton OP, Budgen D (2008b) Protocol for extending a Tertiary Study of Systematic Literature Reviews in Software Engineering. EPIC Technical Report, EBSE-2008-006, June
44. BA Kitchenham OP Brereton D Budgen M Turner J Bailey SG Linkman 2009a Systematic Literature reviews in Software Engineering-A Systematic Literature review Inf Softw Technol 51 7 15 10.1016/j.infsof.2008.09.009
45. Kitchenham BA, Brereton P, Turner M, Niazi M, Linkman S, Pretorius R, Budgen D (2009b) The Impact of Limited Search Procedures for Systematic Literature Reviews-A Participant-Observer Case Study, Proceedings of the Third Symposium on Empirical Software Engineering and Measurement, ESEM09, pp 336-345
46. Kitchenham B, Brereton P, Budgen D, Li Z (2009c). An Evaluation of Quality Checklist Proposals-A participant-observer case study, in Proceedings of EASE 2009, BCS-eWiC
47. Kitchenham B, Pretorius R, Budgen D, Brereton OP, Turner M, Niazi M, Linkman S (2010a) Systematic Literature Reviews-A Tertiary Study, Information and Software Technology, accepted for publication.
48. Kitchenham BA, Budgen D, Brereton P (2010b) The value of mapping studies-An observer-participant case study. EASE 2010
49. Liebchen GA, Shepperd M (2008) Data sets and Data Quality in Software Engineering. PROMISE 08: Proceedings of the 4th international workshop on Predictor models in software engineering, May 2008, pp 39-44.
50. MacDonell S, Shepperd M (2007) Comparing local and global effort estimation models reflections on a systematic reviews. Proceedings of Empirical Software Engineering and Measurement, IEEE Computer Society Press
51. Mair M, Shepperd M, Jørgensen M (2005) An analysis of data sets used to train and validate cost prediction systems, PROMISE05 Workshop
52. BR Martin 1996 The use of multiple indicators in the assessment of basic research Scientometrics 36 3 343 362 10.1007/BF02129599
53. BR Martin J Irvine 1983 Assessing Basic Research. Some partial indicators of scientific progress in radio astronomy Res Policy 12 61 90 10.1016/0048-7333(83)90005-7
54. P Mohagheghi R Conradi 2007 Quality, productivity and economic benefits of software reuse: a review of industrial studies Empirical Softw Eng 12 471 516 10.1007/s10664-007-9040-x (Pubitemid 47476815)
55. Mohagheghi P, Dehlen V (2008) Where Is the Proof?-A Review of Experiences from Applying MDE in Industry. ECMDA-FA 2008, LNCS 5095, pp. 432-443
56. AD Neto R Subramanyan M Viera F Travassos GH Shull 2008 Improving Evidence about Software Technologies. A Look at model-based testing IEEE Softw 25 6 242 249
57. Petticrew Mark, Helen Roberts (2005) Systematic Reviews in the Social Sciences: A Practical Guide, Blackwell Publishing
58. FJ Pino F García M Piattini 2008 Software process improvement in small and medium enterprises: a review Softw Qual J 16 237 261 10.1007/s11219-007-9038-z
59. V Ramesh RL Glass I Vessey 2004 Research in computer science: an empirical study J Syst Softw 70 1-2 165 176 10.1016/S0164-1212(03)00015-3
60. Renger M, Kolfschoten GL, de Vreede G-J (2008) Challenges in Collaborative Modeling: A Literature Review, CIAO! 2008 and EOMAS 2008, LNBIP 10, 2008, pp 61-77
61. Segal J, Grinyer A, Sharp H (2005) The type of evidence produced by empirical software engineers. REBSE05
62. Shaw M, Clements P (2006) The Golden Age of Software Architecture: A Comprehensive Survey. Technical Report CMU-ISRI-06-101, Software Engineering Institute, Carnegie Mellon University
63. Shepperd M (2007) Software project economics: a roadmap, FOSE07.
64. DIK Sjøberg JE Hannay O Hansen VB Kampenes A Karahasanovic NK Liborg AC Rekdal 2005 A survey of controlled experiments in software engineering IEEE Trans SE 31 9 733 753 10.1109/TSE.2005.97 (Pubitemid 41555415)
65. M Staples M Niazi 2008 Systematic review of organizational motivation for adopting CMM-based SPI Inf Softw Technol 50 7-8 605 620 10.1016/j.infsof.2007. 07.003 (Pubitemid 351602003)
66. W Tichy P Lukowicz L Prechelt E Heinze 1995 Experimental Evaluation in Computer Science: A Quantitative Study J Syst Softw 28 9 9 18 10.1016/0164-1212(94)00111-Y
67. Torchiano M, Morisio M (2004) Overlooked Aspects of COTS-Based Development. IEEE Software, pp 88-93
68. Turner M, Kitchenham B, Budgen D, Brereton P (2008) Lessons learnt Undertaking a Large-scale Systematic Literature Review, in Proceedings of EASE 2008, BCS-eWiC
69. Weller AC (2001) Editorial Peer Review. Its Strengths and Weaknesses, Assist Monograph Series, Nre Jersey, USA
70. MN Wicks RG Dewar 2007 A new research agenda for tool integration, J Syst Softw 80 1567 1585
71. Yalaho A (2006) A Conceptual Model of ICT-Supported Unified Process of International Outsourcing of Software Production, 10th International Enterprise Distributed Object Computing Conference Workshops (EDOCW06)
72. Yin Robert K (2003) Case Study Research: Design and Methods, 3 rd Edition, Sage Publications
73. H Zhang B Kitchenham D Pfahl 2008 Reflections on 10 years of Software Process Simulation Modeling: A systematic Review. International Workshop on Software Process Simulation Modeling LNCS 5007 345 356