Three empirical studies on the agreement of reviewers about the quality of software engineering experiments

Information and Software Technology

Volumn 54, Issue 8, 2012, Pages 804-819

  Kitchenham, Barbara Ann ^a   Sjøberg, Dag I K ^b   Dybå, Tore ^b,c   Pfahl, Dietmar ^b,d   Brereton, Pearl ^a   Budgen, David ^e   Höst, Martin ^d   Runeson, Per ^d  

^a KEELE UNIVERSITY   (United Kingdom)

^b UNIVERSITY OF OSLO   (Norway)

^c SINTEF   (Norway)

^d LUND UNIVERSITY   (Sweden)

^e DURHAM UNIVERSITY   (United Kingdom)

Author keywords

Empirical studies; Experimentation; Human intensive experiments; Quality evaluation; Software engineering

Indexed keywords

RELIABILITY; SOFTWARE ENGINEERING;

EMPIRICAL STUDIES; EVALUATION CRITERIA; EXPERIMENTATION; HUMAN-INTENSIVE EXPERIMENTS; INTER-RATER RELIABILITIES; QUALITY EVALUATION; QUALITY OF SOFTWARES; SYSTEMATIC LITERATURE REVIEW;

QUALITY CONTROL;

EID: 84861576818     PISSN: 09505849     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.infsof.2011.11.008     Document Type: Article

References (44)

1. S. Abrahão, and G. Poels Experimental evaluation of an object-oriented function Information and Software Technology 49 4 2007 366 380
2. W. Afzal, R. Torkar, and R. Feldt A systematic review of search-based testing for non-functional system properties Information and Software Technology 51 6 2009 959 976
3. V. Alves, N. Niu, C. Alves, and G. Valença Requirements engineering for software product lines: a systematic review Information and Software Technology 52 8 2010 806 820
4. L. Bornmann, R. Mutz, and H-D. Daniel A Reliability-Generalization Study of Journal Peer Reviews: A Multilevel Meta-Analysis of Inter-Rater Reliability and Its Determinants PLoS ONE 5 12 2010 e14331 10.1371/journal.pone.0014331
5. B.B. Brown, Delphi Process. A Methodology used for Elicitation of Opinions of Experts. Rand Paper P-3925, 1968.
6. L. Chen, M.A. Babar, C. Cawley, A status report on the evaluation of variability management, in: Proceedings of the Conference on Evaluation and Assessment in Software Engineering EASE 2009, BCS eWic, 2009.
7. J. Cohen A coefficient of agreement for nominal scales Education and Psychological Measurement 20 1960 37 46
8. J. Cohen Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit Psychological Bulletin 70 1968 213 220
9. T. Dybå, T. Dingsoyr, Strength of evidence in systematic reviews in software engineering, in: Proceedings of the Conference on Empirical Software Engineering and Metrics, ESEM'2008, 2008, pp 178-187.
10. T. Dybå, and T. Dingsoyr Empirical studies of agile software development: a systematic review Information and Software Technology 50 9-10 2008 833 859
11. T. Dyba, V.B. Kampenes, and D.I.K. Sjoberg A systematic review of statistical power in software engineering experiments Information and Software Technology 48 8 2006 745 755 (Pubitemid 44218930)
12. M. Egger, P. Jüni, C. Bartlett, F. Holenstein, and J. Sterne How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study Health Technology Assessment 7 1 2003
13. D.R. Forsyth Group Dynamics fifth ed. 2009 Wadsworth Publishing
14. J.P.A. Iaonnidis, A-B. Haidich, and J. Lau Any casualties in the clash of randomized and observational evidence? Editorial BMJ 322 2001 879 880
15. I.L. Janis Groupthink: Psychological Studies of Policy Decisions and Fiascoes second ed. 1982 Houghton Mifflin
16. M. Jorgensen, and K. Molokken-Ostvold Impact of effort estimates on software project work? A review of the 1994 Chaos report Information and Software Technology 48 4 2000 297 301
17. P. Jüni, A. Witschi, R. Bloch, and M. Egger The hazards of scoring the quality of clinical trials for meta-analysis JAMA 282 11 1999 1054 1060 (Pubitemid 29436453)
18. V.B. Kampenes, Quality of Design Analysis and Reporting of Software Engineering Experiments, A Systematic Review, PhD Thesis, Dept. Informatics, University of Oslo, 2007.
19. A. Karahasanović, A.K. Levine, and R. Thomas Comprehension strategies and difficulties in maintaining object-oriented systems: an explorative study Journal of Systems and Software 80 2007 1541 1559 (Pubitemid 47243675)
20. L. Karlsson, T. Thelin, B. Regnell, P. Berander, and C. Wohlin Pair-wise comparisons versus planning game partitioning - experts on requirements prioritisation techniques Empirical Software Engineering 12 2007 3 33 (Pubitemid 46190939)
21. S.J. Karau, and J.R. Kelly The effects of time scarcity and time abundance on group performance quality and interaction process Journal of Experimental Social Psychology 28 1992 542 571
22. Barbara Kitchenham, Shari Lawrence Pfleeger, Lesley Pickard, Peter Jones, David Hoaglin, Khaled El Emam, and Jarrett Rosenberg Preliminary guidelines for empirical research in software engineering IEEE Transactions on Software Engineering 28 8 2002 721 734
23. B. Kitchenham, P. Brereton, M. Turner, M. Niazi, S. Linkman, R. Pretorius, and D. Budgen Refining the systematic literature review process - two observer-participant case studies Empirical Software Engineering 15 6 2010 619 653
24. B.A. Kitchenham, D.I.K. Sjoberg, O.P. Brereton, D. Budgen, T. Dybå, M. Host, D. Pfahl, P. Runeson, Can we evaluate the quality of software engineering experiments? in: Proceedings of the Conference on Empirical Software Engineering and Metrics ESEM 2010, 2010.
25. P.R. Laughlin Collective induction: twelve postulates Organizational Behavior and Human Decision Processes 80 1 1999 50 69
26. P.R. Laughlin, E.C. Hatch, J.S. Silver, and L. Boh Groups perform better than the best individuals on letters-to-numbers problems: effects of group size Journal of Personality and Social Psychology 90 4 2006 644 651 (Pubitemid 43825054)
27. H. Liu, and H.B.K. Tan Testing input validation in Web applications through automated model recovery Journal of Systems and Software 81 2007 222 233
28. H.W. Marsh, U.W. Jayasinghe, and N.W. Bond Improving the peer-review process for grant application. Reliability, validity, bias and generalizability American Psychologist 63 3 2008 160 168
29. D.G. Myers, and H. Lamm The group polarization phenomenon Psychological Bulletin 83 4 1976 602 627
30. B.D. Neff, and J.D. Olden Is peer review a game of chance BioScience 56 4 2006 333 340
31. J.F. Nunamaker, R.O. Briggs, D.D. Mittleman, D.R. Vogel, and P.A. Balthazard Lessons from a dozen years of group support systems research: a discussion of lab and field findings Journal of Management Information Systems 13 3 1996 163 207
32. J.F. Nunamaker, A.R. Dennis, J.S. Valacich, D. Vogel, and J.F. George Electronic meeting systems to support group work Communications of the ACM 34 7 1991 40 61
33. G.S. Omenn, G.E. Goodman, M.D. Thornquist, J. Balmes, M.R. Cullen, S. Hammar Alpha-tocppherol, Beta carotene cancer prevention study group, 1996 Effects of a combination of beta carotene and vitamin a on lung cancer and cardiovascular disease New England Journal of Medicine 334 1996 1150 1155 (Pubitemid 26137973)
34. M. Petticrew, and H. Roberts Systematic Reviews in the Social Sciences: A Practical Guide 2005 Blackwell Publishing
35. R.W. Poolman, L.C. Keijser, M.C. de Waal Malefijt, L. Blankevoort, F. Farrokhyar, and M. Bhandari Reviewer agreement in scoring 419 abstracts for scientific orthopedics meetings Acta Orthopaedica 78 2 2007 278 284 (Pubitemid 46676672)
36. B.H. Rowe, T.L. Strome, C. Spooner, S. Blitz, E. Grafstein, and A. Worster Reviewer agreement trends from four years of electronic submissions of conference abstract BMC Medical Research Methodology 6 14 2006
37. D.M. Schultz Are three heads better than two? How the number of reviewers and editor behaviour affect the rejection rate Scientometrics 2009 10.1007/s11192-009-0084-0
38. A. Shang, K. Huwwiler-Müntener, L. Nartney, P. Jüni, S. Dörig, D. Pwesner, and M. Egger Are the clinical effects of homeopathy placebo effects? Comparative study of placebo-controlled trials of homeopathy and allopathy Lancet 366 9487 2005 726 732 (Pubitemid 41207234)
39. M.E. Shaw Group Dynamics: The Psychology of Small Group Behavior third ed. 1981 McGraw-Hill
40. P.E. Shrout, and J.L. Fleiss Intraclass correlations: uses in assessing rater reliability Psychological Bulletin 86 2 1979 420 428
41. D.I.K. Sjoberg, J.E. Hannay, O. Hansen, V.B. Kampenes, A. Karahasanovic, N.K. Liborg, and A.C. Rekdal A survey of controlled experiments in software engineering IEEE Transactions on SE 31 9 2005 733 753 (Pubitemid 41555415)
42. S. Yusuf, G. Dagenais, J. Pogue, J. Bosch, and P. Sleight Vitamin E supplementation and cardiovascular events in high-risk patients. The heart outcomes prevention evaluation study investigators New England Medical Journal 342 2000 150 160
43. A.C. Weller Editorial Peer Review: Its Strengths and Weaknesses 2002 Information Today, Inc. Medford, NJ, USA
44. M. Wood, M. Roberts, and B. Howell The reliability of peer reviews of papers on information systems Journal of Information Science 30 1 2004 2 11