Double-blind review in software engineering venues: The community's perspective

Proceedings - 2017 IEEE/ACM 39th International Conference on Software Engineering Companion, ICSE-C 2017

Volumn , Issue , 2017, Pages 385-396

  Bacchelli, Alberto ^a   Beller, Moritz ^a  

^a DELFT UNIVERSITY OF TECHNOLOGY   (Netherlands)

Author keywords

Academic community; Double blind review; Scientific publishing

Indexed keywords

C (PROGRAMMING LANGUAGE); COST BENEFIT ANALYSIS; COSTS; REVIEWS; SURVEYS;

ACADEMIC COMMUNITY; BENEFITS AND COSTS; COMPLEX TRANSITIONS; PEER-REVIEW PROCESS; SCIENTIFIC METHOD; SIDE EFFECT;

SOFTWARE ENGINEERING;

EID: 85026771406     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ICSE-C.2017.49     Document Type: Conference Paper

References (61)

1. K. S. McKinley, "Editorial: More on improving reviewing quality with double-blind reviewing, external review committees, author response, and in person program committee meetings. " http://www. cs. utexas. edu/ users/mckinley/notes/blind-revised-2015. html, June 2015. Accessed 2015/08/17.
2. M. Weicher, "Peer review and secrecy in the "information age", " Proceedings of the American Society for Information Science and Technology, vol. 45, no. 1, pp. 1-12, 2008.
3. J. S. Armstrong, "Peer review for journals: Evidence on quality control, fairness, and innovation, " Science and engineering ethics, vol. 3, no. 1, pp. 63-84, 1997.
4. L. Bornmann and H.-D. Daniel, "The luck of the referee draw: the effect of exchanging reviews, " Learned publishing, vol. 22, no. 2, pp. 117-125, 2009.
5. J. Campanario, "Rejecting and resisting nobel class discoveries: accounts by nobel laureates, " Scientometrics, vol. 81, no. 2, pp. 549-565, 2009.
6. J. Chen and J. A. Konstan, "Conference paper selectivity and impact, " Communications of the ACM, vol. 53, no. 6, pp. 79-83, 2010.
7. K. Siler, K. Lee, and L. Bero, "Measuring the effectiveness of scientific gatekeeping, " Proceedings of the National Academy of Sciences, vol. 112, no. 2, pp. 360-365, 2015.
8. F. Squazzoni, G. Bravo, and K. Takács, "Does incentive provision increase the quality of peer review? an experimental study, " Research Policy, vol. 42, no. 1, pp. 287-294, 2013.
9. P. J. Roebber and D. M. Schultz, "Peer review, program officers and science funding, " PLoS One, vol. 6, no. 4, p. e18680, 2011.
10. F. Bianchi and F. Squazzoni, "Is three better than one? simulating the effect of reviewer selection and behavior on the quality and efficiency of peer review, " in 2015 Winter Simulation Conference (WSC), pp. 4081-4089, IEEE, 2015.
11. R. R. Snell, "Menage a quoi? optimal number of peer reviewers, " PloS one, vol. 10, no. 4, p. e0120838, 2015.
12. R. Snodgrass, "Single-versus double-blind reviewing: an analysis of the literature, " ACM Sigmod Record, vol. 35, no. 3, pp. 8-21, 2006.
13. Y. Brun, "A case for double-blind reviewing in software engineering. " https://people. cs. umass. edu/?brun/doubleblind. html, Apr 2015.
14. M. Jacovi, V. Soroka, G. Gilboa-Freedman, S. Ur, E. Shahar, and N. Marmasse, "The chasms of cscw: a citation graph analysis of the cscw conference, " in Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work, pp. 289-298, ACM, 2006.
15. J. Kaye, "Some statistical analyses of chi, " in CHI'09 Extended Abstracts on Human Factors in Computing Systems, pp. 2585-2594, ACM, 2009.
16. J. R. Gilbert, E. S. Williams, and G. D. Lundberg, "Is there gender bias in jama's peer review process?, " Jama, vol. 272, no. 2, pp. 139-142, 1994.
17. R. M. Blank, "The effects of double-blind versus single-blind reviewing: Experimental evidence from the american economic review, " The American Economic Review, vol. 81, no. 5, pp. pp. 1041-1067, 1991.
18. R. Walker and P. Rocha da Silva, "Emerging trends in peer review-a survey., " Frontiers in neuroscience, vol. 9, pp. 169-169, 2014.
19. D. Lo, N. Nagappan, and T. Zimmermann, "How practitioners perceive the relevance of software engineering research, " in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, pp. 415-425, ACM, 2015.
20. L. W. Willem Visser, "Insights and Lessons Learned from Analyzing ICSE 2016 Survey and Review Data. " http://2016. icse. cs. txstate. edu/ static/downloads/2016ReviewProcess. pdf, May 2016.
21. R. G. Bachand and P. P. Sawallis, "Accuracy in the identification of scholarly and peer-reviewed journals and the peer-review process across disciplines, " The Serials Librarian, vol. 45, no. 2, pp. 39-59, 2003.
22. J. M. Campanario, "Peer review for journals as it stands today-part 1, " Science Communication, vol. 19, no. 3, pp. 181-211, 1998.
23. J. M. Campanario, "Peer review for journals as it stands today-part 2, " Science Communication, vol. 19, no. 4, pp. 277-306, 1998.
24. R. Snodgrass, "Editorial: Single-versus double-blind reviewing, " ACM Transactions on Database Systems (TODS), vol. 32, no. 1, p. 1, 2007.
25. R. Snodgrass, "Frequently-Asked Questions About Double-Blind Reviewing. " http://tods. acm. org/editorials/doubleblindfaq. pdf.
26. S. Hill and F. Provost, "The myth of the double-blind review?: author identification using only citations, " ACM SIGKDD Explorations Newsletter, vol. 5, no. 2, pp. 179-184, 2003.
27. van Rooyen S, G. F, E. S, S. R, and B. N, "Effect of blinding and unmasking on the quality of peer review: A randomized trial, " JAMA, vol. 280, no. 3, pp. 234-237, 1998.
28. A. E. Budden, T. Tregenza, L. W. Aarssen, J. Koricheva, R. Leimu, and C. J. Lortie, "Double-blind review favours increased representation of female authors, " Trends in ecology & evolution, vol. 23, no. 1, pp. 4-6, 2008.
29. T. J. Webb, B. O'Hara, and R. P. Freckleton, "Does double-blind review benefit female authors?, " Heredity, vol. 77, pp. 282-291, 2008.
30. M. Seeber and A. Bacchelli, "Does single blind peer review hinder newcomers?, " Scientometrics, forthcoming.
31. D. M. Amodio, E. Harmon-Jones, P. G. Devine, J. J. Curtin, S. L. Hartley, and A. E. Covert, "Neural signals for the detection of unintentional race bias, " Psychological Science, vol. 15, no. 2, pp. 88-93, 2004.
32. P. G. Devine, E. A. Plant, D. M. Amodio, E. Harmon-Jones, and S. L. Vance, "The regulation of explicit and implicit race bias: the role of motivations to respond without prejudice., " Journal of personality and social psychology, vol. 82, no. 5, p. 835, 2002.
33. H. N. Garb, "Race bias, social class bias, and gender bias in clinical judgment, " Clinical Psychology: Science and Practice, vol. 4, no. 2, pp. 99-120, 1997.
34. H. Zeisel, "Race bias in the administration of the death penalty: The florida experience, " Harv. L. Rev., vol. 95, p. 456, 1981.
35. A. R. Green, D. R. Carney, D. J. Pallin, L. H. Ngo, K. L. Raymond, L. I. Iezzoni, and M. R. Banaji, "Implicit bias among physicians and its prediction of thrombolysis decisions for black and white patients, " Journal of general internal medicine, vol. 22, no. 9, pp. 1231-1238, 2007.
36. C. Goldin and C. Rouse, "Orchestrating impartiality: The impact of 'blind' auditions on female musicians, " American Economic Review, vol. 90, no. 4, pp. 715-741, 2000.
37. R. Steinpreis, K. Anders, and D. Ritzke, "The impact of gender on the review of the curricula vitae of job applicants and tenure candidates: A national empirical study, " Sex Roles, vol. 41, no. 7-8, pp. 509-528, 1999.
38. G. S. Metrics, "Top publications-Engineering & Computer Science. " https://scholar. google. com/citations?view op=top venues&hl=en&vq= eng.
39. C. L. Goues, "SSBSE with double blind. " https://www. cs. cmu. edu/ ?clegoues/double-blind. html.
40. "ISSTA'16 Call for Papers. " http://issta2016. cispa. saarland/?page id=37.
41. "19th International Conference on Fundamental Approaches to Software Engineering (FASE). " http://www. etaps. org/index. php/2016/fase.
42. J. W. Creswell, Research design: Qualitative, quantitative, and mixed methods approaches. Sage Publications, 3rd ed., 2009.
43. M. B. Alberto Bacchelli, "Double-blind review in software engineering venues-Survey. " http://sback. it/dblind/survey. html.
44. B. Glaser, Doing Grounded Theory: Issues and Discussions. Sociology Press, 1998.
45. B. Taylor and T. Lindlof, Qualitative communication research methods. Sage Publications, Incorporated, 2010.
46. D. Spencer, "Card sorting: a definitive guide. " http://boxesandarrows. com/card-sorting-a-definitive-guide/, April 2004.
47. B. Martin and B. Hanington, Universal Methods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions. Rockport Publishers, 2012.
48. J. E. Shade and S. J. Janis, Improving Performance Through Statistical Thinking. Mcgraw-Hill, 2000.
49. M. L. Patten, Questionnaire Research: A Practical Guide. Pyrczak Pub., 2011.
50. B. Kitchenham and S. Pfleeger, "Personal opinion surveys, " Guide to Advanced Empirical Software Engineering, pp. 63-92, 2008.
51. P. Tyagi, "The effects of appeals, anonymity, and feedback on mail survey response patterns from salespeople, " Journal of the Academy of Marketing Science, vol. 17, no. 3, pp. 235-241, 1989.
52. T. Punter, M. Ciolkowski, B. Freimut, and I. John, "Conducting online surveys in software engineering, " in International Symposium on Empirical Software Engineering, ISESE'03, pp. 80-88, IEEE, 2003.
53. N. Golafshani, "Understanding reliability and validity in qualitative research, " The qualitative report, vol. 8, no. 4, pp. 597-607, 2003.
54. L. Sigelaman, "Question-order effects on presidential popularity, " Public Opinion Quarterly, vol. 45, no. 2, pp. 199-207, 1981.
55. A. Furnham, "Response bias, social desirability and dissimulation, " Personality and Individual Differences, vol. 7, no. 3, pp. 385-400, 1986.
56. N. B. Robbins and R. M. Heiberger, "Plotting likert and other rating scales, " in Proceedings of the 2011 Joint Statistical Meeting, 2011.
57. J. M. Bland and D. G. Altman, "The odds ratio, " Bmj, vol. 320, no. 7247, p. 1468, 2000.
58. M. Ley, "Dblp: some lessons learned, " Proceedings of the VLDB Endowment, vol. 2, no. 2, pp. 1493-1500, 2009.
59. T. Xie, "Software engineering conferences (statistics). " hhttp://taoxie. cs. illinois. edu/seconferences. htm. Accessed 2015/08/17.
60. W. I. Thomas, "The methodology of behavior study, " in The child in America: Behavior problems and programs, ch. 13, pp. 553-576, A. A. Knopf, 1928.
61. A. Bacchelli and M. Beller, "Support package for current submission. " http://sback. it/dblind.