Computational support for academic peer review: A perspective from artificial intelligence

Communications of the ACM

Volumn 60, Issue 3, 2017, Pages 70-79

  Price, Simon ^a   Flach, Peter A ^a  

^a UNIVERSITY OF BRISTOL   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

INFORMATION SERVICES; LEARNING SYSTEMS; TEXT PROCESSING; VOCABULARY CONTROL;

AUTHOR PROFILES; BIBLIOGRAPHIC DATABASE; CLASSIFICATION SYSTEM; EXPERT FINDING; FEATURE-BASED; GOOGLE SCHOLAR; PEER-REVIEW PROCESS; STATE OF THE ART;

ARTIFICIAL INTELLIGENCE;

EID: 85014177346     PISSN: 00010782     EISSN: 15577317     Source Type: Journal    
DOI: 10.1145/2979672     Document Type: Review

References (47)

1. André, P., Zhang, H., Kim, J., Chilton, L.B., Dow, S.P. and Miller, R.C. Community clustering: Leveraging an academic crowd to form coherent conference sessions. In Proceedings of the First AAAI Conference on Human Computation and Crowdsourcing (Palm Springs, CA, Nov. 7-9, 2013). B. Hartman and E. Horvitz, ed. AAAI, Palo Alto, CA.
2. Balog, K., Azzopardi, L. and de Rijke, M. Formal models for expert finding in enterprise corpora. In Proceedings of the 29th Annual International ACM Conference on Research and Development in Information Retrieval (2006). ACM, New York, NY, 43-50.
3. Benferhat, S. and Lang, J. Conference paper assignment. International Journal of Intelligent Systems 16, 10 (2001), 1183-1192.
4. Blei, D.M, Ng, A.Y. and Jordan, M.I. Latent dirichlet allocation. J. Mach. Learn. Res. (Mar. 2003), 993-1022.
5. Bornmann, L., Bowman, B., Bauer, J., Marx, W., Schier, H. and Palzenberger, M. Standards for using bibliometrics in the evaluation of research institutes. Next Generation Metrics, 2013.
6. Boxwala, A.A., Dierks, M., Keenan, M., Jackson, S., Hanscom, R., Bates, D.W. and Sato, L. Review paper: Organization and representation of patient safety data: Current status and issues around generalizability and scalability. J. American Medical Informatics Association 11, 6 (2004), 468-478.
7. Brixey, J., Johnson, T. and Zhang, J. Evaluating a medical error taxonomy. In Proceedings of the American Medical Informatics Association Symposium, 2002.
8. Charlin, L. and Zemel, R. The Toronto paper matching system: An automated paper-reviewer assignment system. In Proceedings of ICML Workshop on Peer Reviewing and Publishing Models, 2013.
9. Charlin, L., Zemel, R. and Boutilier, C. A framework for optimizing paper matching. In Proceedings of the 27th Annual Conference on Uncertainty in Artificial Intelligence (Corvallis, OR, 2011). AUAI Press, 86-95.
10. De Roure, D. Towards computational research objects. In Proceedings of the 1st International Workshop on Digital Preservation of Research Methods and Artefacts (2013). ACM, New York, NY, 16-19.
11. Deng, H., King, I. and Lyu, M.R. Formal models for expert finding on DBLP bibliography data. In Proceedings of the 8th IEEE International Conference on Data Mining (2008). IEEE Computer Society, Washington, D.C., 163-172.
12. Devedzić, V. Understanding ontological engineering. Commun. ACM 45, 4 (Apr. 2002), 136-144.
13. Di Mauro, N., Basile, T. and Ferilli, S. Grape: An expert review assignment component for scientific conference management systems. Innovations in Applied Artificial Intelligence. LNCS 3533 (2005). M. Ali and F. Esposito, eds. Springer, Berlin Heidelberg, 789-798.
14. Dumais S.T. and Nielsen, J. Automating the assignment of submitted manuscripts to reviewers. In Proceedings of the 15th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (1992). ACM, New York, NY, 233-244.
15. Fang, H. and Zhai, C. Probabilistic models for expert finding. In Proceedings of the 29th European Conference on IR Research (2007). Springer-Verlag, Berlin, Heidelberg, 418-430.
16. Ferilli, S., Di Mauro, N., Basile, T., Esposito, F. and Biba, M. Automatic topics identification for reviewer assignment. Advances in Applied Artificial Intelligence. LNCS 4031 (2006). M. Ali and R. Dapoigny, eds. Springer, Berlin Heidelberg, 721-730.
17. Flach, P. Machine Learning: The Art and Science of Algorithms That Make Sense of Data. Cambridge University Press, 2012.
18. Flach, P.A., Spiegler, S., Golénia, B., Price, S., Herbrich, J.G.R., Graepel, T. and Zaki, M. J. Novel tools to streamline the conference review process: Experiences from SIGKDD'09. SIGKDD Explorations 11, 2 (Dec. 2009), 63-67.
19. Garg, N., Kavitha, T., Kumar, A., Mehlhorn, K., and Mestre, J. Assigning papers to referees. Algorithmica 58, 1 (Sept. 2010), 119-136.
20. Goldsmith, J. and Sloan, R.H. The AI conference paper assignment problem. In Proceedings of the 22nd AAAI Conference on Artificial Intelligence (2007).
21. Harnad, S. Open access scientometrics and the U.K. research assessment exercise. Scientometrics 79, 1 (Apr. 2009), 147-156.
22. Hettich, S. and Pazzani, M.J. Mining for proposal reviewers: Lessons learned at the National Science Foundation. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (2006). ACM, New York, NY, 862-871.
23. Jennings, C. Quality and value: The true purpose of peer review. Nature, 2006.
24. Karimzadehgan, M. and Zhai, C. Integer linear programming for constrained multi-aspect committee review assignment. Inf. Process. Manage. 48, 4 (July 2012), 725-740.
25. Karimzadehgan, M., Zhai, C. and Belford, G. Multiaspect expertise matching for review assignment. In Proceedings of the 17th ACM Conference on Information and Knowledge Management (2008). ACM, New York, NY 1113-1122.
26. Kou, N.M., U, L.H. Mamoulis, N. and Gong, Z. Weighted coverage based reviewer assignment. In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. ACM, New York, NY, 2031-2046.
27. Langford, J. and Guzdial, M. The arbitrariness of reviews, and advice for school administrators. Commun. ACM 58, 4 (Apr. 2015), 12-13.
28. Lawrence, P.A. The politics of publication. Nature 422 (Mar. 2003), 259-261.
29. Ley, M. The DBLP computer science bibliography: Evolution, research issues, perspectives. In Proceedings of the 9th International Symposium on String Processing and Information Retrieval (London, U.K., 2002). Springer-Verlag, 1-10.
30. Liu, X., Suel, T. and Memon, N. A robust model for paper reviewer assignment. In Proceedings of the 8th ACM Conference on Recommender Systems (2014). ACM, New York, NY, 25-32.
31. Long, C., Wong, R.C., Peng, Y. and Ye, L. On good and fair paper-reviewer assignment. In Proceedings of the 2013 IEEE 13th International Conference on Data Mining (Dallas, TX, Dec. 7-10, 2013), 1145-1150.
32. Mehlhorn, K., Vardi, M.Y. and Herbstritt, M. Publication culture in computing research (Dagstuhl Perspectives Workshop 12452). Dagstuhl Reports 2, 11 (2013).
33. Meyer, B., Choppy, C., Staunstrup, J. and van Leeuwen, J. Viewpoint: Research evaluation for computer science. Commun. ACM 52, 4 (Apr. 2009), 31-34.
34. Mimno, D. and McCallum, A. Expertise modeling for matching papers with reviewers. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, New York, NY, 2007, 500-509.
35. Minka, T. Expectation propagation for approximate Bayesian inference. In Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence. J.S. Breese and D. Koller, Eds. Morgan Kaufmann, 2001, 362-369.
36. Price, S. and Flach, P.A. Mining and mapping the research landscape. In Proceedings of the Digital Research Conference. University of Oxford, Sept. 2013.
37. Price, S., Flach, P.A., Spiegler, S., Bailey, C. and Rogers, N. SubSift Web services and workflows for profiling and comparing scientists and their published works. Future Generation Comp. Syst. 29, 2 (2013), 569-581.
38. Pritchard, A. et al. Statistical bibliography or bibliometrics. J. Documentation 25, 4 (1969), 348-349.
39. Rodriguez, M.A and Bollen, J. An algorithm to determine peer-reviewers. In Proceedings of the 17th ACM Conference on Information and Knowledge Management. ACM, New York, NY, 319-328.
40. Sidiropoulos, N.D. and Tsakonas, E. Signal processing and optimization tools for conference review and session assignment. IEEE Signal Process. Mag. 32, 3 (2015), 141-155.
41. Surpatean, A., Smirnov, E.N. and Manie, N. Master orientation tool. ECAI 242, Frontiers in Artificial Intelligence and Applications. L.De Raedt, C. Bessière, D. Dubois, P. Doherty, P. Frasconi, F. Heintz, and P.J.F. Lucas, Eds. IOS Press, 2012, 995-996.
42. Tang, W. Tang, J., Lei, T., Tan, C., Gao, B. and Li, T. On optimization of expertise matching with various constraints. Neurocomputing 76, 1 (Jan. 2012), 71-83.
43. Tang, W., Tang, J. and Tan, C. Expertise matching via constraint-based optimization. In Proceedings of the 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology (Vol 1). IEEE Computer Society, Washington, DC, 2010, 34-41.
44. Taylor, C.J. On the optimal assignment of conference papers to reviewers. Technical Report MS-CIS-08-30, Computer and Information Science Department, University of Pennsylvania, 2008.
45. Terry, D. Publish now, judge later. Commun. ACM 57, 1 (Jan. 2014), 44-46.
46. Vardi, M.Y. Scalable conferences. Commun. ACM 57, 1 (Jan. 2014), 5.
47. Yimam-Seid, D. and Kobsa, A. Expert finding systems for organizations: Problem and domain analysis and the DEMOIR approach. J. Organizational Computing and Electronic Commerce 13 (2003).