Peer review and expert panels as techniques for evaluating the quality of academic research

Handbook on the Theory and Practice of Program Evaluation

Volumn , Issue , 2013, Pages 115-142

  Feller, Irwin ^a  

^a PENNSYLVANIA STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84881837448     PISSN: None     EISSN: None     Source Type: Book    
DOI: None     Document Type: Chapter

References (76)

1. Aghion, P., M. Dewatripont, C. Hoxby, A. Mas-colell and A. Sapir (2009), 'The governance and performance of research universities', NBER Working Paper 14851, Cambridge, MA: National Bureau of Economic Research.
2. Boix-mansilla, V., I. Feller and H. Gardner (2006), 'Quality assessment in interdisciplinary research and education', Research Evaluation, 15, 69-74.
3. Bonaccorsi, A. (2010), 'New forms of complementarity in science', Minerva, 48, 355-87.
4. Bonvillian, W. and R. Van Atta (2011), 'ARPA- E and DARPA: applying the DARPA model to energy innovation', Journal of Technology Transfer, 35, 1-45.
5. Bornmann, L., R. Mutz and H.D. Daniel (2007), 'Gender differences in grant peer review: a meta- analysis', Journal of Informetric, 1, 226-38.
6. Bornmann, L., G. Wallon and A. Ledin (2008), 'Does the committee peer review select the best applicants for funding? An investigation of the selection process for two European molecular biology organization programmes', PLoS One 3, no. 10.
7. Bozeman, B. and C. Boardman (2009), 'Broad impacts and narrow perspectives: passing the buck on science and social impacts', Social Epistomology, 23, 183-98.
8. Braben, D. (2004), Pioneering Research, New York: John Wiley & Sons.
9. Brouns, M. and E. Addis (2004), 'Synthesis report on the workshop', in Gender and Excellence in the Making, Brussels: European Commission- Directorate General for Research, pp. 11-33.
10. Butler, L. (2004), 'What happens when funding is linked to publications', in K. Moed, W. Glanzel and U. Schmoch (eds), Handbook of Quantitative Science and Technology Research, Dordrecht: Kluwer Academic Publishers, pp. 389-405.
11. Chubin, D. and E. Hackett (1990), Peerless Science: Peer Review in US Science Policy, Albany, NY: State University of New York Press.
12. Cole, S. (1992), Making Science, Cambridge, MA: Harvard University Press.
13. Cozzens, S. (2007), 'Death by peer review?', in R. Whitley and J. Glaser (eds), Changing Governance of the Sciences, Dordrecht: Springer, pp. 225-42.
14. David, P. (2008), 'The historical origins of "open science": an essay on patronage, reputation, and common agency contracting in the scientific revolution', Capitalism and Society, 3(2).
15. European Science Foundation (2006), 'Peer review', Conference Report, Prague 12-13 October.
16. Feller, I. (2001), 'Elite and/or distributed science', in M. Feldman and A. Link (eds), Innovation Policy in the Knowledge- based Economy, Boston, MA: Kluwer Academic Publishers, pp. 189-209.
17. Feller, I. (2006), 'Multiple actors, multiple settings, multiple criteria: issues in assessing interdisciplinary research', Research Evaluation, 15, 5-15.
18. Feller, I., E. Chubin, E. Derrick and P. Pharityal (forthcoming), 'The challenges of evaluating multi- objective cooperative research centers', in C. Boardman, D. Gray and D. Rivers (eds), Cooperative Research Centers and Their Stakeholders: Public Policies, Industry Strategies, and Organizational Dynamics of the New Science and Engineering Management, Springer Publishing.
19. Frodeman, R. and B. Holbrook (2011), 'NSF's struggle to articulate relevance', Science, 8 July, 201, 157-8.
20. Ginther, D., W. Schaffer, J. Schnell, B. Masimore, F. Liu, L. Haak and R. Kington (2011), 'Race, ethnicity, and NIH awards', Science, 19 August, 333, 1015-19.
21. Glaser, J. and G. Laudel (2007), 'Evaluation without evaluators', in R. Whitley and J. Glaser (eds), Changing Governance of the Sciences, Dordrecht: Springer, pp. 127-51.
22. Glaser, J., S. Lange, G. Laudel and U. Schimank (2010), 'Informed authority', in R. Whitley, J. Glaser and L. Engwall (eds), Reconfiguring Knowledge Production, New York: Oxford University Press, pp. 149-83.
23. Goldstein, H. and D. Spiegelhalter (1996), 'League tables and their limitations: statistical issues in comparisons of institutional performance', Journal of the Royal Statistical Society, Series A, 159, 385-443.
24. Greenberg, D. (1967), The Politics of Pure Science, New York: World Publishing Company.
25. Guena, A. and B. Martin (2003), 'University research evaluation and funding: an international comparison', Minerva, 41, 277-304.
26. Gupta, N., C. Kemelgor, S. Fuchs and H. Etzkowitz (2004), 'The "triple burden": a crosscultural analysis of the consequences of discrimination for women in science', in European Commission, Gender and Excellence in the Making, Luxembourg: Office for Official Publications of the European Communities, pp. 41-50.
27. Guston, D. and K. Keniston (1994), 'Introduction: the social contract for science', in D. Guston and K. Keniston (eds), The Fragile Connection, Cambridge, MA: MIT Press, pp. 1-41.
28. Hackett, E. and D. Chubin (2003), 'Peer review for the 21st century: applications to education research', paper prepared for a National Research Council Workshop, 25 February, Washington, DC.
29. Handler, P. (1977), 'Report of the President, National Academy of Sciences Annual Report, 1976-77', Washington, DC: National Academy Press.
30. Hansson, F. (2010), 'Dialogue in or with the peer review: evaluation research organizations in order to promote organizational learning', Science and Public Policy, 37, 239-51.
31. Harding, S. (1986), The Science Question in Feminism, Ithaca, NY: Cornell University Press.
32. Hart, D. (1998), Forged Consensus, Princeton, NJ: Princeton University Press.
33. Heinze, T. (2008), 'How to sponsor ground- breaking research: a comparison of funding schemes', Science and Public Policy, 35, 302-18.
34. Hicks, D. (2009), 'Evolving regimes of multi- university research evaluation', Higher Education, 57, 393-404.
35. Hicks, D. and J. Katz (2011), 'Equity and excellence in research funding', Minerva, 49(2), 137-51.
36. Holbrook, J. and R. Frodeman (2011), 'Peer review and the ex ante assessment of societal impacts', Research Evaluation, 320, 239-46.
37. Jaffe, A. (2002), 'Building programme evaluation into the design of public research- support programmes', Oxford Review of Economic Policy, 18(1), 22-34.
38. José, J. (2011), 'Statement before the Committee on Science, Space and Technology, Subcommittee on Research and Science Education, United States House of Representatives', 26 July.
39. Kerr, C. (1994), Troubled Times in American Higher Education, Albany, NY: State University of New York Press.
40. Kleinman, D. (1995), Politics on the Endless Frontier, Durham and London: Duke University Press.
41. Kohler, R. (1991), Partners in Science, Chicago, IL: University of Chicago Press.
42. Lamont, M. (2009), How Professors Think, Cambridge, MA: Harvard University Press.
43. Lamont, M. and K. Huutoniemi (2011), 'Comparing customary rules of fairness: evaluative practices in various types of peer review panels', in C. Camic, N. Gross and M. Lamont (eds), Social Knowledge in the Making, Chicago, IL: University of Chicago Press, pp. 209-32.
44. Laudel, G. (2006), 'Conclave in the tower of Babel: how peers review interdisciplinary research projects', Research Evaluation, 15, 57-68.
45. Long, J. and M. Fox (1995), 'Scientific careers: universalism and particularism', American Review of Sociology, 21, 45-71.
46. Margottini, L. (2008), 'A plea for "transparent" funding', Science, 320, 16 May, p. 861.
47. Marrett, C. (2011), 'The merit review process: ensuring limited federal resources are invested in the best science', Testimony before the US House of Representatives, Committee on Science, Space, Technology, Subcommittee on Research and Science Education, 26 July.
48. Marsh, H., U. Jayasinghe and N. Bond (2008), 'Improving the peer- review process for grant applications', American Psychologist, 63, 160-68.
49. Martin, B. (2011), 'The research excellence framework and the "impact agenda": are we creating a Frankenstein monster?', Research Evaluation, 20, 247-54.
50. Martin, B. and R. Whitley (2010), 'The UK research assessment exercise: a case of regulatory capture', in R. Whitley, J. Glaser and L. Engwall (eds), Reconfiguring Knowledge Production, Oxford, UK: Oxford University Press, pp. 51-80.
51. McCullough, J. (1993), 'Federal peer review systems: too conservative to support high- risk research', in S. Nelson, K. Gramp and A. Teich (eds), AAAS Science and Technology Policy Yearbook 1992, Washington, DC: American Association for the Advancement of Science.
52. Mervis, J. (2011a), 'NSF touts family- friendly policies as boon to women', Science, 333, 30 September, p. 1811.
53. Mervis, J. (2011b), 'NSF creates fast track for out- of- the- box proposals', Science, 334, 18 November, p. 883.
54. National Academies (1995), Allocating Federal Funds for Science and Technology, Washington, DC: National Academy Press.
55. National Academies (2000), Experiments in International Benchmarking of US Research Fields, Washington, DC: National Academies Press.
56. National Academies (2007), A Strategy for Assessing Science, Washington, DC: National Academies Press.
57. National Science Board (2007), 'Enhancing support for transformative research at the National Science Foundation', NSB- 07-32, Arlington, VA: National Science Foundation.
58. National Science Board (2010), 'Science and engineering indicators 2010', NSB 10-01, Arlington, VA: National Science Foundation.
59. National Science Board (2011a), 'Report to the National Science Board on the National Science Foundation's merit review process: fiscal year 2010', NSB- 11-41, Arlington, VA: National Science Foundation.
60. National Science Board (2011b), 'Academic research and development expenditures: fiscal year 2009', Arlington, VA: National Science Foundation.
61. Payne, A. (2006), 'Earmarks and EPSCoR', in D. Guston and D. Sarewitz (eds), Shaping Science and Technology Policy, Madison, WI: University of Wisconsin Press, pp. 149-72.
62. Polyani, M. (1962), 'The republic of science', Minerva, 1, 54-73.
63. Rauhvargers, A. (2011), Global University Rankings and Their Impact, Brussels, Belguim: European University Association.
64. Reinhart, M. (2010), 'Peer review practices: a content analysis of external reviews in science funding', Research Evaluation, 19, 317-31.
65. Rosser, S. (2001), 'Are there feminist methodologies appropriate for the natural sciences and do they make a difference?', in M. Lederman and I. Bartsch (eds), The Gender and Science Reader, London: Routledge, pp. 123-44.
66. Rothenberg, M. (2010), 'Making judgments about grant proposals: a brief history of the merit review criteria at the National Science Foundation', Technology and Innovation, 12, 189-201.
67. Savage, J. (1999), Funding Science in America, Cambridge, UK: Cambridge University Press.
68. Smith, S., V. Ward and A. House (2011), '"Impact" in the proposals for the UK's research excellence framework: shifting the boundaries of academic autonomy', Research Policy, 40(10), 1369-79.
69. Stigler, S. (1994), 'Competition and the research university', in J. Cole, E. Barber and S. Graubard (eds), The Research University in a Time of Discontent, Baltimore, MD: The Johns Hopkins University Press, pp. 131-51.
70. Turner, J. (2010), 'Best practices in merit review', A Report to the US Department of Energy, Washington, DC: Association of Public and Land- Grant Universities.
71. US Congress, Office of Technology Assessment (1981), 'An assessment of the United States food and agricultural research system', Washington, DC: US Government Printing Office.
72. US Congress, Office of Technology Assessment (1991), 'Federally funded Research: decisions for a decade', Washington, DC: US Government Printing Office.
73. US General Accounting Office (1983), 'Federal agricultural research funding: issues and concerns', Washington, DC: US Government Printing Office.
74. Van Den Besselaar, P. and L. Leydesdorff (2009), 'Past performance, peer review and project selection: a case study in the social and behavioral sciences', Research Evaluation, 18, 273-88.
75. Wenneras, C. and A. Wold (1997), 'Nepotism and sexism in peer review', Nature, 387, 341-3.
76. Whitley, R. (2007), 'Changing governance of the public sciences', in R. Whitley and J. Glaser (eds), The Changing Governance of the Sciences: The Advent of Research Evaluation Systems, Springer: Dordrecht, pp. 3-27.