Mapping the frontiers of evaluation of public-sector R&D programs

Science and Public Policy

Volumn 34, Issue 10, 2007, Pages 681-690

  Feller, Irwin ^a,b  

^a AAAS   (United States)

^b PENNSYLVANIA STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

INNOVATION; POLICY MAKING; PUBLIC SECTOR; RESEARCH AND DEVELOPMENT;

EID: 41449115333     PISSN: 03023427     EISSN: None     Source Type: Journal    
DOI: 10.3152/030234207X258996     Document Type: Article

References (77)

1. Arkes, H 2003. The nonuse of psychological research at two federal agencies. Psychological Research, 14, 1-6.
2. Audretsch, D, B Bozeman, K Combs, M Feldman, A Link, S Siegel, P Stephan and C Wessner 2002. The economics of science and technology. Journal of Technology Transfer, 27(2), 155-203.
3. Boix-Mansilla, V 2006. Assessing expert interdisciplinary work at the frontier: an empirical exploration. Research Evaluation, 15(l), April, 17-30.
4. Boix-Mansilla, V, I Feller and H Gardner 2006. Conference report: quality assessment in interdisciplinary research and education. Research Evaluation, 15(l), April, 69-74.
5. Boyack, K and K Borner 2003. Indicator assisted evaluation of funding of research: visualizing the influence of grants on the number and citation counts of research papers. Journal of the American Society for Information Science and Technology, 54, 447-461.
6. Bozeman, B and J Melkers eds. 1993. Evaluating R&D Impacts: Methods and Practice. Boston MA: Kluwer Academic Publishers.
7. Bozeman, B and D Sarewitz 2005. Public values and public failure in US science policy. Science and Public Policy, 32(2), April, 119-136.
8. Brenneis, D 1994. Discourse and discipline at the National Research Council: a bureaucratic Bildungsroman. Cultural Anthropology, 9, 23-36.
9. Bromley, A 2003. What criteria should be used to establish funding priorities? Physics Today, 56(6), 54-55.
10. Brown, M 2006. Ethics, politics, and the public: shaping the research agenda. In Shaping Science and Technology Policy, eds. D Guston and D Sarewitz, pp. 10-32. Madison WI: University of Wisconsin Press).
11. Bruun, H, J Hukkinen, K Huutoniemi and J Klein 2005. Promoting Interdisciplinary Research: the Case of the Academy of Finland. Helsinki, Finland: Academy of Finland.
12. Cantner, U and H Graf 2006. The network of innovators in Jena: an application of social network analysis. Research Policy, 35, 463-480.
13. Chubin, D and E Hackett 1990. Peerless Science. Albany NY: State University of New York Press.
14. Cohen, I 1985. Revolution in Science. Cambridge MA: Harvard University Press.
15. Cole, S 1992. Making Science. Chicago IL: University of Chicago Press
16. Cozzens, S, K Bobb and I Bortagaray 2002. Evaluating the distributional consequences of science and technology policies and programs. Research Evaluation, 11(2), August, 101-107.
17. Dupree, A Hunter 1957. Science in the Federal Government. New York: Harper Torchbooks.
18. Ergas, H 1987. Does technology policy matter? In Technology and Global Industry, eds. B Guile and H Brooks, pp. 191-245. Washington DC: National Academy of Engineering.
19. Feller, I 2006. Multiple actors, multiple settings, multiple criteria: issues in assessing interdisciplinary research. Research Evaluation, 15(l), April, 5-15.
20. Feller, I and G Gamota 2007. Science indicators as reliable evidence. Minerva, 45, 17-30.
21. Galison, P 1999. Trading zone: coordinating action and beliefs. In The Science Studies Reader, eds. M Baglioli, pp. 137-160. London: Rutledge
22. Gassler, H, W Polt and C Rammer 2007. Priority setting in research & technology policy-historical developments and recent trends. Working paper no 36-2007, Joanneum Research, Institute of Technology and Regional Policy.
23. Geisler, E 2000. The Metrics of Science and Technology. Westport CT: Quorum Books.
24. Georghiou, L 2003. Evaluation of research and innovation policy in Europe: new policies, new frameworks. In Learning from Science and Technology Policy Evaluation, eds. P Shapira and S Kuhlmann, pp. 65-80. Cheltenham: Edward Elgar.
25. Georghiou, L and D Roessner 2000. Evaluating technology programs: tools and methods. Research Policy, 29, 657-678.
26. Giebe, T, T Grebe and E Wolfstetter 2006. How to allocate R&D (and other) subsidies: an experimentally tested policy recommendation. Research Policy, 35, 1261-1272.
27. Guston, D 2003. The expanding role of peer review processes in the United States. In Learning from Science and Technology Policy Evaluation, eds. P Shapira and S Kuhlmann, pp. 81-97. Cheltenham: Edward Elgar.
28. Guy, K 2003. Assessing RTD program portfolios in the European Union. In Learning from Science and Technology Policy Evaluation, eds. P Shapira and S Kuhlmann, pp. 174-203. Cheltenham: Edward Elgar.
29. Hollingsworth, J and E Hollingsworth 2000. Major discoveries and biomedical research organizations: perspectives on interdisciplinarity, nurturing leadership, and integrated structure and cultures. In Practising Interdisciplinarity, eds. P Weingart and N Stehr, pp. 215-244. Toronto: University of Toronto Press).
30. Jaffe, A 2002. Building programme evaluation into the design of public research-support programmes. Oxford Review of Economic Policy, 18, 22-34.
31. Jarmin, R 1999. Evaluating the impact of manufacturing extension on productivity growth. Journal of Policy Analysis and Management, 18, 99-119.
32. Kortum, S 2006. Innovation and the world economy: thoughts on measurement, theory and policy. Paper presented at the NSF/SRS Workshop on Advancing Measures of Innovations: Knowledge Flows, Business Metrics, and Measurement Strategies, Arlington, VA, 6-7 June.
33. Lamont, M and G Mallard 2005. Peer Evaluation in the Social Sciences and Humanities Compared: The United States, the United Kingdom, and France. Report prepared for the Social Sciences and Humanities Research Council of Canada.
34. Langfeldt, L 2006. The policy challenges of peer review: managing bias, conflict of interests and interdisciplinary assessments. Research Evaluation, 15(1), April, 31-41.
35. Leamer, E 1983. Lets take the con out of econometrics. American Economic Review, 73, 31-43.
36. Link, A 1996. Evaluating Public Sector Research and Development. Westport CT: Praeger.
37. Luukkonen, T 1998. The difficulties of assessing the impact of EU Framework Programmes. Research Policy, 27, 599-610.
38. Luukkonen, T 2002. Research evaluation in Europe: state of the art. Research Evaluation, 11(1), April, 81-84.
39. Mallard, G, M Lamont and J Guetzkow 2002. Epistemological pluralism and fairness in peer evaluation: evidence from the social sciences and humanities. Paper presented at the American Sociological Association Annual Meeting, Chicago, IL, 16-19 August.
40. Marburger, J 2002. Science based science policy. Paper presented at the American Association for the Advancement of Science, Annual Meeting, Washington DC, 11-12 April.
41. Marburger, J 2005. Science based science policy. Keynote address, 2005 AAAS Science and Technology Forum, Washington, DC, 21 April.
42. Mark, M M, G T Henry and G Julnes 2000. Evaluation: an Integrated Framework for Understanding, Guiding, and Improving Public and Nonprofit Policies and Programs. San Francisco CA: Jossey-Bass.
43. Moed, H, W Glänzel and U Schmoch eds. 2004. Handbook of Quantitative Science and Technology Research. Dordrecht: Kluwer Academic Publishers.
44. Molas-Gallart, J and A Davies 2006. Toward theory-led evaluation. Journal of Evaluation, 27, 64-82.
45. Mowery, D 1995. The practice of technology policy. In Handbook of the Economics of Innovation and Technological Change, ed. P Stoneman, pp. 513-557. Oxford: Blackwell.
46. National Academies 1983. Frontiers in Science and Technology - a Selected Outlook. Report of the Committee on Science, Engineering, and Public Policy. Washington DC: National Academies Press.
47. National Academies 1999. Evaluating Federal Research Programs. Washington DC: National Academy Press.
48. National Academies 2005. Measuring Research and Development Expenditures in the U.S. Economy. Washington DC: National Academies Press.
49. National Science Board 2007. Enhancing support of transformative research at the National Science Foundation. NSB-07-32. Arlington VA: National Science Foundation
50. National Science Board 2005. Report of the National Science Board on the National Science Foundation's Merit Review System, NSB-05-119. Washington DC: National Science Foundation.
51. NRC, National Research Council 2004. NIH extramural center programs: criteria for initiation and evaluation. Washington DC: National Academies Press.
52. NRC, National Research Council 2005. Facilitating interdisciplinary research. Washington DC: National Academies Press.
53. NRC, National Research Council 2007a. Decadal Science Strategy Surveys, J Alexander ed. Washington DC: National Academies Press.
54. NRC, National Research Council 2007b. Understanding Business Dynamics: an Integrated Data System for America's Future, J Haltiwanger, L Lynch and C Mackie eds. Washington DC: National Academies Press.
55. NRC, National Research Council 2007c. A Strategy for Assessing Science: Behavioral and Social Research on Aging. Irwin Feller and Paul C Stern eds. Committee on Assessing Behavioral and Social Science Research on Aging. Washington DC: The National Academies Press.
56. NSF, National Science Foundation, Directorate of Social, Behavioral and Economic Sciences 2006. Science of Science and Innovation Policy: a Prospectus. Available at < http://www.nsf.gov/sbe/scisip/ scisip_prospec.pfd>, last accessed 25 November 2007.
57. NSF, National Science Foundation 2007. Advancing measures of innovation. Workshop report, NSF 07-306. Arlington VA: National Science Foundation.
58. Nelson, R ed. 1993. National Innovation Systems. New York: Oxford University Press.
59. Polt, W and N Vonortas, Coordinators 2006. IST Evaluation and Monitoring. Report to the European Commission, Contract 30-CE-0039154/00-94. Vienna: Joanneum Research.
60. Porter, A and F Rossini 1985. Peer review of interdisciplinary research. Science, Technology and Human Values, 10(3), July, 33-38.
61. Radin, B 2006. Challenging the Performance Movement. Washington DC: Georgetown University Press.
62. Reeve, N 2005. On the evaluation of European Union research: the 2004 Five-Year Assessment. Science and Public Policy, 32(5), October, 335-338; special issue edited by N Vonortas and S Hinze.
63. Rip, A 1988. Mapping of science: possibilities and limitations. In Handbook of Quantitative Studies of Science and Technology, ed. A van Raan, pp. 253-273. Amsterdam: North-Holland.
64. Rip, A 2003. Societal challenges for R&D evaluation. In Learning from Science and Technology Policy Evaluation, eds. P Shapira and S Kuhlmann, pp. 32-53. Cheltenham: Edward Elgar.
65. Ruegg, R and I Feller 2003. A Toolkit for Evaluating Public R&D Investment. NIST GCR 03-857. Gaithersburg MD: National Institute of Standards and Technology.
66. Salter, A and B Martin 2001. The economic benefits of publicly funded basic research: a critical review. Research Policy, 30, 509-532.
67. Sharif, N 2006. Emergence and development of the national innovation systems concept. Research Policy, 35, 745-766.
68. Shils, E ed, 1968. Criteria for Scientific Development, Public Policy and National Goals. Cambridge MA: MIT Press.
69. Smith, B 1990. American Science Policy Since World War II. Washington DC: Brookings Institution.
70. Travis, G and H Collins 1991. New light on old boys: cognitive and institutional particularism in the peer review system. Science, Technology and Human Values, 16, 322-341.
71. van Raan, A 2004. Measuring science. In Handbook of Quantitative Science and Technology Research, eds. H Moed, W Glänzel and U Schmoch, pp. 19-50. Dordrecht: Kluwer Academic Publishers.
72. Vonortas, N and M Lackey 2003. Real options for evaluating public sector R&D investments. In Learning from Science and Technology Policy Evaluation, eds. P Shapira and S Kuhlmann, pp. 339-351. Cheltenham: Edward Elgar.
73. Weinberg, A 1963. Criteria for scientific choice. Minerva, 1, 159-171.
74. Weinberg, A 1964. Criteria for scientific choice II: the two cultures. Minerva, 111, 3-14.
75. Wenneras, C and A Wold 1997. Nepotism and sexism in peer-review. Nature, 387, 341-343.
76. Zeckhauser, R 2007. Investing in the unknown and unknowable. Kennedy School of Government, Faculty Research Working Papers Series, RWP07-005. Cambridge MA: Harvard University. Available at , last accessed 25 November 2007.
77. Zinoecker, K and M Stampfer 2006. Peer review and beyond? Paper presented at the conference, Peer Review: Its Present and Future State, Prague, 12-13 October.