Evaluation of research and innovation policies: A discussion of trends with examples from Germany

International Journal of Technology Management

Volumn 26, Issue 2-4, 2003, Pages 131-149

  Kuhlmann, Stefan ^a  

^a FRAUNHOFER INSTITUTE FOR SYSTEMS AND INNOVATION RESEARCH ISI   (Germany)

Author keywords

Evaluation of innovation programs; German research system; Self referential evaluation; Summative and formative evaluation

Indexed keywords

ECONOMIC AND SOCIAL EFFECTS; ECONOMICS; ENGINEERING RESEARCH; EVALUATION; SOCIETIES AND INSTITUTIONS;

FORMATIVE EVALUATION; SELF-REFERENTIAL EVALUATION;

PUBLIC POLICY;

EID: 0038141806     PISSN: 02675730     EISSN: None     Source Type: Journal    
DOI: 10.1504/ijtm.2003.003366     Document Type: Article

References (87)

1. Luhmann, N. (1990) Die Wissenschaft der Gesellschaft, Frankfurt/M.
2. European Commission (1999) Second European Report on S&T Indicators. Key Figures, Luxembourg.
3. Grimmer, K., Häusler, J., Kuhlmann, S. and Simonis, G. (Eds.) (1992) Politische Techniksteuerung - Forschungsstand und Forschungsperspektiven, Opladen.
4. Martinsen, R. and Simonis, G. (Eds.) (1995) Paradigmenwechsel in der Technologiepolitik? Opladen.
5. Krull, W. and Meyer-Krahmer, F. (1996) (Eds.) Science and technology in Germany, London.
6. Ostry, S. and Nelson, R.R. (1995) Techno-Nationalism and Techno-Globalism. Conflict and Cooperation, Washington.
7. Roobeek, A.J.M. (1990) Beyond the Technology Race. An Analysis of Technology Policy in Seven Industrial Countries, Amsterdam et al.
8. Neidhardt, F. (1988) Selbststeuerung in der Forschungsförderung. Das Gutachterwesen der DFG, Opladen.
9. Ergas, H. (1987) 'Does technology policy matter?', in B.R. Guile and H. Brooks (Eds.) Technology and Global Industry. Companies and Nations in the World Economy, Washington D.C., pp.191-245.
10. BMBF - German Federal Department for Education and Research (2002) Faktenbericht Forschung 2002, Bonn.
11. Meyer-Krahmer, F. and Kuntze, U. (1992) 'Bestandsaufnahme der Forschungs- und Technologiepolitik', in K. Grimmer, J. Häusler, S. Kuhlmann and G. Simonis (Eds.) Politische Techniksteuerung - Forschungsstand und Forschungsperspektiven, Opladen, pp.95-118.
12. Kuhlmann, S. (2000) 'Evaluation in der Forschungs- und Innovationspolitik', in R. Stockmann (Ed.) Evaluationsforschung, Leverkusen, pp.287-307.
13. Bozemann, B. and Melkers, J. (Eds.) (1993) Evaluating R&D Impacts: Methods and Practice, Boston et al.
14. Georghiou, L. (1995) 'Research evaluation in European national science and technology systems', Research Evaluation, Vol. 5, pp.3-10.
15. Georghiou, L. (1998) 'Issues in the evaluation of innovation and technology policy', Evaluation, Vol. 4, pp.37-51.
16. OECD - Organisation of Economic Cooperation and Development (Ed.) (1997) Policy Evaluation in Innovation and Technology. Towards Best Practices, Paris.
17. Daniel, H.-D. (1993) Guardians of Science. Fairness and Reliability of Peer Review, Weinheim et al.
18. Hornbostel, S. (1997) Wissenschaftsindikatoren. Bewertungen in der Wissenschaft, Opladen.
19. van Raan, A.F.J. (Ed.) (1988) Handbook of Quantitative Studies of Science and Technology, Amsterdam.
20. Campbell, D.F.J. and Felderer, B. (1997) Evaluating Academic Research in Germany. Patterns and Policies, Vienna.
21. Schön, D. and Rein, M. (1994) Frame Reflection. Toward the Resolution of Intractable Policy Controversies, New York.
22. Widmer, T. (1996) Meta-Evaluation, Kriterien zur Bewertung von Evaluationen, Bern et al.
23. Patton, M.Q. (1997) Utilization-Focused Evaluation. The New Century Text, Thousand Oaks et al.
24. Kuhlmann, S. (1998) Politikmoderation. Evaluationsverfahren in der Forschungs- und Technologiepolitik, Baden-Baden.
25. Wollmann, H. (1998) Evaluation Research and Politics: Between a Science-Driven and a Pluralist Controversy-Responsive Policy-Making Model. Potential and limitations. Rome.
26. Derlien, H-U. (1976) Die Erfolgskontrolle staatlicher Planung. Eine empirische Untersuchung über Organisation, Methode und Politik der Programmevaluation, Baden-Baden.
27. Mayntz, R. (1980) Implementation politischer Programme. Empirische Forschungsberichte, Königstein.
28. Mayntz, R. (1983). Implementation politischer Programme II. Ansätze zur Theoriebildung, Opladen.
29. Hellstern, G.-M. and Wollmann, H. (1984) Handbuch zur Evaluierungsforschung, Bd. 1, Opladen.
30. Meyer-Krahmer, F. (1989) Der Einfluß staatlicher Technologiepolitik auf industrielle Innovationen, Baden-Baden.
31. Kuhlmann, S. and Holland, D. (1995) Evaluation von Technologiepolitik in Deutschland - Konzepte, Anwendung, Perspektiven, Heidelberg.
32. Becher, G. and Kuhlmann, S. (1995) (Eds.) Evaluation of Technology Policy Programmes in Germany, Boston et al.
33. Block, H.-J. and Krull, W. (1990) 'What are the consequences? Reflections on the impact of evaluations conducted by a science policy advisory body', Scientometrics, Vol. 19, pp.427-437.
34. Kuhlmann, S. and Holland, D. (1995) Erfolgsfaktoren der wirtschaftsnahen Forschung, Heidelberg.
35. BLK, Bund-Länder-Kommission für Bildungsplanung und Forschungsförderung, (1999) Bericht der internationalen Kommission zur Systemevaluation der DFG und der MPG, http://www.blk-bonn.de/papers/forschungsfoerderung.pdf.
36. FhG - Fraunhofer-Gesellschaft (1998) Systemevaluierung der Fraunhofer-Gesellschaft, Bericht der Evaluierungskommission, Download ftp://192.76.176.135/eval_fhg.pdf
37. Wissenschaftsrat (2000) Systemevaluation der Blauen Liste - Stellungnahme des Wissenschaftsrates zum Abschluss der Bewertung der Einrichtungen der Blauen Liste, Köln.
38. Röbbecke, M. and Simon, D. (1999) Zwischen Reputation und Markt. Ziele, Verfahren und Instrumente von (Selbst)Evaluierungen außeruniversitärer, öffentlicher Forschungseinrichtungen, Berlin.
39. Wissenschaftsrat (2001) Systemevaluation der HGF - Stellungnahme des Wissenschaftsrates zur Hermann von Helmholtz-Gemeinschaft Deutscher Forschungszentren, Köln.
40. Brockhoff, K. (1999) Forschung und Entwicklung. Planung und Kontrolle, 5th ed., München et al.
41. Gerpott, T.J. (1999) Strategisches Technologie- und Innovationsmanagement. Eine konzentrierte Einführung, Stuttgart.
42. Bürgel, H.D., Haller, C. and Binder, M. (1996) F&E-Management, München.
43. Airaghi, A., Busch, N.E., Georghiou, L., Kuhlmann, S., Ledoux, M.J. van Raan, A.F.J. and Viana Baptista, J. (1999) 'Options and limits for assessing the socio-economic impact of European RTD programmes', Report to the European Commission, DG XII, Evaluation Unit, Brussels/Luxembourg.
44. Grupp, H. (1988) Foundations of the Economics of Innovation: Theory, Measurement and Practice, Cheltenham.
45. Barré, R. (2001) 'Sense and nonsense of S&T productivity indicators', Science and Public Policy, Vol. 28, pp.1-8.
46. European Commission (Ed.) 2002 RTD Evaluation Toolbox, a Report Prepared by the Thematic Network 'Socio - Economic Evaluation of Public RTD Policies (EPUB), forthcoming.
47. Kuhlmann, S. (1998) 'Moderation of policy-making? Science and technology policy evaluation beyond impact measurement: the case of Germany', Evaluation, Vol. 4, pp.130-148.
48. Callon, M., Larédo, P. and Mustar, P. (1995) La gestion stratégique de la recherche et de la technologie. L'evaluation des programmes, Paris.
49. Rip, A. (1990) 'Implementation and evaluation of science & technology priorities and programs', in S. Cozzens (Ed.) The Research System in Transition, Boston, pp.263-280.
50. Cunningham, P.N., Georghiou, L.G., Barker, K.E., Kuhlmann, S., Reger, G., Marciano da Silva, C. and Henriques, L. (1994) Analysis of Experience in the Use of Verifiable Objectives, Brussels/Luxembourg.
51. Mowery, D.C. (1994) Science and Technology Policy in Interdependent Economies, Boston et al.
52. Freeman, C. (1987) Technology Policy and Economic Performance: Lessons from Japan, London.
53. Lundvall, B.-Å. (Ed.) (1992) National Systems of Innovation: Towards a Theory of Innovation and Interactive Learning, London.
54. Nelson, R.R. (Ed.) (1993) National Innovation Systems: A Comparative Analysis, Oxford, New York.
55. Edquist, C. (Ed.) (1997) Systems of Innovation. Technologies, Institutions and Organization, London, Washington.
56. Kuhlmann, S. and Meyer-Krahmer, F. (2001) 'Internationalisation of innovation, interdependence, and innovation policy for sustainable development', in G.P. Sweeney (Ed.) Innovation, Economic Progress and Quality of Life, Cheltenham, pp.86-110.
57. Amable, B., Barré, R. and Boyer, R. (1997) 'Diversity, coherence and transformations of innovation systems', in B. Barré, M. Gibbons, Sir J. Maddox, B. Marin and P. Papon (Eds.) Science in Tomorrow's Europe, Paris, pp.33-49.
58. Schmoch, U., Hinze, S., Jäckel, G., Kirsch, N., Meyer-Krahmer, F. and Münt, G. (1996) 'The role of the scientific community in the generation of technology', in G. Reger and U. Schmoch (Eds.) Organisation of Science and Technology at the Watershed. The Academic and Industrial Perspective, Heidelberg, pp.1-138.
59. Meyer-Krahmer, F. and Schmoch, U. (1998) 'Science-based technologies: university-industry interactions in four fields', Research Policy, Vol. 27, pp.835-851.
60. Kodama, F. (1995) Emerging Patterns of Innovation. Sources of Japan's Technological Edge, Boston.
61. Schmoch, U., Breiner, S., Cuhls, K., Hinze, S. and Münt, G. (1996) 'The organisation of interdisciplinary - research structures in the areas of medical lasers and neural networks', in G. Reger and U. Schmoch. (Eds.) Organisation of Science and Technology at the Watershed. The Academic and Industrial Perspective, Heidelberg, pp.267-372.
62. van der Meulen, B. and Rip, A. (1994) Research Institutes in Transition, Delft.
63. Kuhlman, S. and Roger, G. (1996) 'Technology-intensive SMEs: policies supporting the management of growing technological complexity', in W. Cannell and B. Dankbaar (Eds.) Technology Management and Public Policy in the European Union, Luxembourg, Oxford, pp.73-102.
64. Meyer-Krahmer, F. (1997) 'Science-based technologies and interdisciplinarity: challenges for firms and policy', in Ch. Edquist (Ed.) Systems of Innovation. Technologies, Institutions and Organizations, London, Washington, pp.298-317.
65. Gibbons, M., Limoges, C., Nowttny, H., Schwartzman, S., Scott, P. and Trow, M. (1994) The New Production of Knowledge, The Dynamics of Science and Research in Contemporary Societies, London et al.
66. Etzkowitz, H. and Leydesdorff, L. (2000) 'The dynamics of innovation: from National Systems and 'Mode 2' to a Triple Helix of university-industry-government relations', Research Policy, Vol. 29, pp.109-123.
67. Callon, M. (1992) 'The dynamics of techno-economic networks', in R. Coombs, P. Saviotti, and V. Walsh (Eds.) Technological Change and Company Strategies: Economic and Sociological Perspectives, London et al., 72-102.
68. Niosi, J. (Ed.) (1999) 'The internationalization of industrial R&D', Research Policy (Special Issue), Vol. 28, pp.107-336.
69. Meyer-Krahmer, F. (1999) 'Was bedeutet Globalisierung für Aufgaben und Handlungsspielräume nationaler Innovationspolitiken?', in K. Grimmer, S. Kuhlmann and F. Meyer-Krahmer (Eds.) Innovationspolitik in globalisierten Arenen. Neue Aufgaben für Forschung und Lehre: Forschungs- und Technologiepolitik im Wandel, Leverkusen, pp.35-65.
70. Lundvall, B.-Å, and Borrás, S. (1998) The Globalising Learning Economy: Implications for Innovation Policy, Luxembourg.
71. Shapira, P. and Kuhlmann, S. (Eds.) (2003) Learning from Science and Technology Policy Evaluation: Experiences from the United States and Europe, Cheltenham, forthcoming.
72. Osborne, D. and Gaebler, T. (1993) Reinventing Government: How the Entrepreneurial Spirit Is Transforming the Public Sector, New York.
73. Shapira, P., Kingsley, G. and Youtie, J. (1997) Manufacturing Partnerships: Evaluation in the Context of Government Reform, Evaluation and Program Planning, Vol. 2, pp.103-112.
74. Cozzens, S. and Melkers, J. (1997) 'Use and usefulness of performance measurement in state science and technology programs', Policy Studies Journal, Vol. 25, p.2.
75. Ruegg, R. (1998) 'Symposium overview. Symposium on evaluating a public private partnership: the advanced technology program', The Journal of Technology Transfer, Vol. 23, p.2.
76. Guba, E.G. and Lincoln, Y.S. (1989) Fourth Generation Evaluation, Newbury Park et al.
77. Worthen, B.R., Sanders, J.R. and Fitzpatrick, J.L. (1997) Program Evaluation. Alternative Approaches and Practical Guidelines, 2nd ed., White Plains, NY.
78. Smits, R. and Kuhlmann, S. (2002) Strengthening Interfaces in Innovation Systems: Rationale, Concepts and (New) Instruments, Brussels.
79. van den Daele, W. (1997) 'Risikodiskussionen am 'Runden Tisch'. Partizipative Technikfolgenabschätzung zu gentechnisch erzeugten herbizidresistenten Pflanzen', in R. Martinsen (Ed.) Politik und Biotechnologie. Die Zumutung der Zukunft, Baden-Baden, pp.281-301.
80. Kuhlmann, S., Boekholt, P., Georghiou, L., Guy, K., Héraud, J.-A., Laredo, Ph., Lemola, T., Loveridge, D., Luukkonen, T., Polt, W., Rip, A., Sanz-Menendez, L. and Smits, R. (1999) Improving Distributed Intelligence in Complex Innovation Systems, Brussels, Luxembourg, http://www.isi.fhg.de/abtlg/ti/pb_html/final.pdf.
81. Kuhlmann, S. (2001) Management of Innovation Systems: The Role of Distributed Intelligence, Apeldoorn, Antwerpen.
82. Grupp, H. (1998) (Ed.) 'Technological forecasting & social change', Special Issue on Recent National Foresight Activities, Vol. 57.
83. Cuhls, K. and Kuwahara, T. (1994) Outlook for Japanese and German Future Technology. Comparing Technology Forecast Surveys, Heidelberg.
84. Rip, A., Misa, Th. J. and Schot, J. (Eds.) (1995) Managing Technology in Society. The Approach of Constructive Technology Assessment, London, New York.
85. Braun, D., Hinze, S., Hüsing, B., Kuhlmann, S., Menrad, K. and Peter, V. (1997) Interdisziplinäre Zentren für Klinische Forschung in der Frühphase der Bundesförderung - Vergleichende Analyse, Stuttgart.
86. Bührer, S., Peter, V., Braun, D. and Kuhlmann, S. (1999) Interdisziplinäre Zentren für Klinische Forschung - Kommunikation & Kooperation: Prozessevaluation der Fördermaßnahme 'Interdisziplinäre Zentren für Klinische Forschung an den Hochschulkliniken', Stuttgart.
87. Bührer, S., Bierhals, R., Heinze, Th., Hullmann, A., Studer, Th., Erlinghagen, R. and Lang, C. (2002) Die Kompetenzzentren der Nanotechnologie in der Frühphase der Bundesförderung: Ein Bericht der begleitenden Evaluation, Karlsruhe.