Improving policy understanding by means of secondary analyses of policy evaluation

Research Evaluation

Volumn 17, Issue 3, 2008, Pages 175-186

  Edler, Jakob ^a   Ebersberger, Bernd ^b   Lo, Vivien ^c  

^a UNIVERSITY OF MANCHESTER   (United Kingdom)

^b MANAGEMENT CENTER INNSBRUCK   (Austria)

^c FRAUNHOFER INSTITUTE FOR SYSTEMS AND INNOVATION RESEARCH ISI   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 54249169182     PISSN: 09582029     EISSN: None     Source Type: Journal    
DOI: 10.3152/095820208X331667     Document Type: Article

References (65)

1. Arnold, E 2004 Evaluation research and innovation policy: a systems world needs systems evaluations. Research Evaluation, 13(1), 3-17.
2. Arnold, E, J Clark and A Muscio 2005 What the evaluation record tells us about European Union Framework Program performance. Special issue of Science and Public Policy on evaluation of European Union Framework Programmmes: the 2004 Five-Year Assessment, eds. N S Vonortas and S Hinze, 32(5), October, 385-397.
3. Arnold, E, S Kuhlmann and B A van der Meulen 2001 Singular Council: Evaluation of the Research Council of Norway. Brighton: Technopolis.
4. Beelmann, A and T Bliesener 1994 Aktuelle Probleme und Strategien der Metaanalyse. Psychologische Rundschau, 45, 211-233.
5. Bewyl & Associates GmbH 2004 Glossar wirkungsorientierte Evaluation. Köln: Univation-Institut für Evaluation.
6. Blanpied, W A 2004 Achievements of the science and technology basic plans: impressions of a sympathetic foreigner. Paper presented at the NISTEP international workshop on comprehensive review of Japan's science and technology basic plans, Tokyo, Japan, 13-14 September.
7. Borgmann, M 2005 Evaluation Synthesis zu Angeboten der Wissenschaftskommunikation im Rahmen der Evaluation des Jahrs der Technik 2004. Studie im Auftrag des BMBF. Köln.
8. Borrás, S 2004 System of innovation theory and the European Union. In special issue of Science and Public Policy on a European system of innovation, ed. S Borrás, 31(6), December, 425-433.
9. Carlsson, B ed. 1991 Technological Systems and Industrial Dynamics. Boston: Kluwer.
10. Carlsson, B ed. 1995 Technological Systems and Economic Performance: the Case of Factory Automation. Boston: Kluwer.
11. Carlsson, B and R Stankiewicz 1991 On the nature, function, and composition of technological systems. Journal of Evolutionary Economics, 1(2), 93-118.
12. Carlsson, B, M Holmén, S Jacobsson, A Rickne and R Stankewicz 2002a Analytical approach and methodology. In Technological Systems in the Bio Industries: an International Study, ed. B Carlsson. Boston: Kluwer.
13. Carlsson, B, S Jacobsson, M Holmén and A Rickne 2002b, Innovation systems: analytical and methodological issues. Research Policy, 31, 233-245.
14. Cooke, P, M Heidenreich and H-J Braczyk eds. 1998 Regional Innovation Systems: the Role of Governances in a Globalized World. London: Routledge.
15. Cooksy, L J and V J Caracelli 2005 Quality, context and use: issues in achieving the goals of metaevaluations. American Journal of Evaluation, 26(1), 31-42.
16. DeCoster, J 2004 Meta-analysis notes. Retrieved 3 October 2006 from 〈http://www.stat-help.com/notes.html〉.
17. DeGEval, German Evaluation Society 2001 Evaluation standards. Retrieved 30 April 2007 from 〈http://www.buero-evaluation. de/DeGeval_standards. htm〉.
18. Drinkmann, A 1990 Methodenkritische Untersuchungen zur Metaanalyse. Weinheim: Deutscher Studien Verlag.
19. Edler, J, E Amanatidou, S Bührer, P Cunningham, W Polt, K Guy and J von Oertzen 2006 Perspectives on Evaluation and Monitoring. Project proposal of Fraunhofer ISI, PREST, Joanneum, Wise Guys, and Atlantis for the European Commission DG Enterprise, project awarded November.
20. Edler, J, C Dreher and B Ebersberger 2007 Market and system failures and policy instruments in the technology cycle. In Developing an Assessment Framework to Improve the Efficiency of R&D and the Market Diffusion of Energy Technologies, ed. E Jochem. Project report to the BMWi.
21. Edquist, C ed. 1997 Systems of Innovation: Technologies, Institutions, and Organizations. London: Pinter.
22. European Evaluation Society 2007 Evaluation standards. Retrieved 30 April 2007 from 〈http://www.europeanevaluation.org/?page=756983〉.
23. Fahrenkrog, G, W Polt, J Rojo, A Tübke and K Zinöcker 2002 RTD Evaluation Toolbox: Assessing the Socio-Economic Impact of RTD Policies (EUR 20382 EN). Seville: IPTS.
24. fteval, Austrian Evaluation Platform 2003 Evaluation Standards in Research and Technology Policy. Wien: Plattform Forschungund Technologieevaluierung.
25. GAO, United States General Accounting Office 1992 The Evaluation Synthesis (GAO/PEMD-10.1.6). Washington: GAO.
26. Georghiou, L 1995. Research evaluation in European national science and technology systems. Research Evaluation, 5(1), 3-10.
27. Georghiou, L 1999 Meta evaluation: evaluation of evaluations. Scientometrics, 4(3), 523-530.
28. Georghiou, L and D Roessner 2000 Evaluating technology programs: tools and methods. Research Policy, 29(4-5), 6.
29. Georghiou, L, E Amanatidou, H Belitz, L Cruz, J Edler, C Edquist, O Granstrand, J Guinet, E Leprince, L Orsenigo, J Rigby, J Romanainen, M Stampfer and J van den Biesen 2003 Improving the Effectiveness of Direct Public Support Measures to Stimulate Private Investment in Research (Report of the ETAN Working Group on Direct Measures for Directorate General Research). Brussels: European Commission.
30. Glass, G V 1976 Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3-8.
31. Hedges, L V and I Olkin 1985 Statistical Methods for Meta Analysis. San Diego, CA: Academic Press.
32. Hekkert. M P, R A Suurs, S Kuhlmann and R E Smits 2007 Functions of innovation systems: a new approach for analyzing technological change. Technological Forecasting and Social Change, 74, 413-432.
33. Herman, J L 1987 CSE Program Evaluation Kit. London: Sage.
34. Hunter, J E and F L Schmidt 2004 Methods of Meta-Analysis: Correcting Error and Bias in Research Findings. Newbury Park: Sage.
35. Jochem, E ed. 2007 Developing an Assessment Framework to Improve the Efficiency of R&D and the Market Diffusion of Energy Technologies. Project Report to the BMWi.
36. Joint Committee on Standards for Educational Evaluation 1994 The Program Evaluation Standards: How to Assess Evaluation of Educational Programs, 2nd edn. Thousand Oaks: Sage.
37. Kondo, M 2004 Highlights of the comprehensive review of Japan's science and technology basic plans. Presentation at the NISTEP international workshop on comprehensive review of Japan's science and technology basic plans, Tokyo, Japan, 13-14 September.
38. Koschatzky, K and V Lo 2005 Innovationspolitik in den neuen Ländern. Bestandsaufnahme und Gestaltungsmöglichkeiten. Stuttgart: Fraunhofer IRB.
39. Kuhlmann, S 2001 Management of Innovation System: the Role of Distributed Intelligence, Nijmwegen Lectures on Innovation Management. Antwerp: Apeldorn.
40. Kuhlmann, S, P Boekholt, L Georghiou, K Guy, J-A Héraud, P Laredo, T Lemola, D Loveridge, T Luukkonen, W Polt, A Rip, L Sanz-Mendez and R Smits 1999 Improving Distributed Intelligence in Complex Innovation Systems (Final Report to the ASTPP). Karlsruhe: Fraunhofer ISI.
41. Light, R 1984 Six evaluation issues that synthesis can resolve better than single studies. In New Directions for Program Evaluation: Issues in Data Synthesis, eds. W H Yeaton and P M Wortmann. San Francisco: Verlag.
42. Lipsey, M W and D B Wilson 2000 Practical Meta Analysis. Thousand Oaks: Sage.
43. LL&A, PREST, ANRT, and Reidev Ltd. 2006a SMART INNOVATION: A practical guide to evaluating innovation programs. Retrieved 30 April 2007, from 〈http://cordis.europa.eu/innovation-policy/studies/gen_study14.htm〉.
44. LL&A, PREST, ANRT, and Reidev Ltd. 2006b Supporting the monitoring and evaluation of innovation programs. Retrieved 30 April 2007, from 〈http://cordis.europa.eu/innovation-policy/studies/gen_study14.htm〉.
45. Lundvall, B A ed. 1993 National Innovation Systems: Towards a Theory of Innovation and Interactive Learning. London: Pinter.
46. Lundvall, B A and M Tomlinson 2001 International benchmarking as a policy learning tool. In The New Knowledge Economy in Europe, ed. M Joao Rodrigues. Cheltenham: Edward Elgar.
47. Malerba, F 2002 Sectoral systems of innovation and production. Research Policy, 31, 247-264.
48. Malerba, F 2004 Sectoral systems of innovation: basic concepts. In Sectoral Systems of Innovation: Concepts, Issues and Analyses of Six Major Sectors in Europe, ed. F Malerba. Cambridge: Verlag.
49. Meyer-Krahmer, F and U Kuntze 1992 Bestandsaufnahme der Forschungs- und Technologiepolitik. In Politische Techniksteuerung - Forschungsstand und Forschungsperspektiven, eds. K Grimmer, J Häusler, S Kuhlmann and G Simonis. Opladen: Leske & Budrich.
50. Miller, N and V E Pollock 1994 Meta-analytic synthesis for theory development. In Handbook of Research Synthesis, eds. H Cooper and L V Hedges, New York: Russel Sage Foundation.
51. Nelson, R R ed. 1993 National Innovation Systems: a Comparative Analysis. Oxford: Oxford University Press.
52. OECD, Organization for Economic and Development 1998 Best practice guidelines for evaluation (PUMA Policy Brief No. 5, May). Retrieved 30 April 2006 from 〈http://www.oecd.org/dataoecd/11/56/1902965.pdf〉.
53. Paasi, M 2005 Collective benchmarking of policies: an instrument for policy learning in adaptive research and innovation policy. Science and Public Policy, 32(1), February,17-27.
54. Ruegg, R 2005 The Advanced Technology Program's evaluation plan and progress. Retrieved 10 November 2007 from 〈http://www.atp.nist.gov/eao/7th- iftm.htm〉.
55. Ruegg, R and I Feller 2003 A Toolkit for Evaluating Public R&D Investment, Models, Methods, and Findings, from ATP's First Decade. Washington: Department of Commerce, Technology Administration, and National Institute of Standards and Technology.
56. Sanders, J R ed. 1991 Handbuch der Evaluationsstandards. Opladen: Verlag Leske & Budrich.
57. Scriven, M 1991 Evaluation Thesaurus, 4th edn. Newbury Park: Sage.
58. Stanley, T D 2001 Wheat from chaff: meta-analysis as quantitative literature review. Journal of Economic Perspectives, 15, 131-150.
59. Stufflebeam, D L 2000 The methodology of meta-evaluation as reflected in meta-evaluations by the Western University Evaluation Center. Journal of Personnel Evaluation in Education, 14(1), 95-125.
60. Stufflebeam, D L 2001 The meta-evaluation imperative. American Journal of Evaluation, 22(2), 183-209.
61. Sutton, A 2000 Methods for Meta-Analysis in Medical Research. Chichester: John Wiley & Sons, Ltd.
62. Toulemonde, J 2000 Evaluation culture(s) in Europe : differences and convergence between national practices. Vierteljahrshefte zur Wirtschaftsforschung - DIW, 69(3), 350-357.
63. Vonortas, N S and S Hinze eds. 2005 Special issue of Science and Public Policy on evaluation of European Union Framework Programmmes: the 2004 Five-Year Assessment, 32(5), October.
64. Widmer, T 1996 Meta-evaluation: Kriterien zur Bewertung von Evaluationen. Bern: Haupt.
65. Widmer, T and W Bewyl 1999 Die Übertragbarkeit der Evaluationsstandards auf unterschiedliche Evaluationsfelder. In Handbuch der Evaluationsstandards, ed. J R Sanders. Opladen: Verlag Leske & Budrich.