Toward theory-led evaluation: The experience of European science, technology, and innovation policies

American Journal of Evaluation

Volumn 27, Issue 1, 2006, Pages 64-82

  Molas Gallart, Jordi ^a   Davies, Andrew ^b  

^a CSIC   (Spain)

^b IMPERIAL COLLEGE LONDON   (United Kingdom)

Author keywords

Framework Programme evaluations; Innovation theory; Policy evaluation; Science, technology, and innovation policy

Indexed keywords

EID: 33144473419     PISSN: 10982140     EISSN: None     Source Type: Journal    
DOI: 10.1177/1098214005281701     Document Type: Review

References (60)

1. Arnold, E. (2004). Evaluating research and innovation policy: A systems world needs systems evaluations. Research Evaluation, 13 (1), 3 - 17.
2. Barré, R. (1999). Public research programmes: Socio-economic impact assessment and user needs. IPTS Report Special Issue: Evaluation and Research Activities, 20, 5 - 9.
3. Betz, F. (1994). Basic research and technology transfer. International Journal of Technology Management, 9 (5, 6, 7), 784 - 796.
4. Boden, R., Gummett, P., Cox, D., & Barker, K. (1998). Men in white coats.... Men in grey suits: New Public Management and the funding of scientific research in the UK. Journal of Auditing, Accounting and Accountability, 11 (3), 267 - 291.
5. Bush, V. (1945). Science, the endless frontier(a report to the president by Vannevar Bush, director of the Office of Scientific Research and Development). Washington, DC: U.S. Government Printing Office.
6. Campbell, D., & Julian, S. (1963). Experimental and quasi-experimental designs for research on teaching. Chicago: Rand McNally.
7. Dalpe, R., & Anderson, F. (1993). Evaluating the industrial relevance of public R&D laboratories. In B. Bozeman & J. Melkers (Eds.), Evaluating R&D impacts: Methods and practice (1st ed., pp. 207 - 228). Norwell, MA: Kluwer Academic.
8. Dietz, J. S. (2003). Assessing RTD program portfolios in the European Union. In P. Shapira & S. Kuhlmann (Eds.), Learning for science and technology policy evaluation. experiences from the United States and Europe (pp. 204 - 222). Cheltenham, UK: Edward Elgar.
9. Fahrenkrog, G., Polt, W., Rojo, J., Tubke, A., & Zinocker, K. (Eds.). (2002). RTD evaluation toolbox: Assessing the socio-economic impact of RTD policies. Seville, Spain: Joint Research Centre, European Commission.
10. Feller, I. (2002)Performance measurement redux. American Journal of Evaluation, 23 (4), 435 - 452.
11. Freeman, C., & Soete, L. (1997). The economics of industrial innovation. Cambrdige, MA: MIT Press.
12. Georghiou, L. (1995). Assessing the Framework Programmes - A meta-evaluation. Evaluation, 1 (2), 171 - 188.
13. Georghiou, L. (2003). Evaluation of research and innovation policy in Europe - New policies, new frameworks? In P. Shapira & S. Kuhlmann (Eds.), Learning form science and technology evaluation. Experiences from the United States and Europe (pp. 65 - 79). Cheltenham, UK: Edward Elgar.
14. Georghiou, L., Ribgy, J., & Cameron, H. (2002). Assessing the socio-economic impacts of the Framework Programme. Manchester, UK: Policy Research in Engineering Science and Technology (PREST), University of Manchester.
15. Gibbons, M., & Georghiou, L. (1987). Evaluation of research: A selection of current practices. Paris: Organization for Economic Co-operation and Development.
16. Grupp, H. (2000). R&D evaluation. Research Evaluation, 8 (1), 87 - 99.
17. Guy, K. (2003). Assessing RTD program portfolios in the European Union. In P. Shapira & S. Kuhlmann (Eds.), Learning form science and technology evaluation. Experiences from the United States and Europe (pp. 174 - 203). Cheltenham, UK: Edward Elgar.
18. Guy, K., & Arnold, E. (1998). Strategic options for the evaluation of the R&DProgrammes of the European Union(final report). Brussels, Belgium: European Parliament, Directorate General for Research, Scienctific and Technological Options Assessment.
19. Guy, K., & Nauwelaars, C. (2003). Benchmarking STI policies in Europe: In search of good practice. Institute for Prospective and Technological Studies Report, 71, 20 - 27.
20. Hawkins, R., Montalvo-Corral, C., van Audenhove, L., & Molas-Gallart, J. (2004). Analysis via case studies of the instruments used in the IST programme, submitted to European Commission, DG Information Society, Evaluation and Monitoring (017.31126/01/01).
21. Henry, G., & Mark, M. (2003). Beyond use: Understanding evaluation's influence on attitudes and actions. American Journal of Evaluation, 24 (3), 293 - 314.
22. Hobday, M. (1988). Evaluating collaborative R&Dprogrammes in information technology: The case of the U.K. Alvey programme. Technovation, 8, 271 - 298.
23. Jaffe, A. B. (1998). Measurement issues. In L. M. Branscomb & J. H. Keller (Eds.), Investing in innovation (pp. 64 - 84). Cambridge, MA: MIT Press.
24. Karatzas, I., & Fayl, G. (1999). Editorial. Institute for Prospective and Technological Studies Report Special Issue: Evaluation and Research Activities, 40, 2 - 4.
25. Kastrinos, N. (1994). Evaluating the impact of the EC Framework Programme. Technovation, 14 (10), 679 - 688.
26. Kline, S. J. (1985). Innovation is not a linear process. Research Management, 28 (4), 36 - 45.
27. Kline, S. J., & Rosenberg, N. (1986). An overviewof innovation. In R. Landau & N. Rosenberg (Eds.), The positive sum strategy: Harnessing technology for economic growth (pp. 275 - 305). Washington, DC: National Academy Press.
28. Kostoff, R. N. (1997). The handbook of research impact assessment (No. DTICADA296021). Arlington, VA: Office of Naval Research.
29. Kuhlmann, S. (1999). Distributed intelligence: Combining evaluation, foresight and technology assessment. Institute for Prospective and Technological Studies Report, 40, 16 - 22.
30. Kuhlmann, S. (2003). Evaluation as a source of 'strategic intelligence.' In P. Shapira & S. Kuhlmann (Eds.), Factors affecting technology transfer in industry-US federal laboratory partnerships (pp. 352 - 375). Cheltenham, UK: Edward Elgar.
31. Laredo, P. (1998). The networks promoted by the framework programme and the questions they raise about its formulation and implementation. Research Policy, 27 (6), 589 - 598.
32. Laredo, P., Kahane, B., Meyer, J. B., & Vinck, D. (1992). The research networks built by the MHR4 Programme. Brussels, Belgium: Commission of the European Communities.
33. Leeuw, F. L. (2003). Reconstructing program theories: Methods available and problems to be solved. American Journal of Evaluation, 24 (1), 5 - 20.
34. Lundvall, B.-A. (1992). National systems of innovation: Towards a theory of innovation and interactive learning (1st ed.) London: Pinter.
35. Luukkonen, T. (1998). The difficulties in assessing the impact of EU framework programmes. Research Policy, 27 (6), 599 - 610.
36. Luukkonen, T. (2002). Research evaluation in Europe: State of the art. Research Evaluation, 11 (2), 81 - 84.
37. Mayne, J. (2001). Attribution through contribution analysis: Using performance measures sensibly. Canadian Journal of Program Evaluation, 16 (1), 1 - 24.
38. Meyer-Krahmer, F., & Reiss, T. (1992). Ex ante evaluation and technology assessment - Two emerging elements of technology policy evaluation. Research Evaluation, 2 (1), 47 - 54.
39. Molas-Gallart, J., Salter, A., Patel, P., Scott, A., & Duran, X. (2002). Measuring third stream activities. Brighton, UK: SPRU.
40. Molas-Gallart, J., Tang, P., Sinclair, T., Morrow, S., & Martin, B. (1999). Assessing research impact on non-academic audiences. Brighton, UK: SPRU.
41. Mowery, D. C., & Rosenberg, N. (1979). The influence of market demand upon innovation: A critical review of some recent empirical studies. Research Policy, 8, 105 - 153.
42. O'Doherty, D., & Arnold, E. (2003). Understanding innovation: The need for a systemic approach. Institute for Prospective and Technological Studies Report, 71, 29 - 36.
43. Office of Science and Technology. (1996). Winning through foresight. A strategy taking the foresight programme to the millenium. London: Department of Trade and Industry.
44. Organization for Economic Co-operation and Development. (2002). Benchmarking industry-science relationships. Paris: Author.
45. Policy Research in Engineering Science and Technology (PREST). (1990). The impact and utility of European Commission research programme evaluation reports (EUR13098 EN). Brussels, Belgium: European Commission.
46. Policy Research in Engineering Science and Technology (PREST), AUEB, BETA, ISI, Joanneum Research, IE HAS, et al. (2002). Assessing the socio-economic impact of the framework programme. Manchester, UK: University of Manchester.
47. Rip, A. (2003). Societal challenges for R&Devaluation. In P. Shapira & S. Kuhlmann (Eds.), Learning form science and technology evaluation. Experiences from the United States and Europe (pp. 32 - 53). Cheltenham, UK: Edward Elgar.
48. Roessner, J. D. (2002). Outcome measurement in the USA: State of the art. Research Evaluation, 11 (2), 85 - 93.
49. Rothwell, R. (1992). Successful industrial innovation: Critical factors for the 1990s. R&D Management, 22 (3), 221 - 239.
50. Rothwell, R. (1993, May). Systems integration and networking: The fifth generation innovation process. Paper presented at the Chaire Hydro-Quebec Conference en Gestion de al Technologie, Montreal, Quebec, Canada.
51. Sanz Menéndez, L., Fernández Carro, J. R., & García, C. E. (1999). Centralidad y cohesión en las redes de colaboración empresarial en la I+D subsidiada [Centrality and cohesion in interfirm collaborative networks participating in publicly funded R&D programs]. Papeles de Economía Española, 81, 219 - 241.
52. Schmookler, J. (1966). Invention and economic growth. Boston: Harvard University Press.
53. Scriven, M. (1967). The methodology of evaluation. In R. W. Tyler, R. M. Gagné, & M. Scriven (Eds.), Perspectives of curriculum evaluation (pp. 39 - 83). Chicago: Rand McNally.
54. Shapira, P. (2003). Evaluating manufacturing extension services in the United States: Experiences and insights. In P. Shapira & S. Kuhlmann (Eds.), Factors affecting technology transfer in industry-US federal laboratory partnerships (pp. 260 - 292). Cheltenham, UK: Edward Elgar.
55. Sherwin, C. W., & Isenson, R. S. (1967, March). Estimating the science and technology components of anR&Dbudget. Paper presented at the Second Cost-Effectiveness Symposium of the Washington Operations Research Council, Washington, DC.
56. Soete, L., & Corpakis, D. (2003). Editorial: R&D for competitiveness and employment - The role of benchmarking. Institute for Prospective and Technological Studies Report, 71, 2 - 12.
57. Stame, N. (2004). Theory-led evaluation and types of complexity. Evaluation, 10 (1), 58 - 76.
58. STRATA-ETAN Expert Working Group. (2002). Benchmarking national research policies: The Impact of RTD on Competitiveness and Employment (IRCE). Brussels, Belgium: European Commission Directorate General Research.
59. van Raan, A. F. J. (2000). R&D evaluation at the beginning of the new century. Research Evaluation, 8 (2), 81 - 86.
60. von Hippel, E. (1988). The sources of innovation. New York: Oxford University Press.