Impact oriented monitoring: A new methodology for monitoring and evaluation of international public health research projects

Research Evaluation

Volumn 24, Issue 2, 2015, Pages 131-145

  Guinea, J ^a   Sela, E ^a   Gomez Nunez A J ^a   Mangwende, T ^b   Ambali, A ^b   Ngum, N ^b   Jaramillo, H ^c   Gallego, J M ^c   Patino A ^c   Latorre, C ^d   Srivanichakorn, S ^e   Thepthien, B ^e  

^a INNOVATEC   (Spain)

^b UNL IBM Global Innovation Hub   (South Africa)

^c UNIVERSIDAD DEL ROSARIO   (Colombia)

^d Mathematics Department   (Colombia)

^e MAHIDOL UNIVERSITY   (Thailand)

Author keywords

Impact identification and assessment; International public health; Monitoring and evaluation; Research projects

Indexed keywords

EID: 84942254906     PISSN: 09582029     EISSN: 14715449     Source Type: Journal    
DOI: 10.1093/reseval/rvu034     Document Type: Article

References (69)

1. Banzi, R. et al. (2011) 'Conceptual Frameworks and Empirical Approaches Used to Assess the Impact of Health Research: An Overview of Reviews', Health Research Policy and Systems, 9/26. accessed 14 Oct 2014.
2. Barrett, R. and Curran, B. (2010) Analysis and Outputs of HRB Grants: 2008 and 2009. Retrieved from accessed 14 Oct 2014.
3. Bloch, C. et al. (2014) 'Developing a Methodology to Assess the Impact of Research Grant Funding: A Mixed Methods Approach', Evaluation and Program Planning, 43: 105-17.
4. Boaz, A., Fitzpatrick, S. and Shaw, B. (2008) Assessing the Impact of Research on Policy: A Review of the Literature for a Project on Bridging Research and Policy Through Outcome Evaluation. Retrieved from accessed 14 Oct 2014.
5. -. (2009) 'Assessing the Impact of Research on Policy: A Literature Review', Science and Public Policy, 36/4: 255-70.
6. Bornmann, L. (2012) 'Measuring the Societal Impact of Research: Research Is Less and Less Assessed on Scientific Impact Alone - We Should Aim to Quantify the Increasingly Important Contributions of Science to Society', EMBO Reports, 13/8: 673-6.
7. Boyd, A. et al. (2013) 'Frameworks for Evaluating Health Research Capacity Strengthening: A Qualitative Study', Health Research Policy and Systems, 11/46. accessed 14 Oct 2014.
8. Bozeman, B. and Sarewitz, D. (2011) 'Public Value Mapping and Science Policy Evaluation', Minerva, 49/1: 1-23.
9. Buxton, M. et al. (2000) 'Assessing Benefits from Department of Health and National Health Service Research & Development', Public Money and Management, 20/4: 29-34.
10. Buxton, M. J. and Hanney, S. (1996) 'How Can Payback from Health Services Research Be Assessed?', Journal of Health Services Research & Policy, 1/1: 35-43.
11. Camarinha-Matos, L. M. and Afsarmanesh, H. (2007) 'Results Assessment and Impact Creation in Collaborative Research- An Example from the ECOLEAD Project', Technovation, 27/1-2: 65-77.
12. Clason, D. L. and Dormody, T. J. (1984) 'Analyzing Data Measured by Individual Likert-Type Items', Journal of Agricultural Education, 35/4: 31-5.
13. Council of Canadian Academies. Expert Panel on Science Performance and Research Funding. (2012) Informing Research Choices: Indicators and Judgment. Ottawa. Retrieved from accessed 14 Oct 2014.
14. Council of the European Union. (2009) Council Conclusions on the Evaluation and Impact Assessment of European Research Framework Programmes (FPs). Brussels, Retrieved from accessed 14 Oct 2014.
15. Czarnitzki, D. and Lopes-Bento, C. (2013) 'Value for Money? New Microeconometric Evidence on Public R&D Grants in Flanders', Research Policy, 42/1: 76-89.
16. DG Research and Innovation. (2011) Impact Assessment of Health Research Projects Supported by DG Research and Innovation 2002-2010. Retrieved from accessed 14 Oct 2014.
17. -. (2013) Review of Public Health Research Projects Financed under the Commission's Framework Programmes for Health Research: Subgroup1. Retrieved from accessed 14 Oct 2014.
18. European Commission. (2004) North South Partnership for Health Systems Research: 20 Years of Experience of European Commission Support. Brussels: European Commission Office.
19. -. (2014) DG Research and Innovation. FP7 Research Budgets. Retrieved from accessed 14 Oct 2014.
20. European Science Foundation. (2011) European Peer Review Guide: Integrating Policies and Practices into Coherent Procedures. Strasbourg, France. Retrieved from accessed 14 Oct 2014.
21. Fayl, G. et al. (1998) 'Evaluation of Research and Technological Development Programmes: A Tool for Policy Design', Research Evaluation, 7/2: 93-7.
22. Furman, E. et al. (2006) Experiences in the Management of Research Funding Programmes for Environmental Protection: Including Recommendations for Best Practice. (Finnish Environment, 43.) Helsinki: Finnish Environment Institute (SYKE). Retrieved from accessed 14 Oct 2014.
23. Galsworthy, M. J. et al. (2012) 'Academic Output of 9 Years of EU Investment into Health Research', The Lancet, 380/9846: 971-2.
24. Georghiou, L. and Roessner, D. (2000) 'Evaluating Technology Programs: Tools and Methods', Research Policy, 29/4-5: 657-78.
25. Grant, J. et al. (2010) Capturing Research Impacts: A Review of International Practice. Cambridge. Retrieved from accessed 14 Oct 2014.
26. -. (2000) 'Evaluating "Payback" on Biomedical Research from Papers Cited in Clinical Guidelines: Applied Bibliometric Study', BMJ, 320/7242: 1107-11.
27. Guthrie, S. et al. (2013) Measuring Research: A Guide to Research Evaluation Frameworks and Tools. Cambridge. Retrieved from accessed 14 Oct 2014.
28. Hall, M. and Clauss, D. (2012) ROM Handbook: Results-Oriented Monitoring. Retrieved from accessed 14 Oct 2014.
29. Hanney, S., Davies, A. and Buxton, M. (1999) 'Assessing Benefits from Health Research Projects: Can We Use Questionnaires Instead of Case Studies?', Research Evaluation, 8/3: 189-99.
30. Hanney, S. et al. (2004) 'Proposed Methods for Reviewing the Outcomes of Health Research: The Impact of Funding by the UK's Arthritis Research Campaign', Health Research Policy and Systems, 2/4. accessed 14 Oct 2014.
31. Health Economics Research Group of Brunel University, Office of Health Economics and RAND Europe. (2008) Medical Research: What's It Worth? Estimating the Economic Benefits from Medical Research in the UK. London, Retrieved from accessed 14 Oct 2014.
32. Horton, D. et al. (1993) Monitoring and Evaluating Agricultural Research: A Source Book. Retrieved from accessed 14 Oct 2014.
33. Horton, D., Galleno, V. and Mackay, R. (2007), Evaluation, Learning and Change in Research and Development Organizations: Concepts, Experiences, and Implications for the CGIAR (ILAC Working Paper 2, 2nd edn). Rome. Retrieved from accessed 14 Oct 2014.
34. Jagosh, J. et al. (2011) 'Assessing the Outcomes of Participatory Research: Protocol for Identifying, Selecting, Appraising and Synthesizing the Literature for Realist Review', Implementation Science, 6/24. accessed 14 Oct 2014.
35. Jamieson, S. (2004) 'Likert Scales: How to (Ab)use Them', Medical Education, 38/12: 1217-8.
36. Kalucy, L. et al. (2009) Exploring the Impact of Primary Health Care Research. Final Report Stage 2: Primary Health Care Research Impact Project. Retrieved from accessed 14 Oct 2014.
37. Krawiec, F. (1984) 'Evaluating and Selecting Research Projects by Scoring', Research Management, 27/2: 21-5.
38. Kwan, P. et al. (2007) 'A Systematic Evaluation of Payback of Publicly Funded Health and Health Services Research in Hong Kong', BMC Health Services Research, 7/121. accessed 14 Oct 2014.
39. Link, A. N. and Vonortas, N. S., eds (2013) Handbook on the Theory and Practice of Program Evaluation. Cheltenham: Edward Elgar Publishing.
40. Luukkonen, T. (1998) 'The Difficulties in Assessing the Impact of EU Framework Programmes', Research Policy, 27/6: 599-610.
41. Maredia, M., Byerlee, D. and Anderson, J. (2000) 'Ex Post Evaluation of Economic Impacts of Agricultural Research Programs: A Tour of Good Practice'. In: CGIAR. (ed.) The Future of Impact Assessment in CGIAR: Needs, Constraints, and Options. Standing Panel on Impact Assessment (SPIA) of the Technical Advisory Committee, Rome, 3-5 May, FAO, 2000. Retrieved from accessed 14 Oct 2014.
42. Millstone, E., Van Zwanenberg, P. and Marshall, F. (2010) 'Monitoring and Evaluating Agricultural Science and Technology Projects: Theories, Practices and Problems', IDS Bulletin, 41/6: 75-87.
43. Moore, J. R. and Baker, N. R. (1969) 'An Analytical Approach to Scoring Model Design - Application to Research and Development Project Selection', IEEE Transactions on Engineering Management, 16/3: 90-8.
44. Morgan, M. M. and Grant, J. (2013) 'Making the Grade: Methodologies for Assessing and Evidencing Research Impacts'. In: Dean, A., Wykes, M. and Stevens, H. (eds) 7 Essays on Impact. DESCRIBE Project Report for Jisc. University of Exeter, pp. 25-43. Exeter, UK: University of Exeter.
45. Morris, Z. S., Wooding, S. and Grant, J. (2011) 'The Answer Is 17 Years, What Is the Question: Understanding Time Lags in Translational Research', Journal of the Royal Society of Medicine, 104/12: 510-20.
46. OECD. (2008) OECD Science, Technology and Industry Outlook. Paris. Retrieved from accessed 14 Oct 2014.
47. Oortwijn, W. J. et al. (2008) 'Assessing the Impact of Health Technology Assessment in the Netherlands', International Journal of Technology Assessment in Health Care, 24/3: 259-69.
48. Panel on Return on Investment in Health Research. (2009) Making an Impact: A Preferred Framework and Indicators to Measure Returns on Investment in Health Research. Ottawa: Canadian Academy of Health Sciences. Retrieved from accessed 14 Oct 2014.
49. Penfield, T. et al. (2013) 'Assessment, Evaluations, and Definitions of Research Impact: A Review', Research Evaluation, 23/1: 21-32.
50. Piric, A. and Reeve, N. (1998) 'Evaluation of Public Investment in R&D: Towards a Contingency Analysis', In: OECD Conference on Policy Evaluation in Innovation and Technology: Towards Best Practices, 26-27 June 1997, pp. 49-64. Paris: Organization for Economic Co-operation and Development (OECD). Retrieved from accessed 14 Oct 2014.
51. Pollitt, A. et al. (2011), Project Retrosight: Understanding the Returns from Cardiovascular and Stroke Research: Methodology Report. Cambridge. Retrieved from accessed 14 Oct 2014.
52. Raftery, J. et al. (2009) 'Assessing the Impact of England's National Health Service R&D Health Technology Assessment Program using the "Payback" Approach', International Journal of Technology Assessment in Health Care, 25/1: 1-5.
53. Reeve, N. (2010) 'Case Study of the FP7 Interim Evaluation: Notions of Quality in Evaluation Design', American Evaluation Association (AEA) Evaluation 2010, 8-13 November, San Antonio, Texas.
54. Rengarajan, S. and Jagannathan, P. (1997) 'Project Selection by Scoring for a Large R&D Organisation in a Developing Country', R & D Management, 27/2: 155-64.
55. Research Fish Ltd. (2012) Researchfish. Retrieved from accessed 14 Oct 2014.
56. Røttingen, J. A. et al. (2013) 'Mapping of Available Health Research and Development Data: What's There, What's Missing, and What Role Is There for a Global Observatory?', The Lancet, 382/9900: 1286-307.
57. Ruegg, R. (1999) 'Advanced Technology Program's Approach to Technology Diffusion (NISTR 6385)'. In: NATO Science for Peace Programme Advanced Research Workshop "Industry as a Stimulator of Technology Transfer", Gaithersburg, MD. Retrieved from accessed 14 Oct 2014.
58. SCImago. (2007) SJR - SCImago Journal & Country Rank. Retrieved from accessed 14 Oct 2014.
59. Souder, W. E. (1972) 'A Scoring Methodology for Assessing the Suitability of Management Science Models', Management Science, 18/10: 526-43.
60. Tassey, G. (2003) Methods for Assessing the Economic Impacts of Government R&D: Planning Report 03-1. Retrieved from accessed 14 Oct 2014.
61. The European Court of Auditors. (2008) 'Special Report No. 9/ 2007. Evaluating the EU research and technological development (RTD) framework programmes - Could the Commission's Approach be Improved?', EU Official Journal of the European Union, 30 January, pp. 1-38. Retrieved from accessed 14 Oct 2014.
62. Thomson Reuters. (2014) Journal Citation Reports. Retrieved from accessed 14 Oct 2014.
63. Van Raan, A. F. J. (2000) 'R&D Evaluation at the Beginning of the New Century', Research Evaluation, 9/2: 81-6.
64. Vullings, W., Meijer, I. and Mostert, B. (2008) 'Strategic Research Planning: Increase the Impact of Public Research by Integrating User-Perspective in Planning and Evaluation'. In: PRIME Network's Europe-Latin America Conference on Science and innovation Policy, Mexico D.F., 24-26 September 2008.
65. W. K. Kellogg Foundation. (2004) Logic Model Development Guide. Battle Creek, Michigan.
66. Wang, K., Wang, C. K. and Hu, C. (2005) 'Analytic Hierarchy Process With Fuzzy Scoring in Evaluating Multidisciplinary R&D Projects in China', IEEE Transactions on Engineering Management, 52/1: 119-29.
67. Wooding, S. et al. (2005) 'Payback Arising from Research Funding: Evaluation of the Arthritis Research Campaign', Rheumatology, 44/9: 1145-56.
68. -. (2009) Mapping the Impact: Exploring the Payback of Arthritis Research. Cambridge. Retrieved from accessed 14 Oct 2014.
69. -. (2013) Mental Health Retrosight: Understanding the Returns from Research (Lessons from Schizophrenia): Policy Report. Cambridge. Retrieved from accessed 14 Oct 2014.