Accountability, performance assessment, and evaluation: Policy pressures and responses from research councils

Research Evaluation

Volumn 22, Issue 2, 2013, Pages 105-117

  Sá, Creso M ^a   Kretz, Andrew ^a   Sigurdson, Kristjan ^a  

^a UNIVERSITY OF TORONTO   (Canada)

Author keywords

accountability; performance assessment; research councils; research evaluation; research funding agencies; science policy

Indexed keywords

EID: 84875128906     PISSN: 09582029     EISSN: 14715449     Source Type: Journal    
DOI: 10.1093/reseval/rvs041     Document Type: Article

References (74)

1. Behn, R. D. (2001) Rethinking Democratic Accountability. Washington, D.C.: Brookings Institute Press.
2. Bird, S. M. et al. (2005) 'Performance Indicators: Good, Bad, and Ugly', Journal of the Royal Statistical Society A, 168/1: 1-27. (Pubitemid 40166958)
3. Budd, R. W., Thorp, R. K. and Donohew, L. (1967) Content Analysis of Communications. New York, NY: The Maximillian Company.
4. Canadian Institutes of Health Research (CIHR). (2011a) Departmental Performance Report 2010-11 http://www.tbssct. gc.ca/dpr-rmr/2010-2011/inst/cri/ cri00-eng.asp accessed 29 Aug 2012.
5. -. (2011b) Moving Forward: Canadian Institutes of Health Research Annual Report 2010-11 accessed 29 Aug 2012.
6. -. (2011c) Report on Plans and Priorities 2010-11 accessed 29 Aug 2012.
7. -. (2011d) Show Me the Evidence http://www.cihr-irsc.gc.ca/e/documents/ show-me-evidence-v1-e.pdf accessed 25 Sept 2012.
8. Christensen, T. and Lægreid, P., eds (2007) Transcending New Public Management: The Transformation of Public Sector Reforms. Aldershot, England: Ashgate.
9. Cirpriani, G. (2010) The EU Budget: Responsibility without Accountability? Brussels, Belgium: Centre for European Policy Studies.
10. Cozzens, S. E. (2007) 'Death by Peer Review? The Impact of Results-Oriented Management in US Research'. In: Whitley, R. and Glä ser, J. (eds) The Changing Governance of the Sciences, pp. 225-42. Dordrecht: Springer Science.
11. Cozzens, S. E. and Turpin, T. (2000) 'Processes and Mechanisms for Evaluating and Monitoring Research Outcomes from Higher Education: International Comparisons', Research Evaluation, 8/1: 3-4.
12. Department of Finance Canada. (2006) Advantage Canada: Building a Strong Economy for Canadians. Ottawa, ON: Finance Canada.
13. Dill, D. and van Vught, F. (2010) National Innovation and the Academic Research Enterprise. Baltimore, MD: John Hopkins University Press.
14. Donovan, C. (2011) 'State of the Art in Assessing Research Impact: Introduction to a Special Issue', Research Evaluation, 20/3: 175-79.
15. Economic and Social Research Council. (2007) Report of the ESRC Impact Evaluation Methods Workshop 20th March 2007 http://www.esrc.ac.uk/-images/ESRC- Impact-Evaluation-Methods-Workshop-tcm8-3814.pdf accessed 20 Aug 2012.
16. Economic and Social Research Council (ESRC). (2009) Taking Stock: A Summary of ESRC's Work to Evaluate the Impact of Research on Policy and Practice http://ebookbrowse.com/gdoc.php?id=51187813&url=f11e2edc4e340114719ca40 be4e694ca accessed 20 Aug 2012.
17. -. (2011) Branching Out: New Directions in Impact Evaluation from the ESRC's Evaluation Committee http://www.esrc.ac.uk/-images/Branching%20Out-tcm8- 14881.pdf accessed 25 Sept 2012.
18. European Commission. (2007) FP7 in Brief: How to Get Involved in the EU 7th Framework Programme for Research http://ec.europa.eu/research/fp7/pdf/fp7- inbrief-en.pdf accessed 1 Nov 2012.
19. -. (2009) Towards a World Class Frontier Research Organization: Review of the European Research Council's Structures Mechanisms http://ec.europa.eu/ research/evalu ations/pdf/archive/other-reports-studies-and-documents/erc- review-panel-report-23-july-2009.pdf accessed 25 Sept 2012.
20. -. (2010) Communication from the Commission to the European Parliament, the Council, the European Economic and Social Committee and the Committee of the Regions: Simplifying the Implementation of the Research Framework Programmes http://ec.europa.eu/research/fp7/pdf/communication-on-simplification-2010-en. pdf accessed 1 Nov 2012.
21. Federal Accountability Act. (2006) Retrieved from the Department of Justice Canada Website http://laws-lois. justice.gc.ca/PDF/F-5.5.pdf accessed 25 Sept 2012.
22. Garbarino, S. and Holland, J. (2009) Quantitative and Qualitative Methods in Impact Evaluation and Measuring Results (Governance and Social Development Resource Centre) accessed 25 Sept 2012.
23. Garrett-Jones, S. and Aylward, D. (2000) 'Some Recent Developments in the Evaluation of University Research Outcomes in the United Kingdom', Research Evaluation, 8/1: 69-75.
24. Georghiou, M. and Larédo, P. (2006) OECD Science, Technology and Industry Outlook 2006.
25. Paris: Organisation for Economic Co-operation and Development. Government Performance and Results Act. (1993) Retrieved from U.S. Office of Management and Budget Website http://www.whitehouse.gov/omb/mgmt-gpra/gplaw2m accessed 26 Sept 2012.
26. Government Performance and Results Modernization Act. (2010) Retrieved from the U.S. Office of Management and Budget Website http://www.whitehouse.gov/ omb/perform ance/gprm-act accessed 26 Sept 2012.
27. Grant, J. et al. (2009) Capturing Research Impacts: A Review of International Practice. Report prepared for the Higher Education Funding Council for England. Cambridge: RAND Europe.
28. Harlow, C. (2002) Accountability in the European Union. Oxford, England: Oxford University Press.
29. Himanen, L. et al. (2009) 'Influence of Research Funding and Science Policy on University Research Performance: A Comparison of Five Countries', Science and Public Policy, 36/6: 419-30.
30. HM Treasury. (2010) Spending Review http://cdn.hm-treasury. gov.uk/sr2010-completereport.pdf accessed 20 Aug 2012.
31. HM Treasury, Department of Trade, & Industry and Department of Education and Skills. (2004) Science & Innovation Investment Framework 2004-2014. London: HM Treasury.
32. Holbrook, A. (2000) 'Evaluation of Research Sponsored by Federal Granting Councils in Canada: The Social Contract', Research Evaluation, 8/1: 47-56.
33. Industry Canada. (2007) Mobilizing Science and Technology to Canada's Advantage. Ottawa, ON: Industry Canada.
34. Interagency Task Group. (2008) The Science of Science Policy: A Federal Research Roadmap. Washington, DC: National Science and Technology Council and the Office of Science and Technology Policy.
35. Krippendorf, K. (1980) Content Analysis: An Introduction to Its Methodology. Thousand Oakes, CA: Sage Publications, Inc.
36. Lane, J.-E. (2000) New Public Management. New York: Routledge.
37. -2005) Public Administration and Public Management: The Principal-Agent Perspective. London Routledge.
38. Marburger, J. H. III and Bolten J. B. 2005. Memorandum for the Heads of Executive Departments and Agencies: FY 2007 Administration Research and Development Budget Priorities'.
39. Martin, B. R. (2011) 'The Research Excellence Framework and the "Impact Agenda": Are We Creating a Frankenstein Monster?', Research Evaluation, 20/3: 247-54.
40. Medical Research Council. (2012a) About MRC e-Val (Now Researchfish) (pubd online 2012) http://www.mrc.ac.uk/Achievementsimpact/Outputsoutcomes/e- Val/About/index.htm accessed 25 Sept 2012.
41. -. (2012b) Annual Review (pubd online 2012) http://www.mrc.ac.uk/ Newspublications/Publications/AnnualReview/index.htm accessed 25 Sept 2012.
42. -. (2012c) Delivery Plan Reporting Framework 2012/13 http://www.mrc.ac. uk/consumption/idcplg?IdcService=GET-FILE&dID=36187&dDocName= MRC008769&allow Interrupt=1 accessed 27 Aug 2012.
43. -. (2012d) MRC Economic Impact Report (pubd online 2012) accessed 27 Aug 2012.
44. Mervis, J. (2011) 'Suresh Expects Low-Cost Ideas Will Mean a Big Payoff for NSF', Science, 334: 1492-3.
45. Moed, H. F., Glänzel, W. and Schmoch, U., eds (2005) Handbook of Quantitative Science and Technology Research. New York: Kluwer Academic Publishers.
46. Morris, S. and Townsend, S. (2008) 'Assessment of S&T Outcomes in Canada-The Current State in the Federal Government', Powerpoint slides from The 11th Annual Meeting of the Heads of Research Councils Working Group, Workshop on the Science of Science Policy http://www.nsf. gov/about/performance/ SciencePolicyWrkshp-Presentations/Morris&Townsend.pdf accessed 12 Aug 2012.
47. National Institutes of Health. (2012) NIH Performance Reporting (GPRA) (pubd online February 15, 2012, http://dpcpsi.nih.gov/opep/performance- reporting.aspx accessed 25 Sept 2012.
48. National Science Foundation (2011) FY 2012 Budget Request to Congress http://www.nsf.gov/about/budget/fy2012/pdf/fy 2012-rollup.pdf accessed 25 Sept 2012.
49. Natural Sciences and Engineering Research Council. (2007) Report of the International Review Committee on the Discovery Grants Program accessed 25 Sept 2012.
50. Norrie, J. (2012) Push to Quantify Social Impact of Science Goes Global (The Conversation) https://theconversation.edu.au/push-to-quantify-social- impact-of-science-goes-global-7068 accessed 18 May 2012.
51. Nutley, S. M., Walter, I. and Davies, H. T. O. (2007) Using Evidence: How Research Can Inform Public Services. Bristol, UK: The Policy Press.
52. Olsen, S. and Merrill, S. (2011) 'Measuring the Impacts of Federal Investments in Research: A Workshop Summary', National Academies (US) Committee on Measuring Economic and Other Returns on Federal Research Investments. Washington, DC: National Academies Press.
53. Orians, C. E. et al. (2009) 'Scientific and Public Health Impacts of the NIEHS Extramural Asthma Research Program: Insights from Primary Data', Research Evaluation, 18/5: 375-85.
54. Orszag, P. R. and Holdren, J. P. (2009) 'Memorandum for the Heads of Executive Departments and Agencies: Science and Technology Priorities for the FY 2011 Budget'.
55. Pidd, M. (2012) Measuring the Performance of Public Services: Principles and Practice. Cambridge: Cambridge University Press.
56. Power, M. (1997) The Audit Society: Rituals of Verification. Oxford: Oxford University Press.
57. Prime Minister's Office. (2006) Federal Accountability Act Becomes Law http://www.pm.gc.ca/eng/media.asp?category=1&id=1455 accessed 20 Aug 2012.
58. Radin, B. A. (1998) 'The Government Performance and Results Act (GPRA): Hydra-Headed Monster or Flexible Management Tool?', Public Administration Review, 58/4: 307-16.
59. Research Council Economic Impact Group. (2006) Increasing the Economic impact of the Research Councils ("The Warry Report"), http://www.vitae.ac.uk/cms/files/DTI-Warry-Report-July-2006.pdf accessed 29 Aug 2012.
60. Research Councils UK (RCUK). (2005) Research Councils Publish Delivery Plans accessed 12 Aug 2012.
61. Sá , C., Tamtik, M. and Kretz, A. (2013) 'Evidence and Expertise in Policy Making for University Research'. In: Axelrod, Paul et al. (eds) Policy Formation in Postsecondary Education: Issues and Prospects in Turbulent Times. Montreal: McGill-Queen's University Press.
62. Schiavo-Campo, S. and McFerson, H. M. (2008) Public Management in Global Perspective. Armonk, NY: Sharpe.
63. Stensaker, B. and Harvey, L., eds (2011) Accountability in Higher Education: Global Perspectives on Trust and Power. London: Routledge.
64. Treasury Board of Canada. (2008a) Policy on Transfer Payments accessed 5 Aug 2012.
65. -. (2008b) Report on Plans and Priorities (pubd online December 8 2008) http://www.tbs-sct.gc.ca/rpp/index-eng.asp accessed 25 Sept 2012.
66. -. (2009) Policy on Evaluation accessed 25 Sept 2012.
67. -. (2010) About the Centre of Excellence for Evaluation (pubd online February 25 2010) http://www.tbs-sct.gc.ca/cee/index-eng.asp accessed 25 Sept 2012.
68. -. (2011) Departmental Performance Reports (pubd online November 2011) http://www.tbs-sct.gc.ca/dpr-rmr/indexeng.asp accessed 25 Sept 2012.
69. -. (2012) Policy on Internal Audit http://www.tbs-sct.gc.ca/pol/doc-eng. aspx?id=16484§ion=text accessed 25 Sept 2012.
70. Wagner, R. M. and Jordan, G. B. (2009) 'Moving Towards Impact When Evaluating Research Programs: Introduction to a Special Section', Research Evaluation, 18/5: 339-42.
71. Weber, R. (1996) Basic Content Analysis, 6th edn. Newbury Park, CA: Sage Publications.
72. Weiss, C. H. and Bucuvalas, M. (1980) Social Science Research and Decision-Making. New York, NY: Columbia University Press.
73. Whitley, R. and Gläser, J., eds (2007) The Changing Governance of the Sciences. Dordrecht: Springer.
74. Whitley, R. (2011) 'Changing Governance and Authority Relations in the Public Sciences', Minerva, 49: 359-85.