Models and mechanisms for evaluating government-funded research: An international comparison

American Journal of Evaluation

Volumn 28, Issue 4, 2007, Pages 437-457

  Coryn, Chris L S ^a   Hattie, John A ^b   Scriven, Michael ^c   Hartmann, David J ^a  

^a WESTERN MICHIGAN UNIVERSITY   (United States)

^b UNIVERSITY OF AUCKLAND   (New Zealand)

^c CLAREMONT GRADUATE UNIVERSITY   (United States)

Author keywords

International comparisons; Research evaluation; Research funding; Research governance; Science policy

Indexed keywords

EID: 36549021614     PISSN: 10982140     EISSN: 15570878     Source Type: Journal    
DOI: 10.1177/1098214007308290     Document Type: Article

References (75)

1. ab Iorwerth, A. (2005). Methods of evaluating university research around the world. Ottawa, Canada: Department of Finance.
2. Bartle, D. (2006, November). Increasing the relevance and application of evaluation to government policy: Insights from New Zealand evaluation experience in R&D and economic development policies. Paper presented at the meeting of the American Evaluation Association, Portland, OR.
3. Brennan, R.L. (1983). Elements of generalizability theory. Iowa City, IA: American College Testing Program.
4. Campbell, D.F.J. (2002). Conceptual framework for the evaluation of university research in Europe (Working Paper). Washington, DC: George Washington University, Center for International Science and Technology Policy.
5. Campbell, D.F.J., & Felderer, B. (1997). Evaluating academic research in Germany: Patterns and policies (Political Science Series, no. 48). Wein, Austria: Institut für Höhere Studien.
6. Conraths, B., & Smidt, H. (2005). The funding of university-based research and innovation in Europe: An exploratory study. Brussels, Belgium: European University Association.
7. Cook, T.D., & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago: Rand-McNally.
8. Cooksy, L.J., & Caracelli, V.J. (2005). Quality, context, and use: Issues in achieving the goals of metaevaluation. American Journal of Evaluation, 26 (1), 31 - 42.
9. Coryn, C.L.S. (2007a). Evaluation of researchers and their research: Toward making the implicit explicit. Unpublished doctoral dissertation, Western Michigan University, Kalamazoo.
10. Coryn, C.L.S. (2007b). The fundamental characteristics and quality of international models and mechanisms for evaluating government-funded research. Access: Critical Perspectives on Communication, Cultural & Policy Studies.
11. Coryn, C.L.S. (2007c). The "holy trinity" of methodological rigor: A skeptical view. Journal of MultiDisciplinary Evaluation, 4(7), 26-31.
12. Coryn, C.L.S. (2007d, January). I think...; therefore I need funding. Paper presented at The Evaluation Center's Evaluation Café, Kalamazoo, MI.
13. Coryn, C.L.S. (2007e, April). Models and mechanisms for evaluating publicly-funded research: An international comparison with an emphasis on New Zealand's PBRF. Paper presented to the Tertiary Education Commission, Wellington, New Zealand.
14. Coryn, C.L.S., & Scriven, M. (2007). Are national-level research evaluation models valid, credible, useful, cost-effective, and ethical? Journal of MultiDisciplinary Evaluation, 4(8), 26-30.
15. Coryn, C. L. S., & Scriven, M. (Eds.). (in press). Reforming the evaluation of research. New Directions for Evaluation. San Francisco: Jossey-Bass.
16. Committee on Science, Engineering, and Public Policy. (1999). Evaluating federal research programs: Research and the Government Performance and Results Act. Washington, DC: National Academy Press.
17. Committee on Science, Engineering, and Public Policy. (2001). Implementing the Government Performance and Results Act for research: A status report. Washington, DC: National Academy Press.
18. Cozzens, S.E. (2000). Assessing federally-supported academic research in the United States. Research Evaluation, 8(1), 5-10.
19. Cozzens, S.E., & Turpin, T. (2000). Processes and mechanisms for evaluating and monitoring research outcomes from higher education: International comparisons. Research Evaluation, 8(1), 3-4.
20. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart & Winston.
21. Curtis, B., & Matthewman, S. (2005). The managed university: The PBRF, its impacts and staff attitudes. New Zealand Journal of Employment Relations, 30(2), 1-17.
22. Davidson, E.J. (2005). Evaluation methodology basics: The nuts and bolts of sound evaluation. Thousand Oaks, CA: Sage.
23. Davis, J.H. (1996). Group decision making and quantitative judgments: A consensus model. In E. Witte & J. H. Davis (Eds.), Understanding group behavior: Consensual action by small groups (pp. 35-59). Hillsdale, NJ: Lawrence Erlbaum.
24. Department of Education, Science and Training. (2005). The research quality framework. Retrieved February 4, 2006, from http://www.dest.gov.au/sectors/research_sector/policies_issues_reviews/ key_issues/research_quality_framework/.
25. Dill, D.D. (2003, March). The regulation of academic quality: An assessment of university evaluation systems with emphasis on the United States. Paper presented at the international symposium University Evaluation for the Future: International Trends in Higher Education Reform, Tokyo, Japan.
26. Donovan, C. (in press). The Australian research quality framework: A live experiment in capturing the social, economic, environmental, and cultural returns of publicly-funded research. In C. L. S. Coryn & M. Scriven (Eds.), New directions for evaluation: Reforming the evaluation of research. San Francisco: Jossey-Bass.
27. European Commission. (2007). ERAWATCH. Retrieved March 29, 2007, from http://cordis.europa.eu/erawatch/.
28. Fichtner, D. (1990). Competition in the university system of the Federal Republic of Germany. Scientometrics, 19 (5/6), 331-335.
29. Frederiksen, L.F., Hannson, F., & Wennberg, S.B. (2003). The Agora and the role of research evaluation. Evaluation: The International Journal of Theory, Research and Practice, 9(2), 149-172.
30. Geuna, A., & Martin, B.R. (2003). University research evaluation and funding: An international comparison. Minerva, 41, 277-304.
31. Giorgi, L., & Tandon, A. (2000). The theory and practice of evaluation (ICCR Working Papers, No. 407). Vienna, Austria: Interdisciplinary Center for Comparative Research in the Social Sciences.
32. Greene, J.C. (1992). A case study of evaluation auditing as metaevaluation. Evaluation and Program Planning, 15, 71-74.
33. Government Performance and Results Act of 1993, 31 U.S.C. §1101, 31 U.S.C. §1115 (1993).
34. Guba, E.G., & Lincoln, Y.S. (1989). Fourth generation evaluation. San Francisco: Jossey-Bass.
35. Hackett, E.J. (1997). Peer review in science and science policy. In M. S. Frankel & J. Cave (Eds.), Evaluating science and scientists: An east-west dialogue on research evaluation in post-Communist Europe (pp. 51-60). Budapest, Hungary: Central European University Press.
36. Hansson, F. (2006). Organizational use of evaluation: Governance and control in research evaluation. Evaluation: The International Journal of Theory, Research and Practice, 12(2), 159-178.
37. Henry, G.T. (2001). How modern democracies are shaping evaluation and the emerging challenges for evaluation. American Journal of Evaluation, 22(3), 419-429.
38. Higher Education and Research Opportunities. (1999a). RAE circular 2/99: Guidance on submissions. Retrieved January 21, 2006, from http://www.hero.ac.uk/rae/niss/2_99.html.
39. Higher Education and Research Opportunities. (1999 b). RAE circular 5/ 99: Assessment panel's criteria and working methods. Retrieved January 21, 2006, from http://www.hero.ac.uk/rae/Pubs/5_99/section1.htm.
40. House, E.R., & Howe, K.R. (2000 a). Deliberative democratic evaluation. New Directions for Evaluation, 85. San Francisco: Jossey-Bass.
41. House, E.R., & Howe, K.R. (2000 b). Deliberative democratic evaluation checklist. Retrieved October 27, 2006, from http://www.wmich.edu/evalctr/checklists/dd_checklist.PDF.
42. Jeannin, P., & Devillard, J. (2005). Implementing relevant disciplinary evaluations in the social sciences: National vs international interactions in scientific communities. Scientometrics, 63(1), 121-144.
43. Julnes, G., & Rog, D. L. (Eds.). (2007). Informing federal policies on evaluation methodology: Building the evidence base for method choice in government sponsored evaluation. New Directions for Evaluation, 113, San Francisco: Jossey-Bass.
44. Kenny, D.A., Kashy, D.A., & Cook, W.L. (2006). Dyadic data analysis. New York: Guilford.
45. Kirkhart, K.E. (2004, June). Culture, evaluation, and metaevaluation: A question of validity. Plenary address presented at the Howard University Training Institute, Washington, DC.
46. Kuhlmann, S. (2003). Evaluation of research and innovation policies: A discussion of trends with examples from Germany. International Journal of Technology Management, 26 (2), 131-149.
47. Lal, B. (2006, November). Are we there yet? A review of the "social science of science policy ". Paper presented at the meeting of the American Evaluation Association, Portland, OR.
48. Langfeldt, L. (2002). Decision-making in expert panels evaluating research: Constraints, processes and bias. Doctoral dissertation, University of Oslo, Norway.
49. Laudel, G. (2006). The "quality myth": Promoting and hindering conditions for acquiring research funds. Higher Education, 52, 375-403.
50. Lewis, D.M., Green, D.R., Mitzel, H.C., Baum, K., & Patz, R.J. (1998, April). The bookmark standard setting procedure: Methodology and recent implications. Paper presented at the meeting of the National Council on Measurement in Education, Seattle, WA.
51. Linstone, H. A., & Turoff, M. (Eds.). (2002). The Delphi method: Applications and techniques. Newark, NJ: Authors.
52. Motohashi, K. (2003). Recent developments in research and innovation policy in Japan. Unpublished manuscript.
53. Mrinalini, N., & Nath, P. (2006). Comparative evaluation of practices: Lessons from R&D organizations. Benchmarking: An International Journal, 13 (1/2), 214-223.
54. Nature. (2006). Editorial: Evaluate this-The objective evaluation of research isn't working as it should. Nature, 440, 1-2.
55. Organization for Economic Cooperation and Development. (1987). Evaluation of research: A selection of current practices. Paris: Author.
56. Organization for Economic Cooperation and Development. (1997). The evaluation of scientific research: Selected experiences. Paris: Author.
57. Organization for Economic Cooperation and Development. (2003). Governance of public research: Country case studies. Paris: Author.
58. Orr, D., & Paetzold, M. (2006). Procedures for research evaluation in German higher education: Current fragmentation and future prospects. Access: Critical Perspectives on Communication, Cultural & Policy Studies, 25 (2), 16-30.
59. Patton, M.Q. (1997). Utilization-focused evaluation: The new century text (3 rd ed.). Thousand Oaks, CA: Sage.
60. Przeworski, A. (1987). Methods of cross-national research, 1970-1983: An overview. In M. Dierkes, H. N. Weiler, & A. B. Antal (Eds.), Comparative policy research: Learning from experience (pp. 31-49). Aldershot, England: Gower.
61. Schröter, D.C., Coryn, C.L.S., & Montrosse, B. (2007). Peer review of abstracts submitted to the Graduate Student and New Evaluators Topical Interest Group for the 2006 American Evaluation Association conference. American Journal of Evaluation. Manuscript submitted for publication.
62. Schwartz, R., & Mayne, J. (2004). Quality matters: Seeking confidence in evaluation, auditing, and performance reporting. New Brunswick, NJ: Transaction Publishing.
63. Scriven, M. (1969). An introduction to meta-evaluation. Educational Products Report, 2 (5), 36-38.
64. Scriven, M. (1991). Evaluation thesaurus (4 th ed.). Thousand Oaks, CA: Sage.
65. Scriven, M. (2006). The evaluation of research merit versus evaluation of funding of research. Journal of MultiDisciplinary Evaluation, 3 (5), 120-123.
66. Scriven, M. (2007). Predictive evaluation. Journal of MultiDisciplinary Evaluation, 4 (7), 71-73.
67. Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.
68. Shewan, L.G., & Coats, A.J.S. (2006). The research quality framework and its implications for health and medical research: Time to take stock? Medical Journal of Australia, 184 (9), 463-466.
69. Stufflebeam, D.L. (2001). The metaevaluation imperative. American Journal of Evaluation, 22 (2), 183-209.
70. U.S. Office of Management and Budget. (2003). Budget procedures memorandum No. 861, completing the Program Assessment Rating Tool (PART) for the FY 2005 review process. Washington, DC: Author.
71. U.S. Office of Management and Budget. (2006). Program Assessment Rating Tool (PART). Washington, DC: U.S. Author.
72. Vartiainen, P.L. (2002). On the principles of comparative evaluation. Evaluation, 8 (3), 359-371.
73. von Tunzelmann, N., & Mbula, E.K. (2003). Changes in research assessment practices in other countries since 1999: Final report. Brighton, UK: University of Sussex, SPRU Science and Technology Policy Research.
74. Weiss, C.H. (1998). Evaluation (2 nd ed.). Upper Saddle River, NJ: Prentice Hall.
75. Yin, R.K. (1994). Case study research: Design and methods: Vol. 5 Applied social research methods series (2nd ed.). Thousand Oaks, CA: Sage.