Science indicators as reliable evidence

Minerva

Volumn 45, Issue 1, 2007, Pages 17-30

  Feller, Irwin ^a   Gamota, George ^b  

^a AAAS   (United States)

^b STMA LCC   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 33947195993     PISSN: 00264695     EISSN: None     Source Type: Journal    
DOI: 10.1007/s11024-006-9017-8     Document Type: Article

References (52)

1. Zvi Griliches, 'R&D and Productivity: Econometric Results and Measurement Issues', in Paul Stoneman (ed.), Handbook of the Economics of Innovation and Technological Change (Oxford: Blackwell, 1995), 52-89, esp. 53.
2. Arie Rip, 'Qualitative Conditions of Scientomctrics: The New Challenge', Scientometrica, 38 (1), (1997), 7-26.
3. Metrics generally relate to a system of measurement; indicators, to a broad range of quantitative measures of selected activities. See Eliezer Geisler, The Metrics of Science and Technology (Westport, CT: Quorum Books, 2000), ch. 3.
4. The terms are used interchangeably in this paper. This paper is primarily concerned with American agencies, such as the National Institutes of Health (NIH), the National Science Foundation (NSF), and the Department of Energy's Office of Science (SC). The paper excludes targeted development programmes, such as are represented in the USA by the Department of Commerce, the National Institute of Standards and Technology's Advanced Technology Program, and its Manufacturing Extension Program. The use of performance measures in these programmes is discussed by Phillip Shapira, 'Evaluating Manufacturing Extension Services in the United States: Experiences and Insights', in Phillip Shapira and Stefan Kuhlmann (eds.), Learning from Science and Technology Policy Evaluations (Cheltenham: Edward Elgar, 2003), 260-292;
5. and also by Rosalie Ruegg and Irwin Feller, A Toolkit for Evaluating Public R&D Investment (Washington, DC: National Institutes of Standards and Technology, 2003), NIST GCR-03-857.
6. This cautionary note does not extend to the use of indicators to describe the workings of a specific system, or as the source or byproduct of data used retrospectively to assess the performance of such a system.
7. National Research Council, Revealing the Hidden Nature of Space and Time (Washington, DC: National Academies, 2006);
8. Anthony van Raan, 'The Use of Bibliometric Analysis in Research Performance Assessment and Monitoring of Interdisciplinary Scientific Developments', Technikfolgenabschulzung, 12 (1), (2003), 20-29.
9. 'Space Science: A Space Race to the Bottom Line', Science, 31 (17 March 2006), 1540.
10. More detailed accounts of the differences between GPRA and PART are contained in the proceedings of the Washington Research Evaluation Network (WREN)'s 2003 Workshop. See Rosalie Ruegg, 'Planning for Performance and Evaluating Results of Public R&D Programs: Meeting the OMB PART Challenge', Workshop Executive Summary Report (Washington, DC: 2004), 15-17.
11. Studies of the problematical impact of GPRA and PART on science policy are found in US General Accounting Office, Performance Budgeting: Observations on the Use of OMB's Program Assessment Rating Tool (Washington, DC: General Accounting Office, 2004), GAO-04-174;
12. Beryl Radin, The Government Performance Results Act and the Tradition of Federal Management Reform: Square Pegs in Round Holes?' Journal of Public Administration Research and Theory, 10 (1), 111-135;
13. and Irwin Feller, 'The Good, the Indifferent, and the Less than Attractive: II', Research Evaluation, 11 (2), (2002), 95-99.
14. John Marburger, Keynote Presentation to the 30th Annual AAAS Forum on Science and Technology Policy, Washington, DC, 21 April 2005.
15. Executive Office of the President, Office of Management and Budget, Memorandum for the Heads of Executive Departments and Agencies, FY2005 Interagency Research and Development Priorities, M-03-15, 5 June 2003, 5.
16. Susan Cozzens, 'Research Assessment: What's Next? Final Report on a Workshop', Research Evaluation, 11 (2), (2002), 65-79.
17. William Gormley and David Weiner, Organizational Report Cards (Cambridge, MA: Harvard University Press, 2001).
18. David Guston, Between Politics and Science (Cambridge: Cambridge University Press, 2000).
19. Francis Narin, Kimberly Hamilton, and Dominic Olivastro, 'The Increasing Linkage between US Technology and Public Science', Research Policy, 26 (3), (1997), 317-330.
20. Ed Noyons, 'Science Maps within a Science Policy Context', in Henk F. Moed, Wolfgang Glanzel, and Ulrich Schmoch (eds.), Handbook of Quantitative Science and Technology Research (Dordrecht: Kluwer Academic, 2004), 237-255.
21. E. Rinia, Theo van Leeuwen, H.G. van Vuren, and Anthony F.J. van Raan, 'Comparative Analysis of a Set of Bibliometric Indicators and Central Peer Review Criteria-Evaluation of Condensed Matter Physics in the Netherlands', Research Policy, 27 (1), (1998), 95-107.
22. These trends have not gone unchallenged. According to Michael Dummett, who has characterized bibliometrics as a monitoring system in which performance is measured by the number of words published per year, 'Wittgenstein, who died in 1951 having published only one short article after the Tractatus of 1922, would plainly not have survived such a system'.
23. Michael Dummet, Frege: Philosophy of Mathematics (London: Duckworth, 1991), ix.
24. See also Theo van Leeuwen, 'Descriptive versus Evaluative Bibliometrics', in Moed, Glanzel, and Schmoch, (eds.) op. cit. note 14, 373(388;
25. and Peter Weingart, Impact of Bibliometrics upon the Science System: Inadvertent Consequences', Scientometrics, 62 (1), (2005), 117-131.
26. Irwin Feller, George Gamota, and William Valdez, 'Developing Science Indicators for Basic Science Offices within Mission Agencies', Research Evaluation, 12 (1), (2003), 71-79.
27. The presence of four objectives within the 1982 legislation that established the US Small Business Innovation Research (SBIR) programme in 1982 illustrates the problem. The SBIR programme requires each US Federal agency with an extramural research budget in excess of SUS 100 million to set aside 2.5% of its research and development budget for awards to small firms (defined as those with fewer than 500 employees). Among the programme's objectives are (1) the stimulation of technological innovation; (2) the use of small businesses to meet Federal research and development needs; and (3) the increasing private sector commercialization of innovations derived from Federal research and development. See National Research Council, SBIR: Program Diversity and Assessment Challenges, Report of Symposium [Washington, DC: National Academies Press, 2004), 5-12.
28. The emphasis on commercialization has increased, leading in turn to increased emphasis on metrics associated with sales. As a consequence, the programme has a 'schizophrenic nature': In many cases, There may be a conflict between the goals of procurement and innovation', See James Turner, 'Panel I: History and Current Legislative Perspective on the SBIR Program' in National Research Council, The Small Business Innovation Research Program: Challenges and Opportunities, edited by Charles W. Wessner (Washington, DC: National Academy Press, 1992), 49.
29. Anton J. Nederhof, The Validity and Reliability of Evaluation of Scholarly Performance', in Anthony van Raan (ed.), Handbook of Quantitative Studies of Science and Technology (Amsterdam: North-Holland, 1988), 193-228.
30. Oskar Morgenstein, On the Accuracy of Economic Observations (Princeton: Princeton University Press, 1963).
31. Eugene Garfield, Morton Malin, and Henry Small, 'Citation Data as Science Indicators', in Yehuda Elkana (ed.), Toward a Metric of Science (New York: John Wiley and Sons, 1978), 179-207, esp. 182.
32. Adam Jaffe and Manuel Trajtenberg, Patents, Citations and Innovations (Cambridge, MA: MIT Press, 2002), 3;
33. Griliches, op. cit. note 1, 54.
34. Cited as an example of 'success leading to omission' is Lee Cronback and Paul Meehl, 'Construct Validity in Psychological Tests', Psychological Bulletin, 52 (1955), 281-302, which was described as having 'exerted an enormous, almost incalculable influence', but is now so widely accepted that most researchers never bother to cite the authors.
35. Richard Hebert, 'Highly Cited, Highly Controversial', Observer, 17 (3), (March 2004), 1ff.
36. Bibliometric evidence supports the proposition that important papers do produce high citation counts. For an example from astronomy, see Helmut Abt, 'Do Important Papers Produce High Citation Counts', Scientomeirics, 48 (1), (2000), 65-70.
37. US Department of Energy - Office of Science, Strategic Plan (Washington, DC: Government Printing Office, 2004.
38. Along related lines, in an action designed to insure that NSF's support is directed at fundamental, frontier-level research, the National Science Board has set up a Task Force to 'identify, evaluate, and fund⋯research that has the potential to revolutionize an existing discipline through a paradigm shift or create a new one', (National Science Board, Committee on Programs and Plans, Charge to the Task Force on Transformative Research, December 2004). Likewise, a National Research Council committee has written of the excitement surrounding elementary particle physics research and of its being 'poised to take the next significant step in answering questions that humans have asked for millennia'. NRC, op. cit. note 5.
39. 'The Puzzling Portrait of a Pore', Science, 204 (27 June 2003), 2020.
40. For a similar and more encompassing observation, see Peter Bowler and Iwan Morus, Making Modern Science (Chicago: University of Chicago Press, 2005). Critiquing Popper's emphasis on falsification, they suggest (p. 9) that 'good' scientists do not always actively seek to expose the weaknesses of current hypotheses as quickly as possible: 'On the contrary, they get very attached to a successful theory, especially if they have built their careers on it, and are often reluctant to consider, if not actively hostile to, any suggestion that it should be replaced'.
41. Stephen Cole, Making Science (Cambridge, MA: Harvard University Press, 1992), 1-32, 102-136.
42. Arjo Klamer and David Colander, The Making of an Economist (Boulder, CO: Westview Press, 1990);
43. David Marsh, Knowledge and the Wealth of Nations (New York: W.W. Norton & Company, 2006).
44. Daniel Kelves, 'Pursuing the Unpopular: A History of Viruses, Courage and Cancer', in Robert Silvers (ed.), Hidden Histories of Science (New York: A New York Review Book, 1995), 69-112.
45. As I.B. Cohen put it, 'the rise of Polywater shows how men and women actually behave in the laboratory, under the stress of today's extremely competitive science system: their behavior does not always conform to the ideal search for abstract truth that has long been the classic image. The desire to be an active part of revolutionary movement is often in conflict with the natural reluctance of any scientist to jettison the set of accepted ideas on which he has made his way in the profession'. I.B. Cohen, Revolution in Science (Cambridge, MA: Harvard University Press, 1985).
46. Alexander Kohn, False Prophets (Oxford: Basil Blackwood, 1986).
47. National Research Council, Facilitating Interdisciplinary Research (Washington, DC: National Academies Press, 2005);
48. Irwin Feller, 'Multiple Actors, Multiple Settings: Multiple Interpretations of Quality as a Criterion for Assessing Interdisciplinary Research', Research Evaluation, 15 (1), 5-15.
49. John Marburger III, 'Wanted: Better Benchmarks', Science, 308 (20 May 2005), 1087.
50. See, for example, Daniel Greenburg's account of NSF's conversion in the early 1980s of its extramural science policy research program administered by its Division of Policy Research and Analysis into an internal staff unit and the subsequent discrediting of the unit, leading to what has now been an approximately 20-year hiatus in a coherent set of inquiries along the lines of Marburger's call, Daniel Greenburg, Science, Money, and Politics (Chicago: University of Chicago Press, 2001), 113-128.
51. Robert Losee, The Development and Migration of Concepts from Donor to Borrower Disciplines: Sublanguage Term Use in Hard & Soft Sciences', Proceedings of the Fifth International Conference on Scientometrics and Informetrics (Medford, NJ: Learned Information Inc., 1995), 267-274.
52. Robert Harris, Pompeii (New York: Random House, 2003), 272.