The use of multiple indicators in the assessment of basic research

Scientometrics

Volumn 36, Issue 3, 1996, Pages 343-362

  Martin, B R ^a  

^a UNIVERSITY OF SUSSEX   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 0000043671     PISSN: 01389130     EISSN: None     Source Type: Journal    
DOI: 10.1007/BF02129599     Document Type: Article

References (57)

1. E.g. J. H. WESTBROOK, Identifying significant research, Science, 132 (1960) 1229-1234; N. WADE, Citation analysis: a new tool for science administrators, Science, 188 (1975) 429-432.
2. E.g. J. H. WESTBROOK, Identifying significant research, Science, 132 (1960) 1229-1234; N. WADE, Citation analysis: a new tool for science administrators, Science, 188 (1975) 429-432.
3. J. IRVINE, B. R. MARTIN, What direction for basic scientific research?, Chapter 5 in M. GIBBONS, P. GUMMETT, B. M. UDGAONKAR (eds.), Science and Technology Policy in the 1980s and Beyond, London, Longman, 1984, pp. 67-98.
4. B. R. MARTIN, J. IRVINE, Assessing basic research: some partial indicators of scientific progress in radio astronomy, Research Policy, 12 (1983) 61-90.
5. E.g. ANON, Is your lab well cited?, Nature, 227 (1970) 219; ANON, More games with numbers, Nature, 228 (1970) 698-699.
6. E.g. ANON, Is your lab well cited?, Nature, 227 (1970) 219; ANON, More games with numbers, Nature, 228 (1970) 698-699.
7. E.g. D. LINDSEY, Production and citation measures in the sociology of science: the problem of multiple authorship, Social Studies of Science, 10 (1980) 145-162.
8. The sixth, CERN, was left to a subsequent study two years later. The results were published in a series of three articles: B. R. MARTIN, J. IRVINE, CERN: past performance and future prospects - I - CERN's position in World High-Energy Physics, Research Policy, 13 (1984) 183-210;
9. J. IRVINE, B. R. MARTIN, CERN: past performance and future prospects - II - The scientific performance of the CERN accelerators, Research Policy, 13 (1984) 247-284;
10. and B. R. MARTIN, J. IRVINE, CERN: Past performance and future prospects - III - CERN and the future of world high-energy physics. Research Policy, 13 (1984) 311-342.
11. J. IRVINE, B.R. MARTIN, A methodology for assessing the scientific performance of research groups, Scientia Yugoslavia, 6 (1980) 83-95.
12. MARTIN, IRVINE, op. cit., note 3. This article appeared in April 1983 even though it had been accepted for publication in September 1980.
13. J. IRVINE, B. R. MARTIN, P. A. ISARD, Investing in the Future: An International Comparison of Government Funding of Academic and Related Research, Aldershot and Brookfield, Vermont, Edward Elgar, 1990.
14. IRVINE, MARTIN, op. cit., note 2.
15. H. F. HANSEN, B. H. JØRGENSEN, Science Policy & Research Management: Can Research Indicators Be Used?, Institute of Political Science, University of Copenhagen, Copenhagen, 1995, p. 1.
16. R. N. KOSTOFF, The Handbook of Research Impact Assessment (Fifth Edition), DTIC Report Number ADA296021, 1995.
17. For example, the evaluation of government-funded applied research in Norway employed a combination of peer review and 'customer review' - see J. IRVINE, B. R. MARTIN, M. SCHWARZ, K. PAVITT, R. ROTHWELL, Government Support for Industrial Research in Norway: A SPRU Report, Oslo: Universitetsforlaget Norwegian Official Publication NOU 30B, 1981.
18. See Fig. 1 on p.64 in MARTIN, IRVINE, op. cit., note 3.
19. A good example here would be popular books by scientists such as Stephen Hawking.
20. Ibid., note 3, p. 64.
21. KOSTOFF, op. cit., note 12, p. 8.
22. MARTIN, IRVINE, op. cit., note 3, p. 75.
23. R. MILLER, The influence of primary task on R&D laboratory evaluation: a comparative bibliometric analysis, R&D Management, 22 (1992) 3-20.
24. For an example of how the educational technological outputs from basic research may be assessed, see the references cited in note 21.
25. J. IRVINE, B. R. MARTIN, The Economic Effects of Big Science: The Case of Radio Astronomy, Proceedings of the International Colloquium on the Economic Effects of Space and Other Advanced Technologies, Strasbourg, 28-30 April 1980, Paris, European Space Agency, ESA SP-151, 1980; and B. R. MARTIN, J. IRVINE, Spin-Off from Basic Science: The Case of Radio Astronomy, Physics in Technology, 12 (1981) 204-212.
26. J. IRVINE, B. R. MARTIN, The Economic Effects of Big Science: The Case of Radio Astronomy, Proceedings of the International Colloquium on the Economic Effects of Space and Other Advanced Technologies, Strasbourg, 28-30 April 1980, Paris, European Space Agency, ESA SP-151, 1980; and B. R. MARTIN, J. IRVINE, Spin-Off from Basic Science: The Case of Radio Astronomy, Physics in Technology, 12 (1981) 204-212.
27. Over 150 scientists were interviewed in the 'Big Science Project'.
28. This section draws heavily on MARTIN, IRVINE, op. cit., note 3.
29. See, for example, the discussion in ibid., p. 67.
30. T. LUUKKONEN, The cognitive and social foundation of citation studies - why we still lack a theory of citation, submitted to Science, Technology and Human Values (1995).
31. W. R. SHADISH, D. TOLLIVER, M. GRAY, S. K. SEN GUPTA, Author judgements about works they cite: three studies from psychology journals, Social Studies of Science, 25 (1995) 477-498 - quote on p.481.
32. See also the related distinction between 'quality' and 'relevance' in HANSEN, JØRGENSEN, op. cit., note 11 ,p. 3.
33. MARTIN, IRVINE, op. cit., note 3, p.70.
34. Ibid.
35. Ibid.
36. Ibid.
37. Examples of this in the field of experimental high-energy physics can be found in MARTIN, IRVINE, op. cit., note 6.
38. T. S. KUHN, The Structure of Scientific Revolutions, Chicago, University of Chicago Press, 1970.
39. MARTIN, IRVINE, op. cit., note 3 ; idem., op. cit., note 6.
40. MARTIN, IRVINE, op. cit., note 3 ; idem., op. cit., note 6.
41. MARTIN, IRVINE, op. cit., note 3.
42. J. IRVINE, B. R. MARTIN, Assessing basic research: The case of the Isaac Newton Telescope, Social Studies of Science, 13 (1983) 49-86.
43. B. R. MARTIN, J. IRVINE, Internal criteria for scientific choice: an evaluation of the research performance of electron high-energy physics accelerators. Minerva, XIX (1981) 408-432.
44. MARTIN, IRVINE, op. cit., note 6.
45. L. M. BAIRD, C. OPPENHEIM, Do citations matter?, Journal of Information Science, 20 (1994) 2-15 (quote on p. 13).
46. KOSTOFF, op. cit., note 12, p. 37.
47. Ibid., p. 118.
48. A. H. RUBENSTEIN, E. GEISLER, Evaluating the outputs and impacts of R&D/innovation, International Journal of Technology Management, 6 (1991).
49. Not all scientometric analysts are guilty of this. For example, the ISI analysts who periodically publish lists of leading research institutes in Science Watch normally use three indicators - papers, citations and citations per paper.
50. Full details of the study and the results can be found in B. R. MARTIN, J. E. F. SKEA, Academic Research Performance Indicators: An Assessment of the Possibilities, Brighton, SPRU, 1992.
51. See Table 12 in ibid.
52. KOSTOFF, op. cit., note 12, p. 8.
53. HANSEN, JØRGENSEN, op. cit., note 11, p.5.
54. MARTIN, SKEA, op. cit., note 44, p. 75.
55. Ibid., p. 75.
56. J. P. DE GREVE, A. FRIJDAL, Evaluation of scientific research profile analysis - a mixed method, Higher Education Management, 1 (1989) 83-90.
57. KOSTOFF, op. cit., note 12, p. 9.