Can the technological impact of academic journals be evaluated? The practice of non-patent reference (NPR) analysis

Scientometrics

Volumn 101, Issue 1, 2014, Pages 17-37

  Liaw, Yi Ching ^a   Chan, Te Yi ^b   Fan, Chin Yuan ^b   Chiang, Cheng Hsin ^c  

^a MING CHI UNIVERSITY OF TECHNOLOGY   (Taiwan)

^b NATIONAL APPLIED RESEARCH LABORATORIES   (Taiwan)

^c MACAU UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Macao)

Author keywords

Impact factor; Journal ranking; Non patent reference

Indexed keywords

EID: 84939872969     PISSN: 01389130     EISSN: 15882861     Source Type: Journal    
DOI: 10.1007/s11192-014-1337-0     Document Type: Article

References (65)

1. Abramo, G., Andrea D’Angelo, C., & Di Costa, F. (2009). Mapping excellence in national research systems: The case of Italy. Evaluation Review,33(2), 159–188. doi:10.1177/0193841X08322871.
2. Agrawal, V. K. (2002). Constituencies of journals in production and operations management: Implications on reach and quality. Production and Operations Management,11(2), 101–108. doi:10.1111/j.1937-5956.2002.tb00485.x.
3. Althouse, B. M., West, J. D., Bergstrom, C. T., & Bergstrom, T. (2009). Differences in impact factor across fields and over time. Journal of the American Society for Information Science and Technology,60(1), 27–34. doi:10.1002/asi.20936.
4. Baum, J. A. C. (2011). Free-riding on power laws: Questioning the validity of the Impact Factor as a measure of research quality in organization studies. Organization,18(4), 449–466. doi:10.1177/1350508411403531.
5. Bornmann, L., & Daniel, H.-D. (2008). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation,64(1), 45–80. doi:10.1108/00220410810844150.
6. Brusoni, S., Criscuolo, P., & Geuna, A. (2005). The knowledge bases of the world’s largest pharmaceutical groups: What do patent citations to non-patent literature reveal? Economics of Innovation and New Technology,14(5), 395–415. doi:10.1080/1043859042000307356.
7. Buela-Casal, G. (2004). Assessing the quality of articles and scientific journals: Proposal for weighted impact factor and a quality index. Psychology in Spain,8(1), 60–76.
8. Callaert, J., Grouwels, J., & Looy, B. (2012). Delineating the scientific footprint in technology: Identifying scientific publications within non-patent references. Scientometrics,91(2), 383–398. doi:10.1007/s11192-011-0573-9.
9. Callaert, J., Van Looy, B., Verbeek, A., Debackere, K., & Thijs, B. (2006). Traces of prior art: An analysis of non-patent references found in patent documents. Scientometrics,69(1), 3–20. doi:10.1007/s11192-006-0135-8.
10. Callaham, M., Wears, R. L., & Weber, M. E. (2002). Journal prestige, publication bias, and other characteristics associated with citation of published studies in peer-reviewed journals. The Journal of the American Medical Association,287(21), 1733–1818. doi:10.1001/jama.287.21.2847.
11. Campanario, J. M. (2011). Large increases and decreases in journal impact factors in only 1 year: The effect of journal self-citations. Journal of the American Society for Information Science and Technology,62(2), 230–235. doi:10.1002/asi.21457.
12. Catling, J. C., Mason, V. L., & Upton, D. (2009). Quality is in the eye of the beholder? An evaluation of impact factors and perception of journal prestige in the UK. Scientometrics,81(2), 333–345. doi:10.1007/s11192-009-2124-1.
13. Chang, C. M., & Hung, S. Y. (2004). Distinct journal preference of successful EC researchers: A citation analysis. Electronic Commerce Studies,2(1), 1–18.
14. Chapron, G., & Husté, A. (2006). Open, fair, and free journal ranking for researchers. BioScience,56(7), 558–559. doi:10.1641/0006-3568(2006)56[558:OFAFJR]2.0.CO;2.
15. Christensen, H. F., Ingwersen, P., & Wormell, I. (1997). Online determination of the journal impact factor and its international properties. Scientometrics,40(3), 529–540. doi:10.1007/BF02459298.
16. Claro, J., & Costa, C. A. V. (2010). A made-to-measure indicator for cross-disciplinary bibliometric ranking of researchers performance. Scientometrics,86(1), 113–123. doi:10.1007/s11192-010-0241-5.
17. Cohen, B. (2007). Journal ratings and footprints: a North American perspective of organizations and the natural environment journal quality. Business Strategy and Environment,16(1), 64–74. doi:10.1002/bse.461.
18. Coleman, A. (2007). Assessing the value of a journal beyond the impact factor. Journal of the American Society for Information Science and Technology,58(8), 1148–1161. doi:10.1002/asi.20599.
19. Crewe, I., & Norris, P. (1991). British and American journal evaluation: Divergence or convergence? Political Science and Politics,24(3), 524–531.
20. Crews, J. M., McLeod, A., & Simkin, M. G. (2009). Journal self-citation XII: The ethics of forced journal citations. Communications of AIS,25(12), 97–110.
21. Epstein, W. M. (2004). Confirmational response bias and the quality of the editorial processes among American social work journals. Research on Social Work Practice,14(6), 450–458. doi:10.1177/1049731504265838.
22. Extejt, M. M., & Smith, J. E. (1990). The behavioral sciences and management: An evaluation of relevant journals. Journal of Management,15(3), 539–551. doi:10.1177/014920639001600302.
23. Gao, X., & Guan, J. (2009). Networks of scientific journals: An exploration of Chinese patent data. Scientometrics,80(1), 283–302. doi:10.1007/s11192-007-2013-4.
24. García, J. A., Rodriguez-Sánchez, R., & Fdez-Valdivia, J. (2011). Overall prestige of journals with ranking score above a given threshold. Scientometrics,89(1), 229–243. doi:10.1007/s11192-011-0442-6.
25. Geary, J., Marriott, L., & Rowlinson, M. (2004). Journal rankings in business and management and the 2001 research assessment exercise in the UK. British Journal of Management,15(2), 95–141. doi:10.1111/j.1467-8551.2004.00410.x.
26. Gordon, M. D. (1982). Citation ranking versus subjective evaluation in the determination of journal hierachies in the social sciences. Journal of the American Society for Information Science,33(1), 55–57. doi:10.1002/asi.4630330109.
27. Haddow, G., & Genoni, P. (2010). Citation analysis and peer ranking of Australian social science journals. Scientometrics,85(2), 471–487. doi:10.1007/s11192-010-0198-4.
28. Haynes, J. P. (1983). An empirical method for determining core psychology journals. American Psychologist,38(8), 959–961. doi:10.1037/0003-066X.38.8.959.
29. Hicks, D., & Wang, J. (2013). The New York times as a resource for mode 2. Science, Technology and Human Values,38(6), 851–877. doi:10.1177/0162243913497806.
30. Huang, M.-H., Huang, W.-T., & Chen, D.-Z. (2014). Technological impact factor: An indicator to measure the impact of academic publications on practical innovation. Journal of Informetrics,8(1), 241–251. doi:10.1016/j.joi.2013.12.004.
31. Jones, M. J., Brinn, T., & Pendlebury, M. (1996). Journal evaluation methodologies: a Balanced response. Omega,24(5), 607–612. doi:10.1016/0305-0483(96)00038-2.
32. Kao, C., Lin, H.-W., Chung, S.-L., Tsai, W.-C., Chiou, J.-S., Chen, Y.-L., et al. (2008). Ranking Taiwanese management journals: A case study. Scientometrics,76(1), 95–115. doi:10.1007/s11192-007-1895-5.
33. Kothari, T., & Lahiri, S. (2012). Yesterday, today and tomorrow: An overview of research publications in the Journal of International Management. Journal of International Management,18(1), 102–110. doi:10.1016/j.intman.2011.09.001.
34. Leydesdorff, L., & Bornmann, L. (2011). Integrated impact indicators compared with impact factors: An alternative research design with policy implications. Journal of the American Society for Information Science and Technology,62(11), 2133–2146. doi:10.1002/asi.21609.
35. Lo, S.-C. S. (2009). Scientific linkage of science research and technology development: A case of genetic engineering research. Scientometrics,82(1), 109–120. doi:10.1007/s11192-009-0036-8.
36. Marx, W., Bornmann, L., & Cardona, M. (2010). Reference standards and reference multipliers for the comparison of the citation impact of papers published in different time periods. Journal of the American Society for Information Science and Technology,61(10), 2061–2069. doi:10.1002/asi.21377.
37. McCain, K. W. (1991). Core journal networks and cocitation maps: New bibliometric tools for serials research and management. The Library Quarterly,61(3), 311–336.
38. McMillan, G. S., Narin, F., & Deeds, D. L. (2000). An analysis of the critical role of public science in innovation: The case of biotechnology. Research Policy,29(1), 1–8. doi:10.1016/S0048-7333(99)00030-X.
39. Meyer, M. (2002). Tracing knowledge flows in innovation systems. Scientometrics,54(2), 193–212. doi:10.1023/A:1016057727209.
40. Michel, J., & Bettels, B. (2001). Patent citation analysis: A closer look at the basic input data from patent search reports. Scientometrics,51(1), 185–201.
41. Moed, H., Colledge, L., Reedijk, J., Moya-Anegon, F., Guerrero-Bote, V., Plume, A., et al. (2012). Citation-based metrics are appropriate tools in journal assessment provided that they are accurate and used in an informed way. Scientometrics,92(2), 367–376. doi:10.1007/s11192-012-0679-8.
42. Narin, F., & Olivastro, D. (1992). Status report: Linkage between technology and science. Research Policy,21(3), 237–249. doi:10.1016/0048-7333(92)90018-Y.
43. Narin, F., Hamilton, K. S., & Olivastro, D. (1997). The increasing linkage between U. S. technology and public science. Research Policy,26(3), 317–330. doi:10.1016/S0048-7333(97)00013-9.
44. Narin, F., & Noma, E. (1985). Is technology becoming science? Scientometrics,7(3–6), 369–381. doi:10.1007/BF02017155.
45. Palacios-Huerta, I., & Volij, O. (2004). The measurement of intellectual influence. Econometrica,72(3), 963–977. doi:10.1111/j.1468-0262.2004.00519.x.
46. Pichappan, P. (1995). A dual refinement of journal self-citation measures. Scientometrics,33(1), 13–21. doi:10.1007/BF02020772.
47. Podsakoff, P. M., Mackenzie, S. B., Bachrach, D. G., & Podsakoff, N. P. (2005). The influence of management journals in the 1980s and 1990s. Strategic Management Journal,26(5), 473–488. doi:10.1002/Smj.454.
48. Pontille, D., & Torny, D. (2010). The controversial policies of journal ratings: Evaluating social sciences and humanities. Research Evaluation,19(5), 347–360. doi:10.3152/095820210x12809191250889.
49. Robinson, L. M., & Adler, R. (1981). Measuring the impact of marketing scholars and institutions: An analysis of citation frequency. Journal of the Academy of Marketing Science,9(1), 147–162. doi:10.1007/BF02723573.
50. Rousseau, R. (2002). Journal evaluation: Technical and practical issues. Library Trends,50(3), 418–439.
51. Schoenmakers, W., & Duysters, G. (2010). The technological origins of radical inventions. Research Policy,39(8), 1051–1059. doi:10.1016/j.respol.2010.05.013.
52. Seglen, P. O. (1997). Why the impact factor of journals should not be used for evaluating research. British Medical Journal,314, 497–513.
53. Shearer, B. A., Lundeberg, M. A., & Coballes-Vega, C. (1997). Making the connection between research and reality: Strategies Teachers use to read and evaluate journal articles. Journal of Educational Psychology,89(4), 592–598. doi:10.1037/h0092690.
54. Singleton, A. (1976). Journal ranking and selection: A review in physics. Journal of Documentation,32(4), 258–289. doi:10.1108/eb026628.
55. Sombatsompop, N., & Markpin, T. (2005). Making an equality of ISI impact factors for different subject fields. Journal of the American Society for Information Science and Technology,56(7), 676–683. doi:10.1002/asi.20150.
56. Sutter, M., & Kocher, M. G. (2001). Tools for evaluating research output: Are citation-based rankings of economics journals stable? Evaluation Review,25(5), 555–566. doi:10.1177/0193841x0102500503.
57. Tahai, A., & Meyer, M. J. (1999). A revealed preference study of management journals’ direct influences. Strategic Management Journal,20(3), 279–296.
58. Tijssen, R. J. W., Visser, M. S., & Van Leeuwen, T. N. (2002). Benchmarking international scientific excellence: are highly cited research papers an appropriate frame of reference? Scientometrics,54(3), 381–397. doi:10.1023/A:1016082432660.
59. Todorov, R., & Glanzel, W. (1988). Journal citation measures: A concise review. Journal of Information Science,14(1), 47–56. doi:10.1177/016555158801400106.
60. Turban, E., Zhou, D., & Ma, J. (2004). A group decision support approach to evaluating journals. Information & Management,42(1), 31–44. doi:10.1016/j.im.2003.12.003.
61. Tussen, R. J., Buter, R., & Van Leeuwen, T. N. (2000). Technological relevance of science: An assessment of citation linkages between patents and research papers. Scientometrics,47(2), 389–412. doi:10.1023/A:1005603513439.
62. Ugaz, A. G. (2011). Drilling deeper into the core: an analysis of journal evaluation methodologies used to create the “Basic List of Veterinary Medical Serials,” third edition. Journal of Medical Library Association,99(2), 145–152. doi:10.3163/1536-5050.99.2.006.
63. Vastag, G., & Montabon, F. (2002). Journal characteristics, rankings and social acculturation in operations management. Omega,30(2), 109–126. doi:10.1016/S0305-0483(01)00061-5.
64. Verbeek, A., Debackere, K., Luwel, M., Andries, P., Zimmermann, E., & Deleus, F. (2002). Linking science to technology: Using bibliographic references in patents to build linkage schemes. Scientometrics,54(3), 399–420. doi:10.1023/A:1016034516731.
65. Vinkler, P. (2002). Subfield problems in applying the Garfield (Impact) Factors in practice. Scientometrics,53(2), 267–279. doi:10.1023/A:1014860726532.