Automated Research Impact Assessment: a new bibliometrics approach

Scientometrics

Volumn 106, Issue 3, 2016, Pages 987-1005

  Drew, Christina H ^a   Pettibone, Kristianna G ^a   Finch, Fallis Owen ^b   Giles, Douglas ^b   Jordan, Paul ^c  

^a NATIONAL INSTITUTE OF ENVIRONMENTAL HEALTH SCIENCES   (United States)

^b Open Intelligence   (United States)

^c NATIONAL INSTITUTES OF HEALTH   (United States)

Author keywords

Automated impact analysis; Bibliometrics; Research evaluation; Science of science management

Indexed keywords

EID: 84958666927     PISSN: 01389130     EISSN: 15882861     Source Type: Journal    
DOI: 10.1007/s11192-015-1828-7     Document Type: Article

References (61)

1. Ajiferuke, I., & Wolfram, D. (2010). Citer analysis as a measure of research impact: Library and information science as a case study. Scientometrics,83, 623–638. doi:10.1007/s11192-009-0127-6.
2. Author Unknown. (1970). A trace of “traces”. Mosaic Magazine Science Articles Archive, 1(1), 14–19. Retrieved from http://www.mosaicsciencemagazine.org/index.php?mode=article&pk_magazine=109. Accessed 19 March 2015.
3. Bernstein, B. E., Stamatoyannopoulos, J. A., Costello, J. F., Ren, B., Milosavljevic, A., Meissner, A., et al. (2010). The NIH roadmap epigenomics mapping consortium. [opinion and comment]. Nature Biotechnology, 28(10), 1045–1048. doi:10.1038/nbt1010-1045. Retrieved from http://www.nature.com/nbt/journal/v28/n10/full/nbt1010-1045.html. Accessed 17 March 2015.
4. Borgman, C. L., & Furner, J. (2002). Scholarly communication and bibliometrics. Annual Review of Information Science and Technology,36, 3–72.
5. Boyack, K. W., & Jordan, P. (2011). Metrics associated with NIH funding: A high-level view. Journal of the American Medical Informatics Association,18(4), 423–431. doi:10.1136/amiajnl-2011-000213.
6. Campbell, D., Picard-Aitken, M., Cote, G., Caruso, J., Valentim, R., Edmonds, S., et al. (2010). Bibliometrics as a performance measurement tool for research evaluation: The case of research funded by the National Cancer Institute of Canada. American Journal of Evaluation,31(1), 66–83. doi:10.1177/1098214009354774.
7. Engel-Cox, J. A., Van Houten, B., Phelps, J., & Rose, S. W. (2008). Conceptual model of comprehensive research metrics for improved human health and environment. Environmental Health Perspectives, 116(5), 583–592. doi:10.1289/ehp.10925. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/18470312.
8. Guthrie, K., Louie, J., David, T., & Foster, C. C. (2005). The challenge of assessing policy and advocacy activities: Strategies for a prospective evaluation approach. Los Angeles: The California Endowment.
9. Haak, L., Ferriss, W., Wright, K., Pollard, M., Barden, K., Probus, M., et al. (2012). The electronic Scientific Portfolio Assistant: Integrating scientific knowledge databases to support program impact assessment. Science and Public Policy,39, 464–475. doi:10.1093/scipol/scs030.
10. Hanney, S., Home, P., Frame, I., Grant, J., Green, P., & Buxton, M. (2005). Indentifying the impact of diabetes research. Diabetic Medicine,23, 176–184. doi:10.1111/j.1464-5491.2005.01753.x.
11. Hicks, D., Tomizawa, H., Saitoh, Y., & Kobayashi, S. (2004). Bibliometric techniques in the evaluation of federally funded research in the United States. Research Evaluation, 13(2), 76–86. doi:10.3152/147154404781776446. Retrieved from http://rev.oxfordjournals.org/content/13/2/7. Accessed 19 March 2015.
12. Howell, E. M., & Yemane, A. (2006). An assessment of evaluation designs: Case studies of 12 large federal evaluations. American Journal of Evaluation, 27(2), 219–236. doi:10.1177/1098214006287557. Retrieved from http://aje.sagepub.com/content/27/2/219.abstract.
13. Hughes, D., Docto, L., Peters, J., Lamb, A. L., & Brindis, C. (2013). Swimming upstream: The challenges and rewards of evaluating efforts to address inequities and reduce health disparities. Evaluation and Program Planning, 38, 1–12. Retrieved from http://www.sciencedirect.com/science/article/pii/S0149718913000050.
14. IIT Research Institute. (1968). Technology in retrospect and critical events in science: Prepared for the national science foundation (Vol. Volume 1 & 2). Chicago: Illinois Institute of Technology Research Institute under Contract NSF-C535.
15. Impact-EV. (2014). IMPACT-EV. http://impact-ev.eu/. Accessed 19 Aug 2015.
16. Jones, T., Donovan, C., & Hanney, S. (2012). Tracing the wider impact of biomedical research: A literature search to develop a novel citation categorisation technique. Scientometrics,93, 125–134. doi:10.007/s11192-012-0642-8.
17. Jump, P. (2013). Australia prepares for (research) impact. Times Higher Education. June 22. Retrieved from http://www.timeshighereducation.co.uk/news/australia-prepares-for-research-impact/2005011.article.
18. Kan, M. (2010). ParsCit. https://www.comp.nus.edu.sg/entrepreneurship/ParsCit_kanmy.html. Accessed 19 Aug 2015.
19. Koplan, J. P., Milstein, R., & Wetterhall, S. (1999). Framework for program evaluation in public health. MMWR: Recommendations and Reports,48, 1–40.
20. Kostoff, R. N. (1995). Federal research impact assessment: Axioms, approaches, applications. Scientometrics,34(2), 163–206.
21. Kostoff, R. N. (1998). The use and misuse of citation analysis in research evaluation. Scientometrics,43(1), 27–43.
22. Lane, J. (2010). Let’s make science metrics more scientific. Nature,464(7288), 488–489. doi:10.1038/464488a.
23. Lane, J., & Bertuzzi, S. (2011). Measuring the results of science Investments. Science, 331(6018), 678–680. doi:10.1126/science.1201865. Retrieved from http://www.sciencemag.org/content/331/6018/678.short.
24. Lewison, G., Rippon, I., & Wooding, S. (2005). Tracking knowledge diffusion through citations. Research Evaluation,14(1), 5–14. doi:10.3152/147154405781776319.
25. Leyedesdorff, L. (1998). Theories of citation? Scientometrics,43(1), 5–25.
26. Liebow, E., Phelps, J., Van Houten, B., Rose, S., Orians, C., Cohen, J., et al. (2009). Toward the assessment of scientific and public health impacts of the National Institute of Environmental Health Sciences Extramural Asthma Research Program using available data. Environmental Health Perspectives, 117(7), 1147–1154. doi:10.1289/ehp.0800476. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/19654926.
27. Martin, B. R. (1996). The use of multiple indicators in the assessment of basic research. Scientometrics, 36(3), 343–362. doi:10.1007/BF02129599. Retrieved from http://sci2s.ugr.es/hindex/pdf/Martin1996.pdf.
28. Milat, A. J., Bauman, A. E., & Redman, S. (2015). A narrative review of research impact assessment models and methods. Health Research Policy and Systems, 13(18). doi:10.1186/s12961-015-0003-1. Retrieved from http://www.health-policy-systems.com/content/pdf/s12961-015-0003-1.pdf.
29. Moed, H. F. (Ed.). (2005). Citation analysis in research evaluation (information science and knowledge management). Netherlands Springer: Dordrecht.
30. Narin, F. (1976). Evaluative bibliometrics. Cherry Hill, NJ: Computer Horizons.
31. National Academy of Sciences (NAS). (2001). Implementing the government performance and results act for research: A status report. Washington, DC: National Academy of Sciences, National Academy of Engineering, Institute of Medicine.
32. National Academy of Sciences (NAS). (2004). NIH Extramural center programs: Criteria for initiation and evaluation. Washington, DC: National Academy of Science.
33. National Institute of Environmental Health Sciences (NIEHS). (2012). Partnerships for environmental public health evaluation metrics manual (NIH publication no. 12-7825). Durham, NC. Retrieved from http://www.niehs.nih.gov/research/supported/assets/docs/a_c/complete_peph_evaluation_metrics_manual_508.pdf. Accessed 13 March 2015.
34. National Institutes of Health (NIH). (2014a). NIH RePORTER. http://projectreporter.nih.gov/reporter.cfm. Accessed 11 March 2015.
35. National Institutes of Health (NIH). (2014b). NIH public access policy website. http://publicaccess.nih.gov/policy.htm. Accessed 20 March 2015.
36. National Institutes of Health (NIH). (2015a). iEdison. https://public.era.nih.gov/iedison. Accessed 19 Aug 2015.
37. National Institutes of Health (NIH). (2015b). NIH clinical trials database. https://clinicaltrials.gov/. Accessed 19 Aug 2015.
38. National Library of Medicine (NLM). (2015). Health services/technology assessment texts http://www.ncbi.nlm.nih.gov/books/NBK16710/. Accessed 19 Aug 2015.
39. National Research Council (NRC). (1999). Arsenic in drinking water. Washington, D.C.: National Research Council (NRC). doi:10.17226/6444.
40. National Research Council (NRC). (2001). Arsenic in drinking water: 2001 update. Washington, D.C.: National Research Council (NRC). doi:10.17226/10194.
41. National Research Council (NRC). (1998). Assessing the value of research in the chemical sciences: Report of a workshop. In 6. Patents and publicly funded research. Washington (DC): National Academics Press. doi:10.17226/6200.
42. Orians, C., Abed, J., Drew, C., Rose, S. W., Cohen, J., & Phelps, J. (2009). Scientific and public health impacts of the NIEHS Extramural Asthma Research Program—Insights from primary data. Research Evaluation, 18(5), 375–385. doi:10.3152/095820209X480698. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/21921976.
43. ParaTools. (2004). ParaTools. http://paracite.eprints.org/developers/. Accessed 19 Aug 2015.
44. Pasterkamp, G., Rotmas, J., de Kleijn, D., & Borst, C. (2007). Citation frequency: A biased measure of research impact significantly influenced by the geographical origin of research articles. Scientometrics,70(1), 153–165.
45. Phelan, T. J. (1999). A compendium of issues for citation analysis. Scientometrics,45(1), 117–136. doi:10.1007/BF02458472.
46. Quinlan, K. M., Kane, M., & Trochim, W. M. K. (2008). Evaluation of large research initiatives: Outcomes, challenges, and methodological considerations. New Directions for Evaluation,2008(118), 61–72. doi:10.1002/ev.261.
47. Research Excellence Framework. (2014). http://www.ref.ac.uk/. Accessed 19 Aug 2015.
48. Researchfish. (2014). https://www.researchfish.com/. Accessed 19 Aug 2015.
49. Reuters, T. (1994). The Thomson reuters impact factor. http://thomsonreuters.com/products_services/science/free/essays/impact_factor/. Accessed 19 March 2015.
50. Roessner, D., Carr, R., Feller, I., McGeary, M., & Newman, N. (1998). The role of NSF’s support of engineering in enabling technological innovation: Phase II, final report to National Science Foundation. Arlington, VA: SRI International.
51. Sayers, E. (2009). The E-utlities in-depth: Parameters, syntax and more. In Entrez programming utilities help. Bethesda, MD: National Center for Biotechnology Information, U.S. National Library of Medicine. Retrieved from ESearch: http://www.ncbi.nlm.nih.gov/books/NBK25499/#_chapter4_ESearch_. EFetch: http://www.ncbi.nlm.nih.gov/books/NBK25499/#_chapter4_EFetch_. ECitMatch: http://www.ncbi.nlm.nih.gov/books/NBK25499/#_chapter4_ECitMatch_.
52. Srivastava, C. V., Towery, N. D., & Zuckerman, B. (2007). Challenges and opportunities for research portfolio analysis, management, and evaluation. Research Evaluation, 16(3), 152–156. doi:10.3152/095820207x236385. Retrieved from http://rev.oxfordjournals.org/content/16/3/152.abstract.
53. Stuart, J. (2007). Necessity leads to innovative evaluation approach and practice. Evaluation Exchange,XII(1), 2.
54. Teles, S., & Schmitt, M. (2011). The elusive craft of evaluating advocacy. Stanford Social Innovation Review, 4. Retrieved from http://www.ssireview.org/images/digital_edition/2011SU_Feature_TelesSchmitt.pdf.
55. The Kellogg Foundation. (2004). Logic model development guide. Battle Creek, MI. Retrieved from http://www.wkkf.org/knowledge-center/resources/2006/02/wk-kellogg-foundation-logic-model-development-guide.aspx. Accessed 13 March 2015.
56. U.S. Environmental Protection Agency (EPA). (2009). Integrated science assessment for particulate Matter (EPA/600/R-08/139F). Research Triangle Park, NC. Retrieved from http://cfpub.epa.gov/ncea/cfm/recordisplay.cfm?deid=216546. Accessed 13 March 2015.
57. U.S. Environmental Protection Agency (EPA). (2010). Integrated science assessment for carbon monoxide (EPA/600/R-09/019F). Research Triangle Park, NC. Retrieved from http://cfpub.epa.gov/ncea/cfm/recordisplay.cfm?deid=218686. Accessed 13 March 2015.
58. U.S. Environmental Protection Agency (EPA). (2012). Integrated science assessment for lead (EPA/600/R-10/075B). Research Triangle Park, NC. Retrieved from http://cfpub.epa.gov/ncea/cfm/recordisplay.cfm?deid=255721. Accessed 13 March 2015.
59. U.S. Government Accountability Office. (2012). Designing evaluations (Publication No. GAO-12-208G). Washington, DC. Retrieved from http://www.gao.gov/products/GAO-12-208G. Accessed 11 March 2015.
60. Wooding, S., Hanney, S., Buxton, M., & Grant, J. (2005). Payback arising from research funding: Evaluation of the Arthritis Research Campaign. Rheumatology,44, 1145–1156. doi:10.1093/rheumatology/keh708.
61. World Health Organization (WHO). (2011). Arsenic in drinking water. Geneva: World Health Organization.