Measuring academic influence: Not all citations are equal

Journal of the Association for Information Science and Technology

Volumn 66, Issue 2, 2015, Pages 408-427

  Zhu, Xiaodan ^a   Turney, Peter ^a   Lemire, Daniel ^b   Vellino, André ^c  

^a NATIONAL RESEARCH COUNCIL CANADA   (Canada)

^b Tel e University of Quebec   (Canada)

^c UNIVERSITY OF OTTAWA   (Canada)

Author keywords

natural language processing

Indexed keywords

ARTIFICIAL INTELLIGENCE; INDEXING (OF INFORMATION); LEARNING ALGORITHMS; LEARNING SYSTEMS; NATURAL LANGUAGE PROCESSING SYSTEMS; SUPERVISED LEARNING;

AUTOMATIC FEATURE SELECTION; DATA SET; H INDICES; NATURAL LANGUAGE PROCESSING; SUPERVISED MACHINE LEARNING;

EDUCATION;

EID: 84937764015     PISSN: 23301635     EISSN: 23301643     Source Type: Journal    
DOI: 10.1002/asi.23179     Document Type: Article

References (79)

1. Abu-Jbara, A., &, Radev, D., (2011). Coherent citation-based summarization of scientific papers. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, HLT '11 (pp. 500-509). Stroudsburg, PA: Association for Computational Linguistics.
2. Adler, R., Ewing, J., &, Taylor, P., (2009). Citation statistics. Statistical Science, 24 (1), 1-14.
3. Agarwal, S., Choubey, L., &, Yu, H., (2010). Automatically classifying the role of citations in biomedical articles. In AMIA Annual Symposium Proceedings, Vol. 2010 (p. 11). American Medical Informatics Association.
4. Ajiferuke, I., Lu, K., &, Wolfram, D., (2010). A comparison of citer and citation-based measure outcomes for multiple disciplines. Journal of the American Society for Information Science and Technology, 61 (10), 2086-2096.
5. Athar, A., &, Teufel, S., (2012). Detection of implicit citations for sentiment detection. In Proceedings of the Workshop on Detecting Structure in Scholarly Discourse, ACL '12 (pp. 18-261). Stroudsburg, PA: Association for Computational Linguistics.
6. Balaban, A.T., (2012). Positive and negative aspects of citation indices and journal impact factors. Scientometrics, 92 (2), 241-247.
7. Bergstrom, C., (2007). Measuring the value and prestige of scholarly journals. College & Research Library News, 68 (5), 314-316.
8. Bollen, J., Van de Sompel, H., Hagberg, A., &, Chute, R., (2009). A principal component analysis of 39 scientific impact measures. PLoS ONE, 4 (6), e6022.
9. Bornmann, L., &, Daniel, H.-D., (2008a). Functional use of frequently and infrequently cited articles in citing publications. European Science Editing, 34 (2), 35-38.
10. Bornmann, L., &, Daniel, H.-D., (2008b). What do citation counts measure? A review of studies on citing behavior. Journal of Documentation, 64 (1), 45-80.
11. Bornmann, L., Mutz, R., &, Daniel, H.-D., (2008). Are there better indices for evaluation purposes than the h index? A comparison of nine different variants of the h index using data from biomedicine. Journal of the American Society for Information Science and Technology, 59 (5), 830-837.
12. Buckley, C., &, Voorhees, E.M., (2000). Evaluating evaluation measure stability. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, SIGIR '00 (pp. 33-40). New York, NY: ACM.
13. Chang, C.-C., &, Lin, C.-J., (2011). LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2 (3), 27:1-27:27.
14. Chubin, D.E., &, Moitra, S.D., (1975). Content analysis of references: Adjunct or alternative to citation counting? Social Studies of Science, 5 (4), 423-441.
15. Councill, I.G., Giles, C.L., &, Kan, M.-Y., (2008). Parscit: An open-source crf reference string parsing package. In Proceedings of the Language Resources and Evaluation Conference. European Language Resources Association.
16. Darwin, C., (1859). On the origin of species by means of natural selection. London: Murray.
17. Dietz, L., Bickel, S., &, Scheffer, T., (2007). Unsupervised prediction of citation influences. In Proceedings of the 24th International Conference on Machine Learning (pp. 233-240). ACM.
18. Ding, Y., (2011). Applying weighted pagerank to author citation networks. Journal of the American Society for Information Science and Technology, 62 (2), 236-245.
19. Ding, Y., &, Cronin, B., (2011). Popular and/or prestigious? measures of scholarly esteem. Information Processing & Management, 47 (1), 80-96.
20. Dubin, D., (2004). The most influential paper Gerard Salton never wrote. Library Trends, 52 (4), 748-764.
21. Egghe, L., (2006). Theory and practise of the g-index. Scientometrics, 69 (1), 131-152.
22. Garfield, E., (1965). Can citation indexing be automated? In Statistical association methods for mechanized documentation, Symposium proceedings, ed. M.E. Stevens et al. (Washington: National Bureau of Standards, Miscellaneous Publication 269, 1965), pp. 188-192.
23. Garfield, E., (2006). The history and meaning of the journal impact factor. JAMA: The Journal of the American Medical Association, 295 (1), 90-93.
24. Garfield, E., &, Malin, M.V., (1968). Can nobel prize winners be predicted. In 135th Meetings of the American Association for the Advancement of Science, Dallas, TX.
25. Garzone, M., &, Mercer, R., (2000). Towards an automated citation classifier. In, H. Hamilton, (Ed.), Advances in artificial intelligence, vol. 1822 of lecture notes in computer science (pp. 337-346). Berlin Heidelberg: Springer.
26. Gilbert, G.N., (1977). Referencing as persuasion. Social Studies of Science, 7 (1), 113-122.
27. Giles, C.L., Bollacker, K.D., &, Lawrence, S., (1998). CiteSeer: An automatic citation indexing system. In Proceedings of the third ACM conference on Digital libraries, DL '98 (pp. 89-98). New York, NY: ACM.
28. Gingras, Y., &, Wallace, M.L., (2010). Why it has become more difficult to predict nobel prize winners: A bibliometric analysis of nominees and winners of the chemistry and physics prizes (1901-2007). Scientometrics, 82 (2), 401-412.
29. Gonzalez-Pereira, B., Guerrero-Bote, V.P., &, Moya-Anegõn, F., (2010). A new approach to the metric of journals scientific prestige: The sjr indicator. Journal of Informetrics, 4 (3), 379-391.
30. Hanney, S., Frame, I., Grant, J., Buxton, M., Young, T., &, Lewison, G., (2005). Using categorisations of citations when assessing the outcomes from health research. Scientometrics, 65 (3), 357-379.
31. Haque, A., &, Ginsparg, P., (2011). Phrases as subtopical concepts in scholarly text. In Proceedings of the 11th Annual International ACM/IEEE Joint Conference on Digital Libraries, JCDL '11 (pp. 25-28). New York, NY: ACM.
32. Hirsch, J.E., (2005). An index to quantify an individual's scientific research output. Proceedings of the National Academy of Sciences of the United States of America, 102 (46), 16569.
33. Hirsch, J.E., (2007). Does the h index have predictive power? Proceedings of the National Academy of Sciences, 104 (49), 19193-19198.
34. Hou, W.-R., Li, M., &, Niu, D.-K., (2011). Counting citations in texts rather than reference lists to improve the accuracy of assessing scientific contribution. Bioessays: News and Reviews in Molecular, Cellular and Developmental Biology, 33 (10), 724-727.
35. Ioannidis, J.P., (2006). Concentration of the most-cited papers in the scientific literature: Analysis of journal ecosystems. PLoS ONE, 1 (1), e5.
36. Kaplan, D., Iida, R., &, Tokunaga, T., (2009). Automatic extraction of citation contexts for research paper summarization: A coreference-chain based approach. In Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly Digital Libraries, NLPIR4DL '09 (pp. 88-95). Stroudsburg, PA: Association for Computational Linguistics.
37. Le, M.-H., Ho, T.-B., &, Nakamori, Y., (2006). Detecting citation types using finite-state machines. In, W.-K. Ng, M. Kitsuregawa, J. Li, &, K. Chang, (Eds.), Advances in knowledge discovery and data mining, vol. 3918 of lecture notes in computer science (pp. 265-274). Berlin/Heidelberg: Springer.
38. Lehmann, S., Jackson, A.D., &, Lautrup, B.E., (2006). Measures for measures. Nature, 444 (7122), 1003-1004.
39. Leskovec, J., Backstrom, L., &, Kleinberg, J., (2009). Meme-tracking and the dynamics of the news cycle. In Proceedings of the 15th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '09 (pp. 497-506). New York, NY: ACM.
40. Lewis, D., (1991). Evaluating text categorization. In Proceedings of the Speech and Natural Language Workshop, pp. 312-318.
41. Liang, Y., Li, Q., &, Qian, T., (2011). Finding relevant papers based on citation relations. In, H. Wang, S. Li, S. Oyama, X. Hu, &, T. Qian, (Eds.), Web-age information management, vol. 6897 of lecture notes in computer science (pp. 403-414). Berlin/Heidelberg: Springer.
42. Long, S.J., (1997). Regression models for categorical and limited dependent variables. Thousand Oaks, CA: SAGE.
43. Marchant, T., (2009). Score-based bibliometric rankings of authors. Journal of the American Society for Information Science and Technology, 60 (6), 1132-1137.
44. Di Marco, C., Kroon, F., &, Mercer, R., (2006). Using hedges to classify citations in scientific articles. In, J. Shanahan, Y. Qu, &, J. Wiebe, (Eds.), Computing attitude and affect in text: theory and applications, vol. 20 of the information retrieval series (pp. 247-263). Netherlands: Springer.
45. Mazloumian, A., (2012). Predicting scholars' scientific impact. PLoS ONE, 7 (11), e49246.
46. Mercer, R.E., Di Marco, C., &, Kroon, F.W., (2004). The frequency of hedging cues in citation contexts in scientific writing. In Advances in artificial intelligence, vol. 3060 of lecture notes in computer science (pp. 75-88). Berlin Heidelberg: Springer.
47. Mohammad, S., &, Turney, P., (2010). Emotions evoked by common words and phrases: Using mechanical turk to create an emotion lexicon. In Proceedings of the NAACL-HLT 2010 Workshop on Computational Approaches to Analysis and Generation of Emotion in Text.
48. Moravcsik, M.J., &, Murugesan, P., (1975). Some results on the function and quality of citations. Social Studies of Science, 5 (1), 86-92.
49. Nanba, H., &, Okumura, M., (1999). Towards multi-paper summarization using reference information. In International Joint Conference on Artificial Intelligence, Vol. 16, pp. 926-931.
50. Osgood, C.E., Suci, G.J., &, Tannenbaum, P.H., (1957). The measurement of meaning. University of Illinois Press.
51. Peroni, S., &, Shotton, D., (2012). FaBiO and CiTO: Ontologies for describing bibliographic resources and citations. Web Semantics: Science, Services and Agents on the World Wide Web, 17 (0), 33-43.
52. Pham, S., &, Hoffmann, A., (2003). A new approach for scientific citation classification using cue phrases. In, T. Domonkos Gedeon, &, L. Chun Che Fung, (Eds.), AI 2003: Advances in artificial intelligence, vol. 2903 of lecture notes in computer science (pp. 759-771). Berlin Heidelberg: Springer.
53. Pinski, G., &, Narin, F., (1976). Citation influence for journal aggregates of scientific publications: Theory, with application to the literature of physics. Information Processing & Management, 12 (5), 297-312.
54. Porter, M., (1980). An algorithm for suffix stripping. Program, 14, 130-137.
55. Qazvinian, V., &, Radev, D.R., (2008). Scientific paper summarization using citation summary networks. In Proceedings of the 22nd International Conference on Computational Linguistics-Volume 1, COLING '08 (pp. 689-696). Stroudsburg, PA: Association for Computational Linguistics.
56. Qazvinian, V., &, Radev, D.R., (2010). Identifying non-explicit citing sentences for citation-based summarization. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, ACL '10 (pp. 555-564). Stroudsburg, PA: Association for Computational Linguistics.
57. Qi, X., Nie, L., &, Davison, B.D., (2007). Measuring similarity to detect qualified links. In Proceedings of the 3rd International Workshop on Adversarial Information Retrieval on the Web, AIRWeb '07 (pp. 49-56). New York, NY: ACM.
58. Radev, D.R., Muthukrishnan, P., &, Qazvinian, V., (2009). The ACL anthology network corpus. In Proceedings of the 2009 Workshop on Text and Citation Analysis for Scholarly Digital Libraries, NLPIR4DL '09 (pp. 54-61). Stroudsburg, PA: Association for Computational Linguistics.
59. Ritchie, A., Robertson, S., &, Teufel, S., (2008). Comparing citation contexts for information retrieval. In Proceedings of the 17th ACM Conference on Information and Knowledge Management, CIKM '08 (pp. 213-222). New York, NY: ACM.
60. Rokach, L., Kalech, M., Blank, I., &, Stern, R., (2011). Who is going to win the next association for the advancement of artificial intelligence fellowship award? Evaluating researchers by mining bibliographic data. Journal of the American Society for Information Science and Technology, 62 (12), 2456-2470.
61. Schreiber, M., (2008). To share the fame in a fair way, hm modifies h for multi-authored manuscripts. New Journal of Physics, 10 (4), 040201.
62. Shotton, D., (2010). CiTO, the citation typing ontology. Journal of Biomedical Semantics, 1 (Suppl. 1), S6.
63. Simkin, M.V., &, Roychowdhury, V.P., (2003). Read before you cite! Complex Systems, 14 (3), 269-274.
64. Small, H., (1982). Citation context analysis. Progress in Communication Sciences, 3, 287-310.
65. Soon, W.M., Ng, H.T., &, Lim, D.C.Y., (2001). A machine learning approach to coreference resolution of noun phrases. Computational Linguistics, 27 (4), 521-544.
66. Stone, P.J., Dunphy, D.C., &, Smith, M.S., (1966). The general inquirer: A computer approach to content analysis. Cambridge: MIT Press.
67. Taheriyan, M., (2011). Subject classification of research papers based on interrelationships analysis. In Proceedings of the 2011 Workshop on Knowledge Discovery, Modeling and Simulation, KDMS '11 (pp. 39-54). New York, NY: ACM.
68. Teufel, S., Siddharthan, A., &, Tidhar, D., (2006). Automatic classification of citation function. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, EMNLP '06 (pp. 103-110). Stroudsburg, PA: Association for Computational Linguistics.
69. Turney, P.D., (1993). Robust classification with context-sensitive features. In Proceedings of the Sixth International Conference on Industrial and Engineering Applications of Artificial Intelligence and Expert Systems, pp. 268-276.
70. Turney, P.D., (1996). The management of context-sensitive features: A review of strategies. In Proceedings of Workshop on Learning in Context-Sensitive Domains at the 13th International Conference on Machine Learning (ICML96), pp. 60-66.
71. Turney, P.D., &, Littman, M.L., (2003). Measuring praise and criticism: Inference of semantic orientation from association. ACM Transactions on Information Systems, 21 (4), 315-346.
72. Turney, P.D., &, Pantel, P., (2010). From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37, 141-188.
73. Vanclay, J.K., (2013). Factors affecting citation rates in environmental science. Journal of Informetrics, 7 (2), 265-271.
74. Vellino, A., (2010). A comparison between usage-based and citation-based methods for recommending scholarly research articles. Proceedings of the American Society for Information Science and Technology, 47 (1), 1-2.
75. Waltman, L., &, van Eck, N.J., (2012). The inconsistency of the h-index. Journal of the American Society for Information Science and Technology, 63 (2), 406-415.
76. Wilhite, A.W., &, Fong, E.A., (2012). Coercive citation in academic publishing. Science, 335 (6068), 542-543.
77. Witten, I.H., Frank, E., &, Hall, M.A., (2011). Data mining: Practical machine learning tools and techniques. Burlington, MA: Morgan Kaufmann.
78. Yan, E., &, Ding, Y., (2010). Weighted citation: An indicator of an article's prestige. Journal of the American Society for Information Science and Technology, 61 (8), 1635-1643.
79. Zitt, M., &, Small, H., (2008). Modifying the journal impact factor by fractional citation weighting: The audience factor. Journal of the American Society for Information Science and Technology, 59 (11), 1856-1860.