Character-level neural network for biomedical named entity recognition

Journal of Biomedical Informatics

Volumn 70, Issue , 2017, Pages 85-91

  Gridach, Mourad ^a  

^a IBN ZOHR UNIVERSITY   (Morocco)

Author keywords

Biomedical named entity recognition; Deep neural networks; Natural language processing

Indexed keywords

AUTOMATION; DATA HANDLING; GENES; LINGUISTICS; LONG SHORT-TERM MEMORY; NATURAL LANGUAGE PROCESSING SYSTEMS; NETWORK ARCHITECTURE; NEURAL NETWORKS; RANDOM PROCESSES;

BIOMEDICAL NAMED ENTITY RECOGNITION; CONDITIONAL RANDOM FIELD; FEATURE ENGINEERINGS; NATURAL LANGUAGE PROCESSING; NOVEL NEURAL NETWORK; STATE-OF-THE-ART PERFORMANCE; STATE-OF-THE-ART SYSTEM; WORD AND CHARACTERS;

DEEP NEURAL NETWORKS;

ARTICLE; ARTIFICIAL NEURAL NETWORK; AUTOMATED PATTERN RECOGNITION; COMPUTER LANGUAGE; CONTROLLED STUDY; MEDICAL TECHNOLOGY; NATURAL LANGUAGE PROCESSING; PRIORITY JOURNAL; PROCESS MODEL; WORD PROCESSING; ALGORITHM; GENE; HUMAN;

PROTEIN;

ALGORITHMS; GENES; HUMANS; NATURAL LANGUAGE PROCESSING; NEURAL NETWORKS (COMPUTER); PROTEINS;

EID: 85019375942     PISSN: 15320464     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jbi.2017.05.002     Document Type: Article

References (64)

1. Tang, B., Cao, H., Wang, X., Chen, Q., Xu, H., Evaluating word representation features in biomedical named entity recognition tasks. BioMed Res. Int., 2014.
2. Friedman, C., Alderson, P.O., Austin, J.H., Cimino, J.J., Johnson, S.B., A general natural-language text processor for clinical radiology. J. Am. Med. Inform. Assoc. 1:2 (1994), 161–174.
3. Haug, P.J., Koehler, S., Lau, L.M., Wang, P., Rocha, R., Huff, S.M., Experience with a mixed semantic/syntactic parser. Proceedings of the Annual Symposium on Computer Application in Medical Care, 1995, American Medical Informatics Association, 284.
4. Fukuda, K.-i., Tsunoda, T., Tamura, A., Takagi, T., et al. Toward information extraction: identifying protein names from biological papers. PAC Symp Biocomput, vol. 707, 1998, 707–718.
5. Proux, D., Rechenmann, F., Julliard, L., Pillet, V., Jacq, B., Detecting gene symbols and names in biological texts. Genome Inform. 9 (1998), 72–80.
6. Zeng, Q.T., Goryachev, S., Weiss, S., Sordo, M., Murphy, S.N., Lazarus, R., Extracting principal diagnosis, co-morbidity and smoking status for asthma research: evaluation of a natural language processing system. BMC Med. Inform. Decis. Making, 6(1), 2006, 30.
7. Denny, J.C., Miller, R.A., Johnson, K.B., Spickard, A. III, Development and evaluation of a clinical note section header terminology. AMIA, 2008.
8. Aronson, A.R., Lang, F.-M., An overview of metamap: historical perspective and recent advances. J. Am. Med. Inform. Assoc. 17:3 (2010), 229–236.
9. Savova, G.K., Masanz, J.J., Ogren, P.V., Zheng, J., Sohn, S., Kipper-Schuler, K.C., Chute, C.G., Mayo clinical text analysis and knowledge extraction system (cTAKES): architecture, component evaluation and applications. J. Am. Med. Inform. Assoc. 17:5 (2010), 507–513.
10. Gaizauskas, K.H.R., Demetriou, G., Term recognition and classification in biological science journal articles. Proceedings of the Computational Terminology for Medical and Biological Applications Workshop of the 2nd International Conference on NLP, 2000, 37–44.
11. Rindfleisch, W.J., Thomas, C., Tanabe, L., Edgar: extraction of drugs, genes and relations from the biomedical literature. Proc Pacific Symposium on Biocomputing, 2003.
12. Song, H.Y.M., Han, W.S., Developing a hybrid dictionary-based bio-entity recognition technique. Proceedings of the ACM Eighth International Workshop on Data and Text Mining in Biomedical Informatics, Shanghai, China, 2014.
13. Nobata, N.C.C., Tsujii, J., Automatic term identification and classification in biology texts. Proceedings of the 5th NLPRS, 1999, 369–374.
14. Rabiner, L.R., A tutorial on hidden markov models and selected applications in speech recognition. Proc. IEEE, vol. 77, 1989, 257–285.
15. Zhao, S., Named entity recognition in biomedical texts using an HMM model. Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications, Stroudsburg, PA, USA, 2004, 84–87.
16. Mccallum, D.F.A., Pereira, F., Maximum entropy markov models for information extraction and segmentation. Proceedings of the 17th International Conference on Machine Learning, 2000, 591–598.
17. Finkel, J., Dingare, S., Nguyen, H., Nissim, M., Manning, C., Sinclair, G., Exploiting context for biomedical entity recognition: from syntax to the web. Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications, 2004, Association for Computational Linguistics, 88–91.
18. Lafferty, J., McCallum, A., Pereira, F., et al. Conditional random fields: probabilistic models for segmenting and labeling sequence data. Proceedings of the Eighteenth International Conference on Machine Learning, ICML, vol. 1, 2001, 282–289.
19. Settles, B., Biomedical named entity recognition using conditional random fields and rich feature sets. Proceedings of the International Joint Workshop on Natural Language Processing in Biomedicine and its Applications, Stroudsburg, PA, USA, 2004, 104–107.
20. Lishuang Li, Y.J., Jin, Liuke, Huang, D., Recognizing biomedical named entities based on the sentence vector/twin word embeddings conditioned bidirectional LSTM. Lecture Notes on Computer Science, 2016.
21. Burges, C., A tutorial on support vector machines for pattern recognition. Data Min. Knowl. Disc. 2:2 (1998), 21–167.
22. Wu, Y.-C., Fan, T.-K., Lee, Y.-S., Yen, S.-J., Extracting named entities using support vector machines. International Workshop on Knowledge Discovery in Life Science LIterature, 2006, Springer, 91–103.
23. Tsochantaridis, H.T. I, T, Joachims, Y, A., Large margin methods for structured and interdependent output variables. J. Mach. Learn. Res. 6:1 (2005), 1453–1484.
24. Tang, B., Cao, H., Wu, Y., Jiang, M., Xu, H., Clinical entity recognition using structural support vector machines with rich features. Proceedings of the ACM Sixth International Workshop on Data and Text Mining in Biomedical Informatics, 2012, ACM, 13–20.
25. Li, D., Kipper-Schuler, K., Savova, G., Conditional random fields and support vector machines for disorder named entity recognition in clinical texts. Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing, 2008, Association for Computational Linguistics, 94–95.
26. Hinton, G., Deng, L., Yu, D., Dahl, G.E., Mohamed, A.-r., Jaitly, N., Senior, A., Vanhoucke, V., Nguyen, P., Sainath, T.N., Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups. IEEE Signal Process. Mag. 29:6 (2012), 82–97.
27. S.R. Kaiming He, Xiangyu Zhang, J. Sun, Deep Residual Learning for Image Recognition, arXiv preprint.
28. Silver, D., Huang, A., Maddison, C.J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Mastering the game of go with deep neural networks and tree search. Nature 529:7587 (2016), 484–489.
29. Socher, R., Perelygin, A., Wu, J.Y., Chuang, J., Manning, C.D., Ng, A.Y., Potts, C., et al. Recursive deep models for semantic compositionality over a sentiment treebank. Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), Citeseer, vol. 1631, 2013, 1642.
30. dos Santos, C., Gatti, M., Deep convolutional neural networks for sentiment analysis of short texts. the 25 the International Conference on Computational Linguistics, Dublin, Ireland, 2014.
31. Mikolov, T., Kombrink, S., Burget, L., Černockỳ, J., Khudanpur, S., Extensions of recurrent neural network language model. Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on, 2011, IEEE, 5528–5531.
32. K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, Y. Bengio, Learning Phrase Representations Using RNN Encoder-Decoder for Statistical Machine Translation. Available from: < 1406.1078>.
33. Sutskever, I., Vinyals, O., Le, Q.V., Sequence to sequence learning with neural networks. Advances in Neural Information Processing Systems, 2014, 3104–3112.
34. A. Graves, Generating Sequences with Recurrent Neural Networks, arXiv preprint.
35. D.E. Rumelhart, G.E. Hinton, R.J. Williams, Learning Internal Representations by Error Propagation, Tech. Rep., DTIC Document, 1985.
36. Wu, Y., Xu, J., Jiang, M., Zhang, Y., Xu, H., A study of neural word embeddings for named entity recognition in clinical text. AMIA Annual Symposium Proceedings, vol. 2015, 2015, American Medical Informatics Association, 1326.
37. Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R., Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15:1 (2014), 1929–1958.
38. Bengio, Y., Simard, P., Frasconi, P., Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Netw. 5:2 (1994), 157–166.
39. T.M. Razvan Pascanu, Y. Bengio, On the Difficulty of Training Recurrent Neural Networks. Available from: < 1211.5063>.
40. Hochreiter, S., Schmidhuber, J., Long short-term memory. Neural Comput. 9:8 (1997), 1735–1780.
41. M. Boden, A Guide to Recurrent Neural Networks and Backpropagation, Technical Report, 2002.
42. B. Plank, A. Søgaard, Y. Goldberg, Multilingual Part-of-Speech Tagging with Bidirectional Long Short-term Memory Models and Auxiliary Loss. Available from: < 1604.05529>.
43. M. Ballesteros, C. Dyer, N.A. Smith, Improved Transition-based Parsing by Modeling Characters Instead of Words with LSTMs. Available from: < 1508.00657>.
44. Graves, A., Fernández, S., Schmidhuber, J., Bidirectional LSTM networks for improved phoneme classification and recognition. International Conference on Artificial Neural Networks, 2005, Springer, 799–804.
45. Yu, Z., Ramanarayanan, V., Suendermann-Oeft, D., Wang, X., Zechner, K., Chen, L., Tao, J., Ivanou, A., Qian, Y., Using bidirectional LSTM recurrent neural networks to learn high-level abstractions of sequential features for automated scoring of non-native spontaneous speech. Automatic Speech Recognition and Understanding (ASRU), 2015 IEEE Workshop on, 2015, IEEE, 338–345.
46. Graves, A., Jaitly, N., Mohamed, A.-r., Hybrid speech recognition with deep bidirectional LSTM. Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on, 2013, IEEE, 273–278.
47. Fan, Y., Qian, Y., Xie, F.-L., Soong, F.K., TTS synthesis with bidirectional LSTM based recurrent neural networks. Interspeech, 2014, 1964–1968.
48. Luong, M.-T., Manning, C.D., Achieving open vocabulary neural machine translation with hybrid word-character models. Proceedings of ACL, Berlin, Germany, 2016.
49. W. Ling, T. Luís, L. Marujo, R.F. Astudillo, S. Amir, C. Dyer, A.W. Black, I. Trancoso, Finding Function in form: Compositional Character Models for Open Vocabulary Word Representation. Available from: < 1508.02096>.
50. Junyoung Chung, K.C., Bengio, Y., A character-level decoder without explicit segmentation for neural machine translation. Proceedings of ACL, Berlin, 2016.
51. D.S. Yoon Kim, Yacine Jernite, A.M. Rush, Character-aware neural language models, CoRR. abs/1508.06615.
52. Y. Xiao, K. Cho, Efficient Character-level Document Classification by Combining Convolution and Recurrent Layers. Available from: < 1602.00367>.
53. Luong, T., Socher, R., Manning, C.D., Better word representations with recursive neural networks for morphology. CoNLL, 2013, 104–113.
54. Soricut, R., Och, F., Unsupervised morphology induction using word embeddings. Proceedings of the NAACL-HLT, Denver, Colorado, 2015.
55. James, B., Olivier, B., Frédéric, B., Pascal, L., Razvan, P., Theano: a CPU and GPU math expression compiler. Proceedings of the Python for Scientific Computing Conference (SciPy), 2010.
56. Williams, R.J., Zipser, D., Gradient-based learning algorithms for recurrent networks and their computational complexity. Backpropagation: Theor. Architec. Appl. 1 (1995), 433–486.
57. Geoffrey Hinton, N.S., Swersky, K., Coursera, lecture 6e: rmsprop: divide the gradient by a running average of its recent magnitude. Neural Networks for Machine Learning, 2012.
58. D. Kingma, J. Ba, Adam: A Method for Stochastic Optimization. Available from: < 1412.6980>.
59. M.D. Zeiler, Adadelta: an adaptive learning rate method, CoRR. abs/1212.5701.
60. Jeffrey Pennington, R.S., Manning, C., Glove: Global vectors for word representation. Proceedings of EMNLP, Doha, Qatar, 2014, 1532–1543.
61. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J., Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems, 2013, 3111–3119.
62. Pyysalo, S., Ginter, F., Moen, H., Salakoski, T., Ananiadou, S., Distributional semantics resources for biomedical text processing. Proceedings of LBM 2013, 2013, 39–44 URL .
63. Yao, L., Liu, H., Liu, Y., Li, X., Anwar, M.W., Biomedical named entity recognition based on deep neutral network. Int. J. Hybrid Informat. Technol. 8:8 (2015), 279–288.
64. Zhou, G., Su, J., Exploring deep knowledge resources in biomedical name recognition. Proceedings of the Joint Workshop on Natural Language Processing in Biomedicine and its Applications, Geneva, Switzerland, 2004.