Empirical distributional semantics: Methods and biomedical applications

Journal of Biomedical Informatics

Volumn 42, Issue 2, 2009, Pages 390-405

  Cohen, Trevor ^a   Widdows, Dominic ^b  

^a School of Computing and Informatics   (United States)

^b GOOGLE INC   (United States)

Author keywords

Context vectors; Distributional semantics; Latent semantic analysis; Methodological review; Natural language processing; Random indexing; Semantic similarity

Indexed keywords

CONTEXT VECTORS; DISTRIBUTIONAL SEMANTICS; LATENT SEMANTIC ANALYSIS; METHODOLOGICAL REVIEW; NATURAL LANGUAGE PROCESSING; RANDOM INDEXING; SEMANTIC SIMILARITY;

APPLICATIONS; COMPUTATIONAL LINGUISTICS; INDEXING (MATERIALS WORKING); INDEXING (OF INFORMATION); INFORMATION SERVICES; NATURAL LANGUAGE PROCESSING SYSTEMS; SEMANTICS;

INFORMATION THEORY;

COGNITION; COMPUTER PROGRAM; EVALUATION; INFORMATION PROCESSING; INFORMATION RETRIEVAL; LANGUAGE PROCESSING; LINGUISTICS; MATHEMATICAL COMPUTING; NONHUMAN; PRIORITY JOURNAL; REVIEW; SEMANTICS; SEQUENCE ANALYSIS;

ABSTRACTING AND INDEXING AS TOPIC; COMPUTATIONAL BIOLOGY; INFORMATION STORAGE AND RETRIEVAL; MEDLINE; MODELS, STATISTICAL; NATURAL LANGUAGE PROCESSING; NEURAL NETWORKS (COMPUTER); REPRODUCIBILITY OF RESULTS; SEMANTICS; SOFTWARE; VOCABULARY, CONTROLLED;

EID: 61949432274     PISSN: 15320464     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.jbi.2009.02.002     Document Type: Review

References (119)

1. Zhang J. Representations of health concepts: a cognitive perspective. J Biomed Inform (2002) 3517-3524
2. Hearst M. Automatic acquisition of hyponyms from large text corpora. Proc 14th Conf Comput Linguist (1992) 539-545
3. Friedman C., Kra P., Yu H., Krauthammer M., and Rzhetsky A. GENIES: a natural-language processing system for the extraction of molecular pathways from journal articles. Bioinform (Oxford, England) 17 Suppl. 1 (2001) S74-S82
4. Rindflesch T.C., and Fiszman M. The interaction of domain knowledge and linguistic structure in natural language processing: interpreting hypernymic propositions in biomedical text. J Biomed Inform 36 (2003) 462-477
5. Lund K., and Burgess C. Producing high-dimensional semantic spaces from lexical co-occurrence. Behav Res Methods Instrum Comp 28 (1996) 203-208
6. Schutze H. Word space. Adv Neural Info Process Syst 5 (1993) 895-902
7. Landauer T.K., and Dumais S.T. A solution to Plato's problem: the latent semantic analysis theory of acquisition, induction, and representation of knowledge. Psychol Rev 104 (1997) 211-240
8. Hofmann T. Probabilistic latent semantic analysis. Proc Uncertainty Artif Int UAI'99 (1999) 289-296
9. Blei D.M., Ng A.Y., and Jordan M.I. Latent Dirichlet allocation. J Mach Learn Res 3 (2003) 993-1022
10. Griffiths T., and Steyvers M. A probabilistic approach to semantic representation. Proc 24th Annu Conf Cognit Sci Soc (2002) 381-386
11. Hersh W., Buckley C., Leone T.J., and Hickam D. OHSUMED: an interactive retrieval evaluation and new large test collection for research. Proc 17th Annu Int ACM SIGIR Conf Res Dev Info Retriev (1994) 192-201
12. Cederberg S, Widdows D, Peters S. Infomap NLP software: an open-source package for natural language processing. Available from: December 2008.
13. Schütze H. Automatic word sense discrimination. Comput Linguist 24 (1998) 97-123
14. Jones M.N., and Mewhort D.J.K. Representing word meaning and order information in a composite holographic lexicon. Psychol Rev 114 (2007) 1-37
15. Pado S., and Lapata M. Dependency-based construction of semantic space models. Comput Linguist 33 (2007) 161-199
16. Kanerva P., Kristofersson J., and Holst A. Random indexing of text samples for latent semantic analysis. Proc 22nd Annu Conf Cogn Sci Soc (2000) 10-36
17. Dumais S.T. Improving the retrieval of information from external sources. Behav Res Methods, Instrum Comput 23 (1991) 229-236
18. Gorman J., and Curran J.R. Random indexing using statistical weight functions. Proc Conf Empirical Methods Nat Lang Process (EMNLP), Sydney, Australia (2006) 457-464
19. Strang G. Introduction to linear Algebra. Wellesley, Cambridge Pr; 2003.
20. Karlgren J., and Sahlgren M. From words to understanding. Found Real-World Intell (2001) 294-308
21. Pereira F., Tishby N., and Lee L. Distributional clustering of English words. Proc 31st Conf Assoc Comput Linguist (1993) 183-190
22. Heinrich G. Parameter estimation for text analysis. Web: http://www.arbylon.net/publications/text-est.pdf.2005.
23. Steyvers M., and Griffiths T. Probabilistic Topic Models. In: Landauer T., McNamara D., Dennis S., and Kintsch W. (Eds). Handbook of latent semantic analysis (2007), Lawrence Erlbaum, Mahwah, NJ
24. Shannon C.E. Prediction and entropy of printed English. Bell Syst Tech J 30 (1951) 50-64
25. Birkhoff G., and von Neumann J. The logic of quantum mechanics. Ann Mathemat 37 (1936) 823-843
26. Rijsbergen C.J.V. The geometry of information retrieval (2004), Cambridge University Press
27. Salton G., Wong A., and Yang C.S. A vector space model for automatic indexing. Commun ACM 18 (1975) 613-620
28. Robertson S., and Spark-Jones K. Relevance weighting of search terms. J Am Soc Info Sci 27 (1976) 129-146
29. Quillian MR. Semantic memory. In: Minsky, M., editors. Semantic information processing. MIT Press; 2004. p. 227-70.
30. Lesk M. Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone. In: Proceedings of the 5th annual international conference on Systems documentation. New York, NY, USA: ACM; 1986. p. 24-6.
31. McDonald J.E., Plate T.A., and Schvaneveldt R.W. Using pathfinder to extract semantic information from text. Pathfinder associative networks: studies in knowledge organization (1990), Ablex Publishing Corp. 149-164
32. Yarowsky D. Unsupervised word sense disambiguation rivaling supervised methods. Proceedings of the 33rd annual meeting on Association for Computational Linguistics (1995), Association for Computational Linguistics, Cambridge, Massachusetts 189-196
33. Belkin N.J., and Croft W.B. Retrieval techniques. Annual review of information science and technology vol. 22 (1987), Elsevier Science Inc 109-145
34. Widdows D. Geometry and meaning. Center for the study of language and information/SRI; 2004.
35. Brin S., and Page L. The anatomy of a large-scale hypertextual Web search engine. Comput Netw ISDN Syst (1998) 107-117
36. Volk M., Ripplinger B., Vintar S., Buitelaar P., Raileanu D., and Sacaleanu B. Semantic annotation for concept-based cross-language medical information retrieval. Int J Med Info 67 (2002) 97-112
37. Maedche A., and Staab S. Ontology learning for the semantic web. Intell Syst IEEE 16 (2001) 72-79
38. Charniak E. Statistical language learning. Bradford Books; 1993.
39. Rilo E, Jones R. Learning dictionaries for information extraction by multi-level bootstrapping. In: Proceedings of AAAI-99; 1999. p. 474.
40. Cederberg S., and Widdows D. Using LSA and noun coordination information to improve the precision and recall of automatic hyponymy extraction. Proc 7th Conf Nat Lang Learn HLT-NAACL 2003-vol. 4 (2003) 111-118
41. Domingos P. Toward knowledge-rich data mining. Data Min Knowledge Disc 15 1 (2007) 21-28
42. Deerwester S., Dumais S.T., Furnas G.W., Landauer T.K., and Harshman R. Indexing by latent semantic analysis. J Am Soc Info Sci 41 (1990) 391-407
43. Widdows D, Ferraro K. Semantic vectors: A scalable open source package and online technology management application. To appear in Sixth International Conference on Language Resources and Evaluation (LREC 2008); 2008.
44. VOCABSYN Frameset1 [Internet]. Available from: .
45. Rapp R. Word sense discovery based on sense descriptor dissimilarity. In: Ninth machine translation summit; 2003. p. 315-22.
46. Landauer TK. Personal communication.
47. Pustejowsky J.Y. The generative Lexicon. Comput Linguist 17 4 (1991) 409-441
48. Koehn P. Europarl: a parallel corpus for statistical machine translation. MT summit; 2005.
49. Widdows D, Peters S, Cederberg S, Chan CK, Steffen D, Buitelaar P. Unsupervised monolingual and bilingual word-sense disambiguation of medical documents using UMLS. Natural Language Processing in Biomedicine ACL 2003 Workshop; 2003. p. 9-16.
50. Landauer TK, Laham D, Rehder B, Schreiner ME. How well can passage meaning be derived without using word order. A comparison of latent semantic analysis and humans. Proceedings of the 19th annual conference of the cognitive science society: August 7-10 1997, Stanford University; 1997.
51. Landauer T.K., Laham D., and Foltz P.W. The intelligent essay assessor. IEEE Intell Syst 15 (2000) 27-31
52. Swayne D.F., Lang D.T., Buja A., and Cook D. GGobi: evolving from XGobi into an extensible framework for interactive data visualization. Comput Stat Data Anal 43 (2003) 423-444
53. Landauer T.K., Laham D., and Derr M. From paragraph to graph: latent semantic analysis for information visualization. Proc Nat Acad Sci 101 (2004) 5214-5219
54. Burgess C., and Lund K. The dynamics of meaning in memory. Cogn Dyn Concept Represent Change Humans Mach (2000) 117-156
55. Widdows D., and Cederberg S. Monolingual and bilingual concept visualization from corpora. Proc Conf North American Chap Assoc Comput Linguist Human Lang Technol Demonst-vol. 4 (2003) 31-32
56. Schvaneveldt R.W. Pathfinder associative networks: studies in knowledge organization (1990), Ablex Publishing Corp, Norwood, NJ, USA
57. Cohen T.A. Exploring MEDLINE space with random indexing and pathfinder networks. AMIA Annu Symp Proc (2008) 126-130
58. Heer J., Card S.K., and Landay J.A. Prefuse: a toolkit for interactive information visualization. Conf Human Factors Comput Syst (2005) 421-430
59. Curran J.R. Supersense tagging of unknown nouns using semantic similarity. Proc 43rd Annu Meeting Assoc Comput Linguist (2005) 26-33
60. Harris Z.S. The structure of science information. J Biomed Info 35 (2002) 215-221
61. Homayouni R., Heinrich K., Wei L., and Berry M.W. Gene clustering by latent semantic indexing of MEDLINE abstracts. Bioinformat (Oxford, England) 21 1 (2005) 104-115
62. Zambrano N., Gianni D., Bruni P., Passaro F., Telese F., and Russo T. Fe65 is not involved in the platelet-derived growth factor-induced processing of Alzheimer's amyloid precursor protein, which activates its caspase-directed cleavage. J Biol Chem 279 (2004) 16161-16169
63. Glenisson P., Antal P., Mathys J., Moreau Y., and De Moor B. Evaluation of the vector space representation in text-based gene clustering. Pac Symp Biocomput (2003) 391-402
64. Klein-Seetharaman J. The use of analogies for interdisciplinary research in the convergence of nano-, bio- and information technology. In: NSF report on societal implications of nanoscience and nanotechnology; 2005. p. 128-33.
65. Ganapathiraju M.K., Klein-Seetharaman J., Balakrishnan N., and Reddy R. Characterization of protein secondary structure. Signal Process Mag IEEE 21 (2004) 78-87
66. Stuart G.W., and Berry M.W. A comprehensive whole genome bacterial phylogeny using correlated peptide motifs defined in a high dimensional vector space. J Bioinform Comput Biol (2003) 1475-1493
67. Stuart G.W., and Berry M.W. An SVD-based comparison of nine whole eukaryotic genomes supports a coelomate rather than ecdysozoan lineage. BMC Bioinformat 5 (2004) 204-217
68. Widdows D, Cohen, T. Semantic Vector Combinations and the Synoptic Gospels. In: Proceedings of the third quantum interaction symposium (March 25-27, 2009 - DFKI, Saarbrücken).
69. Gordon M.D., and Dumais S. Using latent semantic indexing for literature based discovery. J Am Soc Info Sci 49 (1998) 674-685
70. Cole R.J., and Bruza P.D. A bare bones approach to literature-based discovery: an analysis of the raynaud's/Fish-Oil and Migraine-Magnesium discoveries in semantic space. Discov Sci (2005) 84-98
71. Bruza P, Cole R, Song D, Bari Z. Towards operational abduction from a cognitive perspective. Oxford University Press; 2006.
72. Hersh W. Information retrieval: a health and biomedical perspective (2003), Springer, New York, NY
73. Lin J., and Wilbur W.J. PubMed related articles: a probabilistic topic-based model for content similarity. BMC Bioinformat 8 (2007) 1-423
74. Vanteru B.C., Shaik J.S., and Yeasin M. Semantically linking and browsing PubMed abstracts with gene ontology. BMC Genomics 9 Suppl. 1 (2008) S10
75. Yang Y., and Chute C.G. An example-based mapping method for text categorization and retrieval. ACM Trans Info Syst (TOIS) 12 3 (1994) 252-277
76. Yang Y. An evaluation of statistical approaches to MEDLINE indexing. Proc AMIA Annu Fall Symp (1996) 358-362
77. Yang Y., and Chute C.G. An application of Expert Network to clinical classification and MEDLINE indexing. Proc Annu Symp Comput Appl Med Care (1994) 157-161
78. Cooper GF, Miller RA. An Experiment Comparing Lexical and Statistical Methods for Extracting MeSH Terms from Clinical Free Text. JAMIA 1988;62-75.
79. Aronson A.R., Bodenreider O., Chang H.F., Humphrey S.M., Mork J.G., Nelson S.J., et al. The NLM indexing initiative. Proc AMIA Symp (2000) 17-21
80. Aronson A.R. Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program. Proc AMIA Symp (2001) 1717-1721
81. Cohen A., Hersh W., Dubay C., and Spackman K. Using co-occurrence network structure to extract synonymous gene and protein names from MEDLINE abstracts. BMC Bioinformat 6 (2005) 103
82. Friedman C. A broad-coverage natural language processing system. Proc AMIA Symp (2000) 19270-19274
83. Sager N, Lyman M, Bucknall C, Nhan N, Tick LJ. Natural language processing and the representation of clinical data. J Am Med Info Assoc JAMIA 1994; 1: 142-60.
84. Harris Z.S. Mathematical structures of language (1968), Interscience Publishers, New York
85. Chute C.G., Yang Y., and Evans D.A. Latent Semantic Indexing of medical diagnoses using UMLS semantic structures. Proc Annu Symp Comput Appl Med Care (1991) 1859
86. Chute C.G., and Yang Y. An evaluation of concept based latent semantic indexing for clinical information retrieval. Proc Annu Symp Comput Appl Med Care 639 (1992) 43
87. Yang Y., and Chute C.G. A Linear Least Squares Fit mapping method for information retrieval from natural language texts. Proc 14th Conf Comput Linguist vol. 2 (1992), Association for Computational Linguistics, Nantes, France 447-453
88. Pedersen T., Pakhomov S.V.S., Patwardhan S., and Chute C.G. Measures of semantic similarity and relatedness in the biomedical domain. J Biomed Info 40 (2007) 288-299
89. Rubenstein H., and Goodenough J.B. Contextual correlates of synonymy. Commun ACM 8 (1965) 627-633
90. Budanitsky A, Hirst G. Semantic distance in WordNet: An experimental, application-oriented evaluation of five measures. Workshop on WordNet and Other Lexical Resources; 2001.
91. Fan J.W., and Friedman C. Semantic classification of biomedical concepts using distributional similarity. J Am Med Info Assoc 14 (2007) 467-477
92. Grefenstette G. Corpus-derived first, second and third-order word affinities. Proc Euralex (1994) 279-290
93. Lin D. Automatic retrieval and clustering of similar words. Proc 17th Int Conf Comput Linguist (1998) 768-774
94. Curran J.R., and Moens M. Scaling context space. Proc 40th Annu Meeting Assoc Comput Linguist (2001) 231-238
95. In: Landauer T.K., McNamara D., Dennis S., and Kintsch W. (Eds). Handbook of latent semantic analysis. Lawrence handbook of latent semantic analysis (2007), Lawrence Erlbaum Associates, Mahwah, N.J
96. Wiemer-Hastings P., and Zipitria I. Rules for syntax, vectors for semantics. Proc 23rd Annu Conf Cogn Sci Soc (2001) 1112-1117
97. Kanejiya D., Kumar A., and Prasad S. Automatic evaluation of students' answers using syntactically enhanced LSA. Proceedings of the HLT-NAACL 03 workshop on Building educational applications using natural language processing 2 (2003) 53-60
98. Cohen T., Blatter B., and Patel V. Exploring dangerous neighborhoods: latent semantic analysis and computing beyond the bounds of the familiar. AMIA? Annual Symposium proceedings/AMIA Symposium. AMIA Symposium (2005) 151-155
99. Cohen T., Blatter B., and Patel V. Simulating expert clinical comprehension: adapting latent semantic analysis to accurately extract clinical concepts from psychiatric narrative. J Biomed Info 41 6 (2008) 1070-1087
100. Sharda P., Das A.K., Cohen T.A., and Patel V. Customizing clinical narratives for the electronic medical record interface using cognitive methods. Int J Med Info 75 (2006) 346-368
101. Widdows D, Peters S. Word Vectors and Quantum Logic Experiments with negation and disjunction. Mathematics of Language, 8, Bloomington, Indiana; 2003.
102. Elvevaag B., Foltz P.W., Weinberger D.R., and Goldberg T.E. Quantifying incoherence in speech: an automated methodology and novel application to schizophrenia. Schizophrenia Res 93 (2007) 304-316
103. Cline R.J., and Haynes K.M. Consumer health information seeking on the Internet: the state of the art. Health Edu Res 16 6 (2001) 671-692
104. Chen G, Warren J, Evans J. Automatically generated consumer health metadata using semantic spaces [Internet]. In: Proceedings of the second Australasian workshop on Health data and knowledge management. Wollongong, NSW, Australia: Australian Computer Society, Inc. 80; 2008. p. 9-15.
105. McArthur R, Bruza P, Warren J, Kralik D. Projecting computational sense of self: a study of transition in a chronic illness online community. In: system sciences, 2006. HICSS '06. Proc 39th Annu Hawaii Int Conf; 2006. p. 91c.
106. Berry MW, Mezher D, Philippe B, Sameh A. Parallel computation of the singular value decomposition. In: E. Kontoghiorghes editor. Handbook on parallel computing and statistics. CRC Press; 2003. p. 117-164.
107. Johnson W., and Lindenstrauss J. Extension of lipshitz mapping to hilbert space. Contemp Math 26 (1984) 189-206
108. Sahlgren M, Holst A, Kanerva P. Permutations as a Means to Encode Order in Word Space. Proceedings of the 30th Annual Meeting of the Cognitive Science Society (CogSci'08), July 23-26, Washington DC, USA.
109. Sahlgren M. The word-space model: using distributional analysis to represent syntagmatic and paradigmatic relations between words in high-dimensional vector spaces. Ph.D. Dissertation, Department of Linguistics, Stockholm University; 2006.
110. Dennis S. Introducing word order within the LSA framework. In: Landauer T, McNamara D, Dennis S, Kintsch W, editors. Handbook of latent semantic analysis. Lawrence Erlbaum Associates; 2007. p. 449-67.
111. Griffiths T.L., Steyvers M., Blei D., and Tenenbaum J.B. Integrating topics and syntax. Adv Neural Info Process Syst 17 (2005) 537-544
112. Widdows D. Unsupervised methods for developing taxonomies by combining syntactic and statistical information. Proc 2003 Conf North American Chap Assoc Comput Linguist Human Lang Technol 1 (2003) 197-204
113. Widdows D. Semantic Vector Products: Some Initial Investigations. Proceedings of the second AAAI symposium on quantum interaction; 2008.
114. Sahlgren M, Coster R. using bag-of-concepts to improve the performance of support vector machines in text categorization. Proceedings of the 20th international conference on computational linguistics, COLING; 2004.
115. Berry M, Do T, O?Brien G, Krishna V, Varadhan S. SVDPACKC (Version 1.0) user?s guide. University of Tennessee Computer Science Department Technical Report. CS-93-194; 1993.
116. Giles JT, Wo L, Berry MW. GTP (General Text Parser) Software for Text Mining. Statistical Data Mining and Knowledge Discovery; 2001.
117. Topic Modeling Toolbox [Internet]. Available from: .
118. Latent Dirichlet Allocation in C [Internet]. Available from: .
119. Arbylon projects: knowceans [Internet]. Available from: .