Topic-based Generative Models for Text Information Access

Textual Information Access: Statistical Models

Volumn , Issue , 2013, Pages 129-177

  Chappelier, Jean Cédric ^a  

^a EPFL   (Switzerland)

Author keywords

Fisher kernels; Generative models; Generative vs. discriminative models; Language models; Latent Dirichlet Allocation (LDA); Probabilistic latent semantic indexing (PLSI); Term models; Text information access; Text models; Topic based models

Indexed keywords

EID: 84870290379     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1002/9781118562796.ch5     Document Type: Chapter

References (109)

1. Agarwal A., Daumé H., "A geometric view of conjugate priors", Machine Learning, vol. 81, no. 1, p. 99-113, Springer, 2010.
2. Ahrendt P., Goutte C., Larsen J., "Co-occurrence models in music genre classification", in IEEE International Workshop on Machine Learning for Signal Processing, 2005.
3. Amari S.I., "Natural gradient works efficiently in learning", Neural Computation, vol. 10, p. 251-276, 1998.
4. Amari S.I., Nagaoka H., Methods of Information Geometry, vol. 191 of Translations of Mathematical Monographs, American Mathematical Society, 2000.
5. Asuncion A., Welling M., Smyth P., Teh Y.W., "On smoothing and inference for topic models", in Proceedings of 25th Conference on Uncertainty in Artificial Intelligence (UAI '09), p. 27-34, 2009.
6. Asuncion A.U., Smyth P., Welling M., "Asynchronous distributed estimation of topic models for document analysis", Statistical Methodology, vol. 8, no. 1, p. 3-17, Elsevier, January 2011.
7. Azzopardi L., Girolami M., van Rijsbergen C., "Topic based language models for ad hoc information retrieval", in Proceedings of the International Joint Conference in Neural Networks, 2004.
8. Baayen R.H., Word Frequency Distributions, Kluwer Academic Publishers, 2001.
9. Banerjee A., Merugu S., Dhillon I.S., Ghosh J., "Clustering with Bregman divergences", Journal of Machine Learning Research, vol. 6, p. 1705-1749, 2005.
10. Bishop C.M., "Graphical model", Pattern Recognition and Machine Learning, p. 359-422, Springer, 2006.
11. Blei D.M., Jordan M.I., "Modeling annotated data", in Proceedings of 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 127-134, 2003.
12. Blei D.M., Ng A.Y., Jordan M.I., "Latent Dirichlet Allocation", Journal of Machine Learning Research, vol. 3, p. 993-1022, January 2003.
13. Blei D.M., Jordan M.I., "Modeling annotated data", in Proceedings of 26th Annual International ACM SIGIR Conference on Research and Development in Informaion Retrieval (SIGIR '03), ACM, p. 127-134, 2003.
14. Blei D.M, Griffiths T., Jordan M.I., Tenenbaum J.B., "Hierarchical topic models and the nested Chinese restaurant process", in Proceedings of NIPS 16, 2004.
15. Blei D.M., Lafferty J.D., "Correlated topic models", Advances in Neural Information Processing Systems 18, p. 147-154, 2005.
16. Blei D.M., Lafferty J.D., "A correlated topic model of science", Annals of Applied Statistics, vol. 1, no. 1, p. 17-35, 2007.
17. Blei D.M., McAuliffe J.D., "Supervised topic models", in Platt J., Koller D., Singer Y., Roweis S. (eds), Advances in Neural Information Processing Systems 20 (NIPS '07), MIT Press, p. 121-128, 2008.
18. Blei D.M., Lafferty J.D., "Topic models", in Srivastava A., Sahami M. (eds), Text Mining: Theory and Applications, 2009.
19. Blei D., Griffiths T., Jordan M., "The nested Chinese restaurant process and Bayesian nonparametric inference of topic hierarchies", Journal of the Association for Computing Machinery, vol. 57, no. 2, p. 1-30, 2010.
20. Bosch A., Zisserman A., Munoz X., "Scene classification via pLSA", in Proceedings of the European Conference on Computer Vision, 2006.
21. Buntine W., "Variational extensions to EM and multinomial PCA", in Proceedings of ECML '02, vol. 2430 of LNAI, p. 23-34, 2002.
22. Buntine W., Jakulin A., "Applying discrete PCA in data analysis", in Proceedings of 20th Conference on Uncertainty in Artificial Intelligence, p. 47-52, 2004.
23. Buntine W., Jakulin A., "Discrete component analysis", in Proceedings of SLSFS '05, vol. 3940 of LNCS, p. 1-33, 2006.
24. Buntine W., "Estimating likelihoods for topic models", in Proceedings of ACML '09, vol. 5828 of LNAI, p. 51-64, 2009.
25. Canny J., "GaP: a factor model for discrete data", in Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '04), 2004.
26. Chandalia G., Beal M., "Using Fisher kernels from topic models for dimensionality reduction", in Proceedings of NIPS 2006 Workshop on Novel Applications of Dimensionality Reduction, 2006.
27. Chapelle O., Zien A., Schölkopf B. (eds), Semi-Supervised Learning, MIT Press, 2006.
28. Chappelier J.C., Eckard E., "An ad hoc information retrieval perspective on PLSI through language model identification", in Azzopardi L., et al. (eds), Advances in Information Retrieval Theory (Proceedings of ICTIR '09), vol. 5766 of Lecture Notes in Computer Science, Springer-Verlag, p. 346-349, 2009.
29. Chappelier J.C., Eckard E., "PLSI: The true Fisher kernel and beyond - IID processes, information matrix and model identification in PLSI", in Buntine W., et al. (eds), Machine Learning and Knowledge Discovery in Databases (Proceedings ECMLPKDD '09), vol. 5781 of Lecture Notes in Computer Science, Springer-Verlag, p. 195-210, 2009.
30. Chappelier J.C., Eckard E., "Utilisation de PLSI en recherche documentaire", in Actes 16ème Conférence sur le Traitement Automatique des Langues Naturelles (TALN 09), ATALA, 2009.
31. Church K.W., Gale W.A., "Poisson mixtures", Natural Language Engineering, vol. 1, p. 163-190, 1995.
32. Clinchant S., Gaussier E., "The beta-negative binomial for text modeling", in Advances in Information Retrieval (Proceedings of 30th European Conference on Information Retrieval), vol. 4956 of Lecture Notes in Computer Science, Springer, p. 150-161, 2008.
33. Cohn D., Chang H., "Probabilistically identifying authoritative documents", in Proceedings of 17th International Conference on Machine Learning (ICML '00), p. 167-174, 2000.
34. Dickey J., "Multiple hypergeometric functions: probabilistic interpretations and statistical uses", Journal of the American Statistical Association, vol. 78, p. 628-637, 1983.
35. Doyle G., Elkan C., "Accounting for burstiness in topic models", in Proceedings of the 26th Annual International Conference on Machine Learning (ICML '09), p. 281-288, 2009.
36. Elkan C., "Deriving TF-IDF as a Fisher kernel", in Consens M., Navarro G. (eds), Proceedings of String Processing and Information Retrieval (SPIRE '05), vol. 3772 of Lecture Notes in Computer Science, Chapter 33, p. 295-300, Springer, 2005.
37. Elkan C., "Clustering documents with an exponential-family approximation of the Dirichlet compound multinomial distribution", in Proceedings of the 23rd International Conference on Machine Learning (ICML '06), p. 289-296, 2006.
38. de Finetti B., "Funzione caratteristica di un fenomeno aleatorio", Atti della R. Academia Nazionale dei Lincei, Serie 6: Memorie, Classe di Scienze Fisiche, Mathematice e Naturale, vol. 4, p. 251-299, 1931.
39. Flaherty P., Giaever G., Kumm J., Jordan M.I., Arkin A.P., "A latent variable model for chemogenomic profiling", Bioinformatics, vol. 21, no. 15, p. 3286-3293, Oxford University Press, August 2005.
40. Gaussier E., Goutte C., Popat K., Chen F., "A hierarchical model for clustering and categorising documents", in Proceedings of 24th BCS-IRSG European Collaboration on IR Research, p. 229-247, 2002.
41. Gaussier E., Goutte C., "Relation between PLSA and NMF and implications", in Proceedings of 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 601-602, 2005.
42. Girolami M., Kabán A., "On an equivalence between PLSI and LDA", in Proceedings of 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, p. 433-434, 2003.
43. Gomes R., Welling M., Perona P., "Memory bounded inference in topic models", in McCallum A., Roweis S. (eds), Proceedings 25th International Conference on Machine Learning (ICML '08), Omnipress, p. 344-351, 2008.
44. Griffiths T.L., Steyvers M., "A probabilistic approach to semantic representation", in Proceedings of the 24th Annual Conference of the Cognitive Science Society, 2002.
45. Griffiths T., Steyvers M., "Finding scientific topics", Proceedings of the National Academy of Sciences, vol. 101 (suppl. 1), p. 5228-5235, 2004.
46. Griffiths T.L., Steyvers M., Blei D.M.I., Tenenbaum J.B., "Integrating topics and syntax", in Proceedings of Advances in Neural Information Processing Systems 17, 2005.
47. Harman D., "Overview of the fourth text retrieval conference (TREC-4)", in Proceedings of the 4th Text Retrieval Conference, p. 1-23, 1995.
48. Heinrich G., Kindermann J., Lauth C., Paaß G., Sanchez-Monzon J., "Investigating word correlation at different scopes - a latent topic approach", in Workshop Learning and Extending Lexical Ontologies at 22nd International Conference on Machine Learning, 2005.
49. Heinrich G., "A generic approach to topic models", in Buntine W., et al. (eds), Machine Learning and Knowledge Discovery in Databases (Proceedings ECML-PKDD '09), vol. 5781 of Lecture Notes in Computer Science, Springer-Verlag, p. 517-532, 2009.
50. Heinrich G., Parameter Estimation for Text Analysis (v. 2.9), Report, Fraunhofer Institute for Computer Graphics Research, Darmstadt, Germany, 2009.
51. Hewitt E., Savage L., "Symmetric measures on Cartesian products", Transactions of the American Mathematical Society, vol. 80, no. 2, p. 470-501, 1955.
52. Hofmann T., "Probabilistic latent semantic indexing", in Proceedings of 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Berkeley, California, p. 50-57, 1999.
53. Hofmann T., "Learning the similarity of documents: an information-geometric approach to document retrieval and categorization", in Advances in Neural Information Processing Systems, vol. 12, p. 914-920, 2000.
54. Hofmann T., "Unsupervised learning by probabilistic latent semantic analysis", Machine Learning, vol. 42, no. 1, p. 177-196, 2001.
55. Hyvärinen A., Karhunen J., Oja E., Independent Component Analysis, John Wiley & Sons, 2001.
56. Jaakkola T., Haussler D., "Exploiting generative models in discriminative classifiers", in Advances in Neural Information Processing Systems, vol. 11, MIT Press, p. 487-493, 1999.
57. Jensen F.V., Bayesian Networks and Decision Graphs, Springer, 2001.
58. Jin X., Zhou Y., Mobasher B., "Web usage mining based on probabilistic latent semantic analysis", in Proceedings of 10th International Conference on Knowledge Discovery and Data Mining, p. 197-205, 2004.
59. Jordan M., Ghahramani Y., Jaakkola T., Saul L., "Introduction to variational methods for graphical models", Machine Learning, vol. 37, p. 183-233, 1999.
60. Katz S.M., "Distribution of content words and phrases in text and language modelling", Natural Language Engineering, vol. 2, p. 15-59, 1996.
61. Koller D., Friedman N., Probabilistic Graphical Models: Principles and Techniques, MIT Press, 2009.
62. Lacoste-Julien S., Sha F., Jordan M., "DiscLDA: discriminative learning for dimensionality reduction and classification", in Koller D., Schuurmans D., Bengio Y., Bottou L. (eds), Advances in Neural Information Processing Systems 21 (NIPS '08), p. 897-904, 2009.
63. Lafferty J., Zhai C., "Document language models, query models, and risk minimization for information retrieval", in Proceedings of 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '01), p. 111-119, 2001.
64. Lee D.D., Seung H.S., "Algorithms for non-negative matrix factorization", Nature, vol. 401, p. 788-791, 1999.
65. Lewis D., "Naive (bayes) at forty: the independence assumption in information retrieval", in Proceedings of 10th European Conference on Machine Learning (ECML '98), vol. 1398 of Lecture Notes in Computer Science, Springer, p. 4-15, 1998.
66. Li W., McCallum A., "Pachinko allocation: DAG-structured mixture models of topic correlations", in Proceedings of 23rd International Conference on Machine Learning (ICML '06), p. 577-584, 2006.
67. Lienhart R., Slaney M., "PLSA on large-scale image databases", in Proceedings of the 2007 International Conference on Acoustics, Speech and Signal Processing, IEEE, (ICASSP 2007), vol. 4, p. 1217-1220, 2007.
68. Lin J., "Divergence measures based on Shannon entropy", IEEE Transactions on Information Theory, vol. 37, no. 14, p. 145-151, 1991.
69. Madsen R.E., Kauchak D., Elkan C., "Modeling word burstiness using the Dirichlet distribution", in Proceedings of the 22nd International Conference on Machine Learning (ICML '05), p. 545-552, 2005.
70. McCallum A., Nigam K., "A comparison of event models for naive bayes text classification", in Proceedings of AAAI-98 Workshop on Learning for Text Categorization, p. 137-142, 1998.
71. McCallum A., Pal C., Druck G., Wang X., "Multi-conditional learning: generative/discriminative training for clustering and classification", in Proceedings of AAAI '06, 2006.
72. McCallum A., Wang X., Corrada-Emmanuel A., "Topic and role discovery in social networks with experiments on enron and academic email", Journal of Artificial Intelligence Research, vol. 30, no. 1, p. 249-272, AI Access Foundation, 2007.
73. Mei Q., Zhai C., "A mixture model for contextual text mining", in Proceedings of 12th International Conference on Knowledge Discovery and Data Mining, p. 649-655, 2006.
74. Meilǎ M., "Comparing clusterings-an information based distance", Journal of Multivariate Analysis, vol. 98, no. 5, p. 873-895, 2007.
75. Minka T., "Estimating a Dirichlet distribution", available on the Internet (revision 2009), 2000.
76. Minka T., Lafferty J., "Expectation-propagation for the generative aspect model", in Proceedings of the 18th Conference on Uncertainty in Artificial Intelligence, p. 352-359, 2002.
77. Misra H., Cappé O., Yvon F., "Using LDA to detect semantically incoherent documents", in Proceedings 12th Conference Computational Natural Language Learning (CONLL '08), p. 41-48, 2008.
78. Monay F., Gatica-Perez D., "PLSA-based image auto-annotation: constraining the latent space", in Proceedings of ACM International Conference on Multimedia (ACM MM), 2004.
79. Monay F., Gatica-Perez D., "Modeling semantic aspects for cross-media image indexing", IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, 2007.
80. Nallapati R., Ahmed A., Xing E.P., Cohen W.W., "Joint latent topic models for text and citations", in Proceedings of KDD '08, 2008.
81. Newman D., Asuncion A.U., Smyth P., Welling M., "Distributed algorithms for topic models", Journal of Machine Learning Research, vol. 10, p. 1801-1828, 2009.
82. Nigam K., McCallum A.K., Thrun S., Mitchell T., "Text classification from labeled and unlabeled documents using EM", Machine Learning, vol. 39, p. 103-134, 2000.
83. Nyffenegger M., Chappelier J.C., Gaussier E., "Revisiting Fisher kernels for document similarities", in Proceedings of 17th European Conference on Machine Learning, p. 727-734, 2006.
84. Ponte J.M., Croft W.B., "A language modeling approach to information retrieval", in 21st SIGIR Conference on Research and Development in Information Retrieval, p. 275-281, 1998.
85. Popescul A., Ungar L.H., Pennock D.M., Lawrence S., "Probabilistic models for unified collaborative and content-based recommendation in sparse-data environments", in Proceedings of the 17th Conference in Uncertainty in Artificial Intelligence, p. 437-444, 2001.
86. Pritchard J.K., Stephens M., Donnelly P., "Inference of population structure unsing multilocus genotype data", Genetics, vol. 155, p. 945-959, 2000.
87. Quelhas P., Monay F., Odobez J.M., Gatica-Perez D., Tuytelaars T., Gool L.V., "Modeling scenes with local descriptors and latent aspects", in Proceedings of ICCV 2005, vol. 1, p. 883-890, 2005.
88. Ramage D., Hall D., Nallapati R., Manning C.D., "Labeled LDA: a supervised topic model for credit attribution in multi-labeled corpora", in Proceedings of 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP '09), Association for Computational Linguistics, p. 248-256, 2009.
89. Rennie J.D.M., Shih L., Teevan J., Karger D.R., "Tackling the poor assumption of naive bayes text classifiers", in Proceedings of ICML '03, p. 616-623, 2003.
90. Robert C.P., Casella G., Monte Carlo Statistical Methods, Springer-Verlag, 1999.
91. Rosen-Zvi M., Griffiths T., Steyvers M., Smyth P., "The author-topic model for authors and documents", in Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence (UAI '04), p. 487-494, 2004.
92. Sebastiani F., "Machine learning in automated text categorization", ACM Computing Surveys, vol. 34, no. 1, p. 1-47, 2002.
93. Steyvers M., Smyth P., Rosen-Zvi M., Griffiths T., "Probabilistic author-topic models for information discovery", in Proceedings of 10th International Conference on Knowledge Discovery and Data Mining, p. 306-315, 2004.
94. Steyvers M., Griffiths T., "Probabilistic topic models", in Landauer T., McNamara D., Dennis S., Kintsch W. (eds), Handbook of Latent Semantic Analysis, Chapter 21, p. 427-448, Laurence Erlbaum, 2007.
95. Sun Q., Li R., Luo D., Wu X., "Text segmentation with LDA-based Fisher kernel", in Proceedings of 46th Annual Meeting of the ACL on Human Language Technologies (HLT '08), ACL, p. 269-272, 2008.
96. Teh Y.W., Jordan M.I., Beal M.J., Blei D.M., "Hierarchical dirichlet processes", Journal of the American Statistical Association, vol. 101, no. 476, p. 1566-1581, 2006.
97. Teh Y.W., Newman D., Welling M., "A collapsed variational bayesian inference algorithm for latent Dirichlet allocation", in Advances in Neural Information Processing Systems 19 (NIPS '06), p. 1353-1360, 2007.
98. Vinokourov A., Girolami M., "A probabilistic framework for the hierarchic organisation and classification of document collections", Journal of Intelligent Information Systems, vol. 18, no. 2-3, p. 153-172, 2002.
99. Wallach H., "Topic modeling: beyond bag-of-words", in Proceedings of 23rd International Conference on Machine Learning (ICML '06), p. 977-984, 2006.
100. Wallach H.M., Murray I., Salakhutdinov R., Mimno D., "Evaluation methods for topic models", in Proceedings of 26th Annual International Conference on Machine Learning (ICML '09), ACM, p. 1105-1112, 2009.
101. Wallach H., Mimno D., McCallum A., "Rethinking LDA: why priors matter", in Bengio Y., Schuurmans D., Lafferty J., Williams C.K.I., Culotta A. (eds), Advances in Neural Information Processing Systems 22 (NIPS '09), p. 1973-1981, 2010.
102. Wei X., Croft W.B., "LDA-based document models for ad-hoc retrieval", in Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '06), p. 178-185, 2006.
103. Welling M., Chemudugunta C., Sutter N., "Deterministic latent variable models and their pitfalls", in Proceedings of SIAM Conference on Data Mining (SDM '08), 2008.
104. Zhai C., Lafferty J., "Model-based feedback in the language modeling approach to information retrieval", in Proceedings of 10th International Conference on Information and Knowledge Management (CIKM), p. 403-410, 2001.
105. Zhai C., Lafferty J., "A study of smoothing methods for language models applied to information retrieval", ACM Transactions on Information Systems, vol. 22, no. 2, p. 179-214, 2004.
106. Zhai C., "Statistical language models for information retrieval a critical review", Foundations and Trends in Information Retrieval, vol. 2, no. 3, p. 137-213, 2008.
107. Zhai C., Statistical Language Models for Information Retrieval, Synthesis Lectures on Human Language Technologies, Morgan & Claypool, 2009.
108. Zhong S., Ghosh J., "Generative model-based document clustering: a comparative study", Knowledge and Information Systems, vol. 8, no. 3, p. 374-384, Springer-Verlag, 2005.
109. Zhu X., Goldberg A., Introduction to Semi-Supervised Learning, Morgan & Claypool, 2009.