Large-scale bayesian logistic regression for text categorization

Technometrics

Volumn 49, Issue 3, 2007, Pages 291-304

  Genkin, Alexander ^a   Lewis, David D ^b   Madigan, David ^c  

^a Center for Discrete Mathematics and Theoretical Computer Science

^b David D Lewis Consulting   (United States)

^c RUTGERS UNIVERSITY   (United States)

Author keywords

Information retrieval; Lasso; Penalization; Ridge regression; Support vector classifier; Variable selection

Indexed keywords

CLASSIFICATION (OF INFORMATION); FEATURE EXTRACTION; MATHEMATICAL MODELS; MAXIMUM LIKELIHOOD ESTIMATION; NATURAL LANGUAGE PROCESSING SYSTEMS; REGRESSION ANALYSIS; SUPPORT VECTOR MACHINES;

PREDICTIVE MODELS; RIDGE REGRESSION; SUPPORT VECTOR CLASSIFIER; VARIABLE SELECTION;

INFORMATION RETRIEVAL;

EID: 34548105186     PISSN: 00401706     EISSN: None     Source Type: Journal    
DOI: 10.1198/004017007000000245     Document Type: Article

References (65)

1. Ackham, P. J. (2004), "An Algorithm for Computing the Inverse Normal Cumulative Distribution Function," available at http://home.online. no/∼pjacklam/notes/invnorm/.
2. Berry, M. W. (ed.) (2004), Survey of Text Mining: Clustering, Classification, and Retrieval, New York: Springer-Verlag.
3. Dayanik, A., Lewis, D., Madigan, D., Menkov, V., and Genkin, A. (2006), "Constructing Informative Prior Distributions From Domain Knowledge in Text Classification," in Proceedings of SIGIR 2006: The Twenty-Ninth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New York: ACM, pp. 493-500.
4. Dennis, J. E., Jr., and Schnabel, R. B. (1989), "A View of Unconstrained Optimization," in Optimization, eds. G. L. Nemhauser, A. H. G. Rinnooy Kan, and M. J. Todd, Amsterdam: Elsevier, pp. 1-72.
5. Efron, B., Hastie, T., Johnstone, I., and Tibshirani, R. (2004), "Least Angle Regression," The Annals of Statistics, 32, 407-499.
6. Figueiredo, M. A. T. (2003), "Adaptive Sparseness for Supervised Learning," IEEE Transactions on Pattern Analysis and Machine Intelligence, 25, 1150-1159.
7. Figueiredo, M. A. T., and Jain, A. K. (2001), "Bayesian Learning of Sparse Classifiers," in Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 1, pp. 1-35-1-41.
8. Forman, G. (2003), "An Extensive Empirical Study of Feature Selection Metrics for Text Classification," Journal of Machine Learning Research, 3, 1289-1305.
9. Girosi, F. (1998), "An Equivalence Between Sparse Approximation and Support Vector Machines," Neural Computation, 10, 1455-1480.
10. Greenland, S., Schwartzbaum, J. A., and Finkle, W. D. (2000), "Problems From Small Samples and Sparse Data in Conditional Logistic Regression Analysis," American Journal of Epidemiology, 151, 531-539.
11. Hadjicostas, P. (2003), "Consistency of Logistic Regression Coefficient Estimates Calculated From a Training Sample," Statistics and Probability Letters, 62, 293-303.
12. Hastie, T. J., and Pregibon, D. (1992), "Generalized Linear Models," in Statistical Models in S, eds. J. M. Chambers and T. J. Hastie, Pacific Grove, CA: Wadsworth & Brooks, pp. 377-420.
13. Hastie, T., Tibshirani, R., and Friedman, J. (2001), The Elements of Statistical Learning: Data Mining, Inference and Prediction, New York: Springer.
14. Hauben, M., Madigan, D., Gerrits, C., and Meyboom, R. (2005), "The Role of Data Mining in Pharmacovigilence," Expert Opinion in Drug Safety, 4, 929-948.
15. Hersh, W., Buckley, C., Leone, T. J., and Hickman, D. (1994), "OHSUMED: An Interactive Retrieval Evaluation and New Large Test Collection for Research," in SIGlR '94: Proceedings of the Seventeenth Annual InternationalACM-SIGIR Conference on Research and Development in Information Retrieval, New York: ACM, pp. 192-201.
16. Hoerl, A. E., and Kennard, R. W. (1970), "Ridge Regression: Biased Estimation for Nonorthogonal Problems," Technometrics, 12, 55-67.
17. Jin, R., Yan, R., Zhang, J., and Hauptmann, A. (2003), "A Faster Iterative Scaling Algorithm for Conditional Exponential Models," in Proceedings of the Twentieth International Conference on Machine Learning (ICML-2003), Menlo Park: AAAI, pp. 282-289.
18. Joachims, T. (1998), "Text Categorization With Support Vector Machines: Learning With Many Relevant Features," in Machine Learning: ECML '98, 10th European Conference on Machine Learning, Heildelberg: Springer, pp. 137-142.
19. _ (2002), Learning to Classify Text Using Support Vector Machines, Boston: Kluwer.
20. Jurafsky, D., and Martin, J. H. (2000), Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition, Englewood Cliffs, NJ: Prentice-Hall.
21. Kittler, J. (1986), "Feature Selection and Extraction," in Handbook of Pattern Recognition and Image Processing, eds. T. Y. Young and K.-S. Fu, Orlando, FL: Academic Press, pp. 59-83.
22. Kivinen, J., and Warmuth, M. K. (2001), "Relative Loss Bounds for Multidimensional Regression Problems," Machine Learning, 45, 301-329.
23. Komarek, P., and Moore, A. (2003), "Fast Robust Logistic Regression for Large Sparse Datasets With Binary Outputs," in Proceedings of the Ninth International Workshop on Artificial Intelligence and Statistics, pp. 197-204.
24. Krishnapuram, B., Hartemink, A. J., Carin, L., and Figueiredo, M. A. T. (2005), "Sparse Multinomial Logistic Regression: Fast Algorithms and Generalization Bounds," IEEE Transactions on Pattern Analysis and Machine Intelligence, 27, 957-968.
25. le Cessie, S., and van Houwelingen, J. C. (1997), "Ridge Estimators in Logistic Regression," Applied Statistics, 41, 191-201.
26. Li, F., and Yang, Y. (2004), "Recovering Genetic Regulatory Networks From Micro-Array Data and Location Analysis Data," Genome Informatics, 15, 131-140.
27. Lewis, D. D. (1995), "Evaluating and Optimizing Autonomous Text Classification Systems," in SIGIR'95: Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New York: ACM, pp. 246-254.
28. _ (1998), "Naive (Bayes) at Forty: The Independence Assumption in Information Retrieval," in Machine Learning: ECML'98, 10th European Conference on Machine Learning, Heildelberg: Springer, pp. 4-15.
29. _ (2004), "Reuters-21578 Text Categorization Test Collection: Distribution 1.0 README File (v 1.3)," available at http://www. daviddlewis.com/ resources/testcollections/reuters21578/readme.txt.
30. Lewis, D. D., Schapire, R. E., Callan, J. P., and Papka, R. (1996), "Training Algorithms for Linear Text Classifiers," in Proceedings of SIGIR-96, 19th ACM International Conference on Research and Development in Information Retrieval, New York: ACM, pp. 298-306.
31. Lewis, D. D., Yang, Y., Rose, T., and Li, F. (2004), "RCV1: A New Benchmark Collection for Text Categorization Research," Journal of Machine Learning Research, 5, 361-397. Data set available at http://jmlr.csail.mit.edu/ papers/volume5/lewis04a/lyrl2004_rcv1v2_README. htm.
32. Lowe, H. J., and Barnett, G. O. (1994), "Understanding and Using the Medical Subject Headings (MeSH) Vocabulary to Perform Literature Searches," Journal of the American Medical Association, 271, 1103-1108.
33. Luenberger, D. G. (1984), Linear and Nonlinear Programming (2nd ed.), Reading, MA: Addison-Wesley.
34. Madigan, D. (2005), "Statistics and the War on Spam," in Statistics: A Guide to the Unknown, eds. R. Peck, G. Casella, G. Cobb, R. Hoerl, D. Nolan, R. Starbuck, and H. Stern, Duxbury Press, pp. 135-147.
35. Madigan, D., and Ridgeway, G. (2004), Discussion of "Least Angle Regression," by B. Efron, T. Hastie, I. Johnstone, and R. Tibshirani, The Annals of Statistics, 32, 465-469.
36. Madigan, D., Genkin, A., Lewis, D. D., Argamon, S., Fradkin, D., and Ye, L. (2005a), "Author Identification on the Large Scale," in Proceedings of the CSNA & INTERFACE Annual Meetings, CD-ROM.
37. Madigan, D., Genkin, A., Lewis, D. D., and Fradkin, D. (2005b), "Bayesian Multinomial Logistic Regression for Author Identification," in Bayesian Analysis and Maximum Entropy Methods in Science and Engineering: 25th International Workshop on Bayesian Inference and Maximum Entropy Methods in Science and Engineering, AIP Conference Proceedings, Vol. 803, Melville, NY: AIP, pp. 509-516.
38. Mallick, B. K., Ghosh, D., and Ghosh, M. (2005), "Bayesian Classification of Tumors Using Gene Expression Data," Journal of the Royal Statistical Society, Ser. B, 67, 219-234.
39. Malouf, R. (2002), "A Comparison of Algorithms for Maximum Entropy Parameter Estimation," in Proceedings of the Sixth Conference on Natural Language Learning (CoNLL-2002), pp. 49-55.
40. Manning, C., and Schütze, H. (1999), Foundations of Statistical Natural Language Processing, Cambridge, MA: MIT Press.
41. Maron, M. E. (1961), "Automatic Indexing: An Experimental Inquiry," Journal of the ACM, 8, 404-417. Meinshausen, N. (2005), "Lasso With Relaxation," technical report, University of California Berkeley, Dept. of Statistics.
42. Mitchell, T. J., and Beauchamp, J. J. (1988), "Bayesian Variable Selection in Linear Regression," Journal of the American Statistical Association, 83, 1023-1032.
43. Mosteller, F., and Wallace, D. L. (1964), Inference and Disputed Authorship: The Federalist, Addison-Wesley.
44. Park, M.-Y, and Hastie, T. (2006), "An L1 Regularization-Path Algorithm for Generalized Linear Models," available at http://www-stat. stanford.edu/∼" hastie/Papers/glmpath.pdf.
45. Pike, M. C., Hill, A. P., and Smith, P. G. (1980), "Bias and Efficiency in Logistic Analysis of Stratified Case-Control Studies," American Journal of Epidemiology, 9, 89-95.
46. Porter, M. F. (1980), "An Algorithm for Suffix Stripping," Program, 14 (3), 130-137.
47. _ (2003), "The Porter Stemming Algorithm," available at http://www.tartarus,org/∼martin/PorterStemmer/index.html.
48. Press, W. H., Teukolsky, S. A., Vetterling, W. T., and Flannery, B. P. (1992), Numerical Recipes in C: The Art of Scientific Computing (2nd ed.), Cambridge, U.K.: Cambridge University Press.
49. Rocchio, J. J., Jr. (1971), "Relevance Feedback Information Retrieval," in The SMART Retrieval System: Experiments in Automatic Document Processing, ed. G. Salton, Englewood Cliffs, NJ: Prentice-Hall, pp. 313-323.
50. Salton, G., and Buckley, C. (1988), "Term-Weighting Approaches in Automatic Text Retrieval," Information Processing and Management, 24, 513-523.
51. Santner, T., and Duffy, D. (1989), The Statistical Analysis of Discrete Data, New York: Springer-Verlag.
52. Schapire, R. E., and Singer, Y. (2000), "BoosTexter: A Boosting-Based System for Text Categorization," Machine Learning, 39, 135-168.
53. Sebastiani, F. (2002), "Machine Learning in Automated Text Categorization," ACM Computing Surveys, 34, 1-47.
54. Shevade, S. K., and Keerthi, S. S. (2003), "A Simple and Efficient Algorithm for Gene Selection Using Sparse Logistic Regression," Bioinformatcs, 19, 2246-2253.
55. Smith, R. L. (1999), "Bayesian and Frequentist Approaches to Parametric Predictive Inference" (with discussion), in Bayesian Statistics 6, eds. J. M. Bernardo, J. O. Berger, A. P. Dawid, and A. F. M. Smith, Oxford, U.K.: Oxford University Press, pp. 589-612.
56. Tibshirani, R. (1996), "Regression Shrinkage and Selection via the Lasso," Journal of the Royal Statistical Society, Ser. B, 58, 267-288.
57. Tipping, M. E. (2001), "Sparse Bayesian Learning and the Relevance Vector Machine," Journal of Machine Learning Research, 1, 211-244. Rijsbergen
58. van Rijsbergen, C. J. (1972), "Automatic Information Structuring and Retrieval," unpublished doctoral thesis, King's College, Cambridge.
59. _ (1979), Information Retrieval (2nd ed.), London: Butterworths.
60. Weiss, S. M., Indurkhya, N., Zhang, T., and Damerau, F. J. (2005), Text Mining: Predictive Methods for Analyzing Unstructured Information, New York: Springer.
61. Yang, Y., and Pedersen, J. O. (1997), "A Comparative Study on Feature Selection in Text Categorization," in Proceedings of ICML-97, 14th International Conference on Machine Learning ICML97, San Francisco: Morgan Kaufmann, pp. 412-420.
62. Zhang, J., and Yang, Y. (2003), "Robustness of Regularized Linear Classification Methods in Text Categorization," in Proceedings of SIGIR 2003: The Twenty-Sixth Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, New York: ACM, pp. 190-197.
63. Zhang, T., and Oles, F. (2001), "Text Categorization Based on Regularized Linear Classifiers," Information Retrieval, 4, 5-31.
64. Zhao, P., and Yu, B. (2006), "On Model Selection Consistency of Lasso," Journal of Machine Learning Research, 7, 2541-2567.
65. Zou, H., and Hastie, T. (2005), "Regularization and Variable Selection via the Elastic Net," Journal of the Royal Statistical Society, Ser. B, 67, 301-320.