Surrogate maximization/minimization algorithms and extensions

Machine Learning

Volumn 69, Issue 1, 2007, Pages 1-33

  Zhang, Zhihua ^a   Kwok, James T ^b   Yeung, Dit Yan ^b  

^a UNIVERSITY OF CALIFORNIA   (United States)

^b HONG KONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (Hong Kong)

Author keywords

AdaBoost; Convexity; Log linear model; Logistic regression; Surrogate function

Indexed keywords

APPROXIMATION THEORY; CONVERGENCE OF NUMERICAL METHODS; MATHEMATICAL MODELS; REGRESSION ANALYSIS; TAYLOR SERIES;

ADABOOST; LOG LINEAR MODELS; LOGISTIC REGRESSION MODELS; SURROGATE MAXIMIZATION ALGORITHMS; SURROGATE MINIMIZATION ALGORITHMS;

ALGORITHMS;

EID: 35148838927     PISSN: 08856125     EISSN: 15730565     Source Type: Journal    
DOI: 10.1007/s10994-007-5022-x     Document Type: Article

References (35)

1. Becker, M. P., Yang, I., & Lange, K. (1997). EM algorithms without missing data. Statistical Methods in Medical Research, 6, 38-54.
2. Böhning, D., & Lindsay, B. G. (1988). Monotonicity of quadratic-approximation algorithms. Annals of the Institute of Statistical Mathematics, 40(4), 641-663.
3. Borg, I., & Groenen, P. (1997). Modern multidimensional scaling. New York: Springer.
4. Boyd, S., & Vandenberghe, L. (2004). Convex optimization. Cambridge: Cambridge University Press.
5. Collins, M., Schapire, R. E., & Singer, Y. (2002). Logistic regression, AdaBoost and Bregman distances. Machine Learning, 47(2-3), 253-285.
6. Craven, M., Dopasquo, D., Freitag, D., McCallum, A., Mitchell, T., Nigam, K., & Slattery, S. (1998). Learning to extract symbolic knowledge from the World Web Wide. In The fifteenth national conference on artificial intelligence.
7. Darroch, J. N., & Ratcliff, D. (1972). Generalized iterative scaling for log-linear models. The Annals of Mathematical Statistics, 43(5), 1470-1480.
8. De Leeuw, J. (1994). Block relaxation algorithms in statistics. In H. H. Bock, W. Lenski, & M. M. Richter (Eds.), Information systems and data analysis (pp. 308-325). Berlin: Springer.
9. Della Pietra, S., Della Pietra, V., & Lafferty, J. (1997). Inducing features of random fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(4), 380-393.
10. Della Pietra, S., Della Pietra, V., & Lafferty, J. (2001). Duality and auxiliary functions for Bregman distances (Technical Report CMU-CS-01-109), School of Computer Science, CMU.
11. Dempster, A. P., Laird, N. M., & Rubin, D. B. (1977). Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society Series B, 39(1), 1-38.
12. De Pierro, A. R. (1995). A modified expectation maximization algorithm for penalized likelihood estimation in emission tomography. IEEE Transactions on Medical Imaging, 14(1), 132-137.
13. Edwards, D., & Lauritzen, S. L. (2001). The TM for maximising a conditional likelihood function. Biometrika, 88(4), 961-972.
14. Fletcher, R. (1987). Practical methods of optimization (2nd ed.). New York: Wiley.
15. Freund, Y., & Schapire, R. E. (1997). A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences, 55(1), 119-139.
16. Friedman, J. H., Hastie, T., & Tibshirani, R. (2000). Additive logistic regression: a statistical view of boosting. Annals of Statistics, 28(2), 337-374.
17. Jaakkola, T., & Jordan, M. (1997). A variational approach to Bayesian logistic regression models and their extensions. In The sixth international workshop on artificial intelligence and statistics.
18. Jebara, T., & Pentland, A. (1999). Maximum conditional likelihood via bound maximization and the CEM algorithm. In Advances in neural information processing systems (Vol. 11).
19. Joachims, T. (1997). A probabilistic analysis of the Rocchio algorithm with TFIDF for text categorization. In The fourteenth international conference on machine learning (pp. 143-151). San Francisco: Kaufmann.
20. Kivinen, J., & Warmuth, M. K. (1997). Boosting as entropy projection. In The twelfth annual conference on computational learning theory (pp. 134-144).
21. Lafferty, J. (1999). Additive models, boosting and inference for generalized divergences. In The twelfth annual conference on computational learning theory (pp. 125-133).
22. Lange, K. (1995). A gradient algorithm locally equivalent to the EM algorithm. Journal of the Royal Statistical Society Series B, 57(2), 425-437.
23. Lange, K., Hunter, D. R., & Yang, I. (2000). Optimization transfer using surrogate objective functions with discussion. Journal of Computational and Graphical Statistics, 9(1), 1-59.
24. Lebanon, G., & Lafferty, J. (2001). Boosting and maximum likelihood for exponential models (Technical Report CMU-CS-01-144), School of Computer Science, Carnegie Mellon University.
25. Liu, C., & Rubin, D. B. (1994). The ECME algorithm: a simple extension of EM and ECM with faster monotone convergence. Bionmetrika, 84(4), 633-648.
26. Meng, X.-L. (2000). Discussion on "optimization transfer using surrogate objective functions". Journal of Computational and Graphical Statistics, 9(1), 35-43.
27. Meng, X.-L., & Rubin, D. B. (1993). Maximum likelihood estimation via the ECM algorithm: a general framework. Bionmetrika, 80(2), 267-278.
28. Minka, T. P. (2003). A comparison of numerical optimizers for logistic regression (Technical report). Available from http://www.stat.cmu.edu/~minka/ papers/logreg/.
29. Ostrowski, A. M. (1960). Solution of equations and systems of equations. New York: Academic Press.
30. Rockafellar, T. (1970). Convex analysis. Princeton: Princeton University Press.
31. Salakhutdinov, R., & Roweis, S. (2003). Adaptive overrelazed bound optimization methods. In The 20th international conference on machine learning.
32. Schapire, R. E. (1990). The strength of weak learnability. Machine Learning, 5, 197-227.
33. Schapire, R. E., & Singer, Y. (1999). Improved boosting algorithms using confidence-rated predictions. Machine Learning, 37, 297-336.
34. Wu, C. F. J. (1983). On the convergence properties of the EM algorithm. Annals of Statistics, 11, 95-103.
35. Yuille, A., & Rangarajan, A. (2001). The convex-concave computational procedure (CCCP). In Advances in neural information processing systems (Vol. 13).