Accelerating cyclic update algorithms for parameter estimation by pattern searches

Neural Processing Letters

Volumn 17, Issue 2, 2003, Pages 191-203

  Honkela, Antti ^a   Valpola, Harri ^a   Karhunen, Juha ^a  

^a AALTO UNIVERSITY   (Finland)

Author keywords

Alternating optimization; Bayesian methods; Cyclic updating; Line search; Parameter estimation; Pattern search; Variational Bayesian learning

Indexed keywords

ALGORITHMS; LEARNING SYSTEMS; OPTIMIZATION;

CYCLIC UPDATING;

PARAMETER ESTIMATION;

EID: 0038173234     PISSN: 13704621     EISSN: None     Source Type: Journal    
DOI: 10.1023/A:1023655202546     Document Type: Article

References (20)

1. Attias, H.: Independent factor analysis, Neural Computation, 11(4) (1999), 803-851.
2. Bazaraa, M. S., Sherali, H. D. and Shetty, C. M.: Nonlinear Programming: Theory and Algorithms, J. Wiley, 1993, Second Edition.
3. 3. Bezdek, J. C. and Hathaway, R. J.: Some notes on alternating optimization, In: N. R. Pal and M. Sugeno, (eds), Advances in Soft Computing - AFSS 2002, Vol. 2275 of Lecture Notes in Artificial Intelligence, Springer-Verlag, pp. 288-300, 2002.
4. Bishop, C.: Neural Networks for Pattern Recognition, Clarendon Press, 1995.
5. Dempster, A. P., Laird, N. M. and Rubin, D. B.: Maximum likelihood from incomplete data via the EM algorithm, J. of the Royal Statistical Society, Series B (Methodological), 39(1) (1977), 1-38.
6. Fletcher, R.: Practical Methods of Optimization, J. Wiley, 1987, Second Edition.
7. Ghahramani, Z. and Hinton, G. E.: Variational learning for switching state-space models, Neural Computation, 12(4) (2000), 963-996.
8. Hinton, G. E. and van Camp, D.: Keeping Neural Networks Simple by Minimizing the Description Length of the Weights, In: Proc. of the 6th Ann. ACM Conf. on Computational Learning Theory, Santa Cruz, CA, USA, pp. 5-13, 1993.
9. Hooke, R. and Jeeves, T. A.: "Direct search" solution of numerical and statistical problems, J. of the ACM, 8(2) (1961), 212-229.
10. Hyvärinen, A., Karhunen, J. and Oja, E.: Independent Component Analysis, J. Wiley, 2001.
11. Jamshidian, M. and Jennrich, R. I.: Conjugate gradient acceleration of the EM algorithm, J. of the American Statistical Association, 88(421) (1993), 221-228.
12. Jordan, M., Ghahramani, Z., Jaakkola, T. and Saul, L.: An introduction to variational methods for graphical models, In: M. Jordan, (ed), Learning in Graphical Models, Cambridge, MA, USA: The MIT Press, pp. 105-161, 1999.
13. Lappalainen, H. and Honkela, A.: Bayesian nonlinear independent component analysis by multi-layer perceptrons, In: M. Girolami, (ed), Advances in Independent Component Analysis, Berlin: Springer-Verlag, pp. 93-121, 2000.
14. Lappalainen, H. and Miskin, J.: Ensemble learning, In: M. Girolami, (ed), Advances in Independent Component Analysis, Berlin: Springer-Verlag, pp. 75-92, 2000.
15. MacKay, D. J. C.: Developments in Probabilistic Modelling with Neural Networks -Ensemble Learning, In: Neural Networks; Artificial Intelligence and Industrial Applications. Proc. of the 3rd Annual Symposium on Neural Networks, pp. 191-198, 1995.
16. MacKay, D. J. C.: Ensemble learning for hidden Markov models, Available from http://wol.ra.phy.cam.ac.uk/mackay/1997.
17. Neal, R. M. and Hinton, G. E.: A view of the EM algorithm that justifies incremental, sparse, and other variants, In: M. I. Jordan (ed), Learning in Graphical Models, Cambridge, MA, USA: The MIT Press, pp. 355-368, 1999.
18. Valpola, H. and Karhunen, J.: An unsupervised ensemble learning method for nonlinear dynamic state-space models, Neural Computation, 14(11) (2002), 2647-2692.
19. Valpola, H., Östman, T. and Karhunen, J.: Nonlinear Independent Factor Analysis by Hierarchical Models, In: Proc. 4th Int. Symp. on Independent Component Analysis and Blind Signal Separation (ICA2003), 2003, to appear.
20. Valpola, H., Raiko, T. and Karhunen, J.: Building Blocks for Hierarchical Latent Variable Models, In: Proc. 3rd Int. Conf. on Independent Component Analysis and Signal Separation (ICA2001), San Diego, USA, pp. 710-715, 2001.