Identification of recurrent neural networks by bayesian interrogation techniques

Journal of Machine Learning Research

Volumn 10, Issue , 2009, Pages 515-554

  Póczos, Barnabás ^a   Lorincz, András ^a  

^a EÖTVÖS LORÁND UNIVERSITY   (Hungary)

Author keywords

A optimality; Active learning; D optimality; Infomax control; Online bayesian learning; Optimal design; System identification

Indexed keywords

BAYESIAN NETWORKS; COST FUNCTIONS; DYNAMICAL SYSTEMS; EDUCATION; INDEPENDENT COMPONENT ANALYSIS; INTERNET; LEARNING ALGORITHMS; OPTIMAL SYSTEMS; OPTIMIZATION; RECURRENT NEURAL NETWORKS;

A-OPTIMALITY; ACTIVE LEARNING; D-OPTIMALITY; INFOMAX CONTROL; ONLINE BAYESIAN LEARNING; OPTIMAL DESIGN; SYSTEM IDENTIFICATION;

IDENTIFICATION (CONTROL SYSTEMS);

EID: 61749089973     PISSN: 15324435     EISSN: 15337928     Source Type: Journal    
DOI: None     Document Type: Article

References (77)

1. J. Abonyi, R. Babuska, A. Ayala-Botto, A. Szeifert, and L. Nagy. Identification and control of nonlienar systems using Hammerstein-models. Ind. Eng. Chem. Res, 39:4302-4314, 2000.
2. L. A. Aguirre, M. C. S. Coelho, and M. V. Correa. On the interpretation and practice of dynamical differences between Hammerstein and Wiener models. In IEE. Proc. of Control Theory Application, volume 152, pages 349-354, 2005.
3. S. Amari, A. Cichocki, and H. H. Yang. A new learning algorithm for blind signal separation. In Advances in Neural Information Processing Systems, volume 8, pages 757-763, Cambridge, MA, 1996. MIT Press.
4. B. Anderson and A. Moore. Active learning for hidden Markov models: objective functions and algorithms. In ICML '05: Proceedings of the 22nd international conference on Machine learning, pages 9-16, New York, NY, USA, 2005. ACM.
5. F. R. Bach. Active learning for misspecified generalized linear models. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems, volume 19, pages 65-72, Cambridge, MA, 2007. MIT Press.
6. E. W. Bai. A blind approach to the Hammerstein-Wiener model identification. Automática, 38: 967-979, 2002.
7. M. J. Beal. Variational Algorithms for Approximate Bayesian Inference. PhD thesis, Gatsby Computational Neuroscience Unit, University College London, 2003.
8. J. M. Bernardo. Expected information as expected utility. The Annals of Statistics, 7(3):686-690, 1979.
9. S. A. Billings and I. J. Leontaritis. Identifiaction of nonlinear systems using parametric estimation techniques. In IEE. Proc. of Control and its Applications, pages 183-187, 1981.
10. X. Boyen and D. Koller. Tractable inference for complex stochastic processes. In Fourteenth Conference on Uncertainty in Artificial Intelligence, pages 33-42, 1998.
11. J. F. Cardoso. High-order contrasts for independent component analysis. Neural Computation, 11 (1):157-192, 1999.
12. R. Castro, J. Jaupt, and R. Nowak. Compressed sensing vs. active learning. In ICASSP 2006, IEEE International Conference on Acoustics, Speech and Signal Processing, volume 3, 2006a.
13. R. Castro, R. Willett, and R. Nowak. Faster rates in regression via active learning. In Y. Weiss, B. Schölkopf, and J. Piatt, editors, Advances in Neural Information Processing Systems 18, pages 179-186. MIT Press, Cambridge, MA, 2006b.
14. P. Celka, N. J. Bershad, and J. Vesin. Stochastic gradient identification of polynomial Wiener systems: Analysis and application. IEEE Trans. Signal Process., 49(2):301-313, 2001.
15. K. Chaloner. Optimal bayesian experimental design for linear models. The Annals of Statistics, 12 (1):283-300, 1984.
16. K. Chaloner and I. Verdinelli. Bayesian experimental design: A review. Statist. Sci., 10:273-304, 1995.
17. N. Chiras, C. Evans, and D. Rees. Nonlinear gas turibne modeling using NARMAX structures. IEEE Trans. Instrum. Meas., 50(4):893-898, 2001.
18. D. A. Cohn. Neural network exploration using optimal experiment design. In Advances in Neural Information Processing Systems, volume 6, pages 679-686, 1994.
19. D. A. Cohn, Z. Ghahramani, and M. I. Jordan. Active learning with statistical models. Journal of Artificial Intelligence Research, 4:129-145, 1996.
20. T. F. Coleman and Y. Li. A reflective newton method for minimizing a quadratic function subject to bounds on some of the variables. SIAM Journal on Optimization, 6(4): 1040-1058, 1996.
21. P. Comon. Independent component analysis, a new concept? Signal Processing, 36(3):287-314, 1994.
22. T. M. Cover and J. A. Thomas. Elements of Information Theory. Wiley-Interscience, 1991.
23. R. C. deCharms, D. T. Blake, and M. M. Merzenich. Optimizing sound features for cortical neurons. Science, 280:1439-1444, 1998.
24. G. Duncan and M. H. DeGroot. A mean squared error approach to optimal design theory. In Proceedings of the 1976 Conference on Information: Sciences and Systems, pages 217-221. The Johns Hopkins University, 1976.
25. V. V. Fedorov. Theory of Optimal Experiments. Academic Press, New York, 1972.
26. P. Földiák. Stimulus optimization in primary visual cortex. Neurocomputing, 38-40:1217-1222, 2001.
27. Y. Freund, H. S. Seung, E. Shamir, and N. Tishby. Selective sampling using the query by committee algorithm. Machine Learning, 28:133-168, 1997.
28. K. Fukumizu. Active learning in multilayer perceptrons. In David S. Touretzky, Michael C. Mozer, and Michael E. Hasselmo, editors, Advances in Neural Information Processing Systems, volume 8, pages 295-301. The MIT Press, 1996.
29. K. Fukumizu. Statistical active learning in multilayer perceptrons. IEEE Transactions on Neural Networks, 11(1): 17-26, 2000.
30. M. Gäfvert. Modelling the Furuta pendulum. Technical report ISRN LUTFD2/TFRT-7574-SE, Department of Automatic Control, Lund University, Sweden, April 1998.
31. A. Gelman, J. B. Carlin, H. S. Stern, and D. B. Rubin. Bayesian Data Analysis. CRC Press, 2nd erdition, 2003.
32. Z. Ghahramani. Online variational Bayesian learning, 2000. Slides from talk presented at NIPS 2000 workshop on Online Learning.
33. G. H. Golub and C. F. Van Loan. Matrix Computations. Johns Hopkins, Baltimore, MD, 3rd ed. edition, 1996.
34. A. K. Gupta and D. K. Nagar. Matrix Variate Distributions, volume 104 of Monographs and Surveys in Pure and Applied Mathematics. Chapman and Hall/CRC, 1999.
35. R. Haber and H. Unbehauen. Structure identification of nonlinear dynamic systems - a survey on input/output approaches. Automatica, 26(4):651-677, 1990.
36. D. A. Harville. Matrix Algebra From a Statistician's Perspective. Springer-Verlag, 1997.
37. A. Honkela and H. Valpola. On-line variational Bayesian learning. In 4th International Symposium on Independent Component Analysis and Blind Signal Separation, pages 803-808, 2003.
38. A. Hyvärinen. Fast and robust fixed-point algorithms for independent component analysis. IEEE Trans, on Neural Networks, (10):626-634, 1999.
39. A. Hyvärinen, J. Karhunen, and E. Oja. Independent Component Analysis. John Wiley, New York, 2001. URL http://www.cis.hut.fi/projects/ ica/book/.
40. A. Hyvärinen. Independent component analysis for time-dependent stochastic processes. In Proc. of ICANN'98, International Conference on Artificial Neural Networks, Skövde, Sweden, pages 541-546, 1998.
41. H. Jaeger. Short term memory in echo state networks. GMD Report, 152, Fraunhofer AIS, 2001. http://publica.fraunhofer.de/starweb/pub08/en/index.htm.
42. C. Jutten and J. Hérault. Blind separation of sources: An adaptive algorithm based on neuromimetic architecture. Signal Processing, 24:1-10, 1991.
43. J. Kiefer. Optimum experimental designs. Journal of the Royal Statistical Society, Series B, 21: 272-304, 1959.
44. S. Kotz and S. Nadarajah. Multivariate T-Distributions and Their Applications. Cambridge University Press, 2004.
45. A. Krause and C. Guestrin. Nonmyopic active learning of gaussian processes: an exploration-exploitation approach. In ICML '07: Proceedings of the 24th international conference on Machine learning, pages 449-456, New York, NY, USA, 2007. ACM.
46. J. Lewi, R. Butera, and L. Paninski. Real-time adaptive information-theoretic optimization of neurophysiology experiments. In Advances in Neural Information Processing Systems, volume 19, 2007.
47. D. D. Lewis and J. Catlett. Heterogeneous uncertainty sampling for supervised learning. In William W. Cohen and Haym Hirsh, editors, Proceedings of ICML-94, 11th International Conference on Machine Learning, pages 148-156, New Brunswick, US, 1994. Morgan Kaufmann Publishers, San Francisco, US.
48. D. D. Lewis and W. A. Gale. A sequential algorithm for training text classifiers. In Proceedings of SIGIR-94, 17th ACM International Conference on Research and Development in Information Retrieval, pages 3-12, Dublin, IE, 1994. Springer Verlag, Heidelberg, DE.
49. D. V Lindley. Bayesian Statistics: A Review. SIAM, 1971.
50. W. Maass, T. Natschläger, and H. Markram. Real-time computing without stable states: A new framework for neural computation based on perturbations. Neural Computation, 14:2531-2560, 2002.
51. C. K. Machens, T. Gollisch, O. Kolesnikova, and A. V. M. Herz. Testing the efficiency of sensory coding with optimal stimulus ensembles. Neuron, 47:447-456, 2005.
52. D. J. C. MacKay. Information-based objective functions for active data selection. Neural Computation, 4(4):590-604, 1992.
53. T. Minka. Bayesian linear regression, 2000. MIT Media Lab note.
54. T. Minka. A Family of Algorithms for Approximate Bayesian Inference. PhD thesis, MIT Media Lab, MIT, 2001.
55. M. Opper and O. Winther. A Bayesian approach to online learning. In Online Learning in Neural Networks. Cambridge University Press, 1999.
56. R. K. Pearson and M. Pottmann. Gray-box identification of block-oriented nonlnear models. Journal of Process Control, 10:301-315, 2000.
57. F. Pukelsheim. Optimal Design of Experiments. John Wiley & Sons, 1993.
58. H. Rabitz. Shaped laser pulses as reagents. Science, 299:525-527, 2003.
59. H. Raiffa and R. Schlaifer. Applied Statistical Decision Theory. Boston, MIT Press, 1961.
60. R. Rangarajan, R. Raich, and A. O. Hero. Optimal sequential energy allocation for inverse problems. IEEE Journal of Selected Topics in Signal Processing, 1:67-78, 2007.
61. N. Roy and A. McCallum. Toward optimal active learning through sampling estimation of error reduction. In Proc. 18th International Conference on Machine Learning, pages 441-448. Morgan Kaufmann, San Francisco, CA, 2001.
62. A. I. Schein. Active Learning for Logistic Regression. PhD thesis, University of Pennsylvania, 2005.
63. A. I. Schein and L. H. Ungar. Active learning for logistic regression: an evaluation. Machine Learning, 68(3):235-265, 2007.
64. H. S. Seung, M. Opper, and H. Sompolinsky. Query by committee. In Computational Learning Theory, 1992.
65. S. Solla and O. Winther. Optimal perceptron learning: An online Bayesian approach. In Online Learning in Neural Networks. Cambridge University Press, 1999.
66. D. M. Steinberg and W.G. Hunter. Experimental design: review and comment. Technometrixs, 26: 71-97, 1984.
67. M. Stone. Application of a measure of information to the design and comparison of regression experiments. Ann. Math. Statist, 30(1):55-70, 1959.
68. M. Sugiyama. Active learning in approximately linear regression based on conditional expectation of generalization error. The Journal of Machine Learning Research, 7:141-166, 2006.
69. Z. Szabó and A. Lorincz. Towards independent subspace analysis in controlled dynamical systems. ICA Research Network International Workshop, 2008.
70. B. Toman and J. L. Gastwirth. Robust Bayesian experimental design and estimation for analysis of variance models using a class of normal mixtures. Journal of statistical planning and inference, 35(3):383-398, 1993.
71. S. Tong and D. Koller. Active learning for parameter estimation in Bayesian networks. In Advances in Neural Information Processing Systems, pages 647-653, 2000.
72. S. Tong and D. Koller. Active learning for structure in Bayesian networks. In Proceedings of the International Joint Conference on Artificial Intelligence, 2001a.
73. S. Tong and D. Koller. Support vector machine active learning with applications to text classification. Journal of Machine Learning Research, pages 45-66, 2001b.
74. T. M. Vaughan, W. J. Heetderks, L. J. Trejo, W. Z. Rymer, M. Weinrich, M. M. Moore, A. Kubier, B. H. Dobkin, N. Birbaumer, E. Donchin, E. W. Wolpaw, and J. R. Wolpaw. Brain-computer interface technology: a review of the Second International Meeting. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 11:94-109, 2003.
75. I. Verdinelli. A note on Bayesian design for the normal linear model with unknown error variance. Biometrika, 87:222-227, 2000.
76. M. Yamakita, M. Iwashiro, Y. Sugahara, and K. Furuta. Robust swing-up control of double pendulum. In American Control Conference, volume 1, pages 290-295, 1995.
77. K. Zografos and S. Nadarajah. Expressions for Rényi and Shannon entropies for multivariate distributions. Statistics and Probability Letters, 71(1):71-84, 2005.