System Parameter Identification: Information Criteria and Algorithms

System Parameter Identification: Information Criteria and Algorithms

Volumn , Issue , 2013, Pages 1-249

  Chen, Badong ^a   Zhu, Yu ^b   Hu, Jinchun ^b   Principe, Jose ^c  

^a XI'AN JIAOTONG UNIVERSITY   (China)

^b TSINGHUA UNIVERSITY   (China)

^c UNIVERSITY OF FLORIDA   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

EID: 84894584520     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1016/C2012-0-01233-1     Document Type: Book

References (240)

1. Zadeh L.A. From circuit theory to system theory. Proc. IRE 1962, 50(5):856-865.
2. Eykhoff P. System Identification-Parameter and State Estimation 1974, John Wiley & Sons, Inc., London.
3. Ljung L. Convergence analysis of parametric identification methods. IEEE Trans. Automat. Control 1978, 23:770-783.
4. Ljung L. System Identification: Theory for the User 1999, Prentice Hall PTR, Upper Saddle River, New Jersey. second ed.
5. Zhang P. Model selection via multifold cross validation. Ann. Stat. 1993, 299-313.
6. H. Akaike, Information theory and an extension of the maximum likelihood principle, Proceedings of the Second International Symposium on Information Theory, 1973, pp. 267-281.
7. Akaike H. A new look at the statistical model identification. IEEE Trans. Automat. Control 1974, 19(6):716-723.
8. Schwarz G. Estimating the dimension of a model. Ann. Stat. 1978, 6:461-464.
9. Rissanen J. Modeling by shortest data description. Automatica 1978, 14:465-471.
10. Barron A., Rissanen J., Yu B. The minimum description length principle in coding and modeling. IEEE Trans. Inf. Theory 1998, 44(6):2743-2760.
11. Rojas C.R., Welsh J.S., Goodwin G.C., Feuer A. Robust optimal experiment design for system identification. Automatica 2007, 43(6):993-1008.
12. Liu W., Principe J., Haykin S. Kernel Adaptive Filtering: A Comprehensive Introduction 2010, Wiley.
13. Wolberg J.R., Wolberg J. Data Analysis Using the Method of Least Squares: Extracting the Most Information from Experiments 2006, 1. Springer, Berlin, Germany.
14. Kailath T., Sayed A.H., Hassibi B. Linear Estimation 2000, 1. Prentice Hall, New Jersey.
15. Aldrich J. R. A. Fisher and the making of maximum likelihood 1912-1922. Stat. Sci. 1997, 12(3):162-176.
16. Hald A. On the history of maximum likelihood in relation to inverse probability and least squares. Stat. Sci. 1999, 14(2):214-222.
17. Tikhonov A.N., Arsenin V.Y. Solution of Ill-posed Problems 1977, Winston & Sons, Washington.
18. Widrow B., Sterns S. Adaptive Signal Processing 1985, Prentice Hall, Englewood Cliffs, NJ.
19. Haykin S. Adaptive Filter Theory 2002, Prentice Hall, Englewood Cliffs, NJ.
20. Least-Mean-Square Adaptive Filters 2003, Wiley, New York. S.S. Haykin, B. Widrow (Eds.).
21. Sherman S. Non-mean-square error criteria. IRE Trans. Inf. Theory 1958, 4:125-126.
22. Brown J.L. Asymmetric non-mean-square error criteria. IRE Trans. Automat. Control 1962, 7:64-66.
23. Zakai M. General error criteria. IEEE Trans. Inf. Theory 1964, 10(1):94-95.
24. Hall E.B., Wise G.L. On optimal estimation with respect to a large family of cost function. IEEE Trans. Inf. Theory 1991, 37(3):691-693.
25. Ljung L., Soderstrom T. Theory and Practice of Recursive Identification 1983, MIT Press, Cambridge, MA.
26. Walach E., Widrow B. The least mean fourth (LMF) adaptive algorithm and its family. IEEE. Trans. Inf. Theory 1984, 30(2):275-283.
27. Walach E. On high-order error criteria for system identification. IEEE Trans. Acoust. 1985, 33(6):1634-1635.
28. Douglas S.C., Meng T.H.Y. Stochastic gradient adaptation under general error criteria. IEEE Trans. Signal Process 1994, 42:1335-1351.
29. Al-Naffouri T.Y., Sayed A.H. Adaptive filters with error nonlinearities: mean-square analysis and optimum design. EURASIP J. Appl. Signal. Process 2001, 4:192-205.
30. Pei S.C., Tseng C.C. Least mean p-power error criterion for adaptive FIR filter. IEEE J. Sel. Areas Commun. 1994, 12(9):1540-1547.
31. Shao M., Nikias C.L. Signal processing with fractional lower order moments: stable processes and their applications. Proc. IEEE 1993, 81(7):986-1009.
32. Nikias C.L., Shao M. Signal Processing with Alpha-Stable Distributions and Applications 1995, Wiley, New York.
33. Rousseeuw P.J., Leroy A.M. Robust Regression and Outlier Detection 1987, John Wiley & Sons, Inc., New York.
34. Chambers J.A., Tanrikulu O., Constantinides A.G. Least mean mixed-norm adaptive filtering. Electron. Lett. 1994, 30(19):1574-1575.
35. Tanrikulu O., Chambers J.A. Convergence and steady-state properties of the least-mean mixed-norm (LMMN) adaptive algorithm. IEE Proc. Vis. Image Signal Process 1996, 143(3):137-142.
36. Chambers J., Avlonitis A. A roust mixed-norm adaptive filter algorithm. IEEE Signal Process. Lett. 1997, 4(2):46-48.
37. Boel R.K., James M.R., Petersen I.R. Robustness and risk-sensitive filtering. IEEE Trans. Automat. Control 2002, 47(3):451-461.
38. Lo J.T., Wanner T. Existence and uniqueness of risk-sensitive estimates. IEEE Trans. Automat. Control 2002, 47(11):1945-1948.
39. Delopoulos A.N., Giannakis G.B. Strongly consistent identification algorithms and noise insensitive MSE criteria. IEEE Trans. Signal Process 1992, 40(8):1955-1970.
40. C.Y. Chi, W.T. Chen, Linear prediction based on higher order statistics by a new criterion, Proceedings of Sixth IEEE SP Workshop Stat. Array Processing, 1992.
41. Chi C.Y., Chang W.J., Feng C.C. A new algorithm for the design of linear prediction error filters using cumulant-based MSE criteria. IEEE Trans. Signal Process 1994, 42(10):2876-2880.
42. Feng C.C., Chi C.Y. Design of Wiener filters using a cumulant based MSE criterion. Signal Process 1996, 54:23-48.
43. Cover T.M., Thomas J.A. Elements of Information Theory 1991, John Wiley &Sons, Inc., Chichester.
44. Shannon C.E. A mathematical theory of communication. J. Bell Syst. Technol. 1948, 27(379-23):623-656.
45. J.P. Burg, Maximum entropy spectral analysis, Proceedings of the Thirty-Seventh Annual International Social Exploration Geophysics Meeting, Oklahoma City, OK, 1967.
46. Lagunas M.A., Santamaria M.E., Figueiras A.R. ARMA model maximum entropy power spectral estimation. IEEE Trans. Acoust. 1984, 32:984-990.
47. Ihara S. Maximum entropy spectral analysis and ARMA processes. IEEE Trans. Inf. Theory 1984, 30:377-380.
48. Kay S.M. Modern Spectral Estimation: Theory and Application 1988, Prentice Hall, Englewood Cliffs, NJ.
49. Linsker R. Self-organization in perceptual networks. Computer 1988, 21:105-117.
50. Linsker R. How to generate ordered maps by maximizing the mutual information between input and output signals. Neural Comput. 1989, 1:402-411.
51. R. Linsker, Deriving receptive fields using an optimal encoding criterion, in: S.J. Hansor (Ed.), Proceedings of Advances in Neural Information Processing Systems, 1993, pp. 953-960.
52. Deco G., Obradovic D. An Information-Theoretic Approach to Neural Computing 1996, Springer-Verlag, New York.
53. Haykin S. Neural Networks: A Comprehensive Foundation 1999, Prentice Hall, Inc., Englewood Cliffs, NJ.
54. Comon P. Independent component analysis, a new concept?. Signal Process 1994, 36(3):287-314.
55. Lee T.W., Girolami M., Sejnowski T. Independent component analysis using an extended infomax algorithm for mixed sub-Gaussian and super-Gaussian sources. Neural Comput. 1999, 11(2):409-433.
56. Lee T.W., Girolami M., Bell A.J. A unifying information-theoretic framework for independent component analysis. Comput. Math. Appl. 2000, 39(11):1-21.
57. Erdogmus D., Hild K.E., Rao Y.N., Principe J.C. Minimax mutual information approach for independent component analysis. Neural Comput. 2004, 16:1235-1252.
58. Cardoso J.F. Infomax and maximum likelihood for blind source separation. IEEE Signal Process. Lett. 1997, 4:109-111.
59. Yang H.H., Amari S.I. Adaptive online learning algorithms for blind separation: maximum entropy minimum mutual information. Neural Comput. 1997, 9:1457-1482.
60. Pham D.T. Mutual information approach to blind separation of stationary source. IEEE Trans. Inf. Theory 2002, 48(7):1-12.
61. Zadeh M.B., Jutten C. A general approach for mutual information minimization and its application to blind source separation. Signal Process 2005, 85:975-995.
62. Principe J.C., Xu D., Fisher J.W. Information theoretic learning. Unsupervised Adaptive Filtering 2000, Wiley, New York. S. Haykin (Ed.).
63. Principe J.C., Xu D., Zhao Q., et al. Learning from examples with information theoretic criteria. J. VLSI Signal Process. Syst. 2000, 26:61-77.
64. Principe J.C. Information Theoretic Learning: Renyi's Entropy and Kernel Perspectives 2010, Springer, New York.
65. Fisher J.W. Nonlinear Extensions to the Minimum Average Correlation Energy Filter 1997, University of Florida, USA.
66. Xu D. Energy, Entropy and Information Potential for Neural Computation 1999, University of Florida, USA.
67. Erdogmus D. Information Theoretic Learning: Renyi's Entropy and Its Applications to Adaptive System Training 2002, University of Florida, USA.
68. Erdogmus D., Principe J.C. From linear adaptive filtering to nonlinear information processing. IEEE Signal Process. Mag. 2006, 23(6):15-33.
69. Zaborszky J. An information theory viewpoint for the general identification problem. IEEE Trans. Automat. Control 1966, 11(1):130-131.
70. Van Trees H.L. Detection, Estimation, and Modulation Theory, Part I 1968, John Wiley & Sons, New York.
71. Snyder D.L., Rhodes I.B. Filtering and control performance bounds with implications on asymptotic separation. Automatica 1972, 8:747-753.
72. Galdos J.I.A. Cramer-Rao bound for multidimensional discrete-time dynamical systems. IEEE Trans. Automat. Control 1980, 25:117-119.
73. Friedlander B., Francos J. On the accuracy of estimating the parameters of a regular stationary process. IEEE Trans. Inf. Theory 1996, 42(4):1202-1211.
74. Stoica P., Marzetta T.L. Parameter estimation problems with singular information matrices. IEEE Trans. Signal Process 2001, 49(1):87-90.
75. Seidman L.P. Performance limitations and error calculations for parameter estimation. Proc. IEEE 1970, 58:644-652.
76. Zakai M., Ziv J. Lower and upper bounds on the optimal filtering error of certain diffusion process. IEEE Trans. Inf. Theory 1972, 18(3):325-331.
77. Galdos J.I. A rate distortion theory lower bound on desired function filtering error. IEEE Trans. Inf. Theory 1981, 27:366-368.
78. Washburn R.B., Teneketzis D. Rate distortion lower bound for a special class of nonlinear estimation problems. Syst. Control Lett. 1989, 12:281-286.
79. Weidemann H.L., Stear E.B. Entropy analysis of estimating systems. IEEE Trans. Inf. Theory 1970, 16(3):264-270.
80. Duncan T.E. On the calculation of mutual information. SIAM J. Appl. Math. 1970, 19:215-220.
81. Guo D., Shamai S., Verdu S. Mutual information and minimum mean-square error in Gaussian Channels. IEEE Trans. Inf. Theory 2005, 51(4):1261-1282.
82. Zakai M. On mutual information, likelihood-ratios and estimation error for the additive Gaussian channel. IEEE Trans. Inf. Theory 2005, 51(9):3017-3024.
83. Binia J. Divergence and minimum mean-square error in continuous-time additive white Gaussian noise channels. IEEE Trans. Inf. Theory 2006, 52(3):1160-1163.
84. T.E. Duncan, B. Pasik-Duncan, Estimation and mutual information, Proceedings of the Forty-Sixth IEEE Conference on Decision and Control, New Orleans, LA, USA, 2007, pp. 324-327.
85. Verdu S. Mismatched estimation and relative entropy. IEEE Trans. Inf. Theory 2010, 56(8):3712-3720.
86. Weidemann H.L., Stear E.B. Entropy analysis of parameter estimation. Inf. Control 1969, 14:493-506.
87. Tomita Y., Ohmatsu S., Soeda T. An application of the information theory to estimation problems. Inf. Control 1976, 32:101-111.
88. Kalata P., Priemer R. Linear prediction, filtering, and smoothing: an information theoretic approach. Inf. Sci. (Ny) 1979, 17:1-14.
89. Minamide N. An extension of the entropy theorem for parameter estimation. Inf. Control 1982, 53(1):81-90.
90. Janzura M., Koski T., Otahal A. Minimum entropy of error principle in estimation. Inf. Sci. (Ny) 1994, 79:123-144.
91. Chen T.L., Geman S. On the minimum entropy of a mixture of unimodal and symmetric distributions. IEEE Trans. Inf. Theory 2008, 54(7):3166-3174.
92. Chen B., Zhu Y., Hu J., Zhang M. On optimal estimations with minimum error entropy criterion. J. Franklin Inst. 2010, 347(2):545-558.
93. Chen B., Zhu Y., Hu J., Zhang M. A new interpretation on the MMSE as a robust MEE criterion. Signal Process 2010, 90(12):3313-3316.
94. Chen B., Principe J.C. Some further results on the minimum error entropy estimation. Entropy 2012, 14(5):966-977.
95. Chen B., Principe J.C. On the smoothed minimum error entropy criterion. Entropy 2012, 14(11):2311-2323.
96. Parzen E. On estimation of a probability density function and mode. Time Series Analysis Papers 1967, Holden-Day, Inc., San Diego, CA.
97. Silverman B.W. Density Estimation for Statistic and Data Analysis 1986, Chapman & Hall, NY.
98. Devroye L., Lugosi G. Combinatorial Methods in Density Estimation 2000, Springer-Verlag, New York.
99. Santamaria I., Erdogmus D., Principe J.C. Entropy minimization for supervised digital communications channel equalization. IEEE Trans. Signal Process 2002, 50(5):1184-1192.
100. Erdogmus D., Principe J.C. An error-entropy minimization algorithm for supervised training of nonlinear adaptive systems. IEEE Trans. Signal Process 2002, 50(7):1780-1786.
101. Erdogmus D., Principe J.C. Generalized information potential criterion for adaptive system training. IEEE Trans. Neural Netw. 2002, 13:1035-1044.
102. Erdogmus D., Principe J.C. Convergence properties and data efficiency of the minimum error entropy criterion in Adaline training. IEEE Trans. Signal Process 2003, 51:1966-1978.
103. Erdogmus D., Hild K.E., Principe J.C. Online entropy manipulation: stochastic information gradient. IEEE Signal Process. Lett. 2003, 10:242-245.
104. Morejon R.A., Principe J.C. Advanced search algorithms for information-theoretic learning with kernel-based estimators. IEEE Trans. Neural Netw. 2004, 15(4):874-884.
105. Han S., Rao S., Erdogmus D., Jeong K.H., Principe J.C. A minimum-error entropy criterion with self-adjusting step-size (MEE-SAS). Signal Process 2007, 87:2733-2745.
106. Chen B., Zhu Y., Hu J. Mean-square convergence analysis of ADALINE training with minimum error entropy criterion. IEEE Trans. Neural Netw. 2010, 21(7):1168-1179.
107. Guo L.Z., Billings S.A., Zhu D.Q. An extended orthogonal forward regression algorithm for system identification using entropy. Int. J. Control. 2008, 81(4):690-699.
108. Kullback S. Information Theory and Statistics 1959, John Wiley & Sons, New York.
109. Kulhavý R.A. Kullback-Leibler distance approach to system identification. Annu. Rev. Control 1996, 20:119-130.
110. Matsuoka T., Ulrych T.J. Information theory measures with application to model identification. IEEE. Trans. Acoust. 1986, 34(3):511-517.
111. Cavanaugh J.E. A large-sample model selection criterion based on Kullback's symmetric divergence. Stat. Probab. Lett. 1999, 42:333-343.
112. Seghouane A.K., Bekara M. A small sample model selection criterion based on the Kullback symmetric divergence. IEEE Trans. Signal Process 2004, 52(12):3314-3323.
113. Seghouane A.K., Amari S.I. The AIC criterion and symmetrizing the Kullback-Leibler divergence. IEEE Trans. Neural Netw. 2007, 18(1):97-106.
114. Seghouane A.K. Asymptotic bootstrap corrections of AIC for linear regression models. Signal Process 2010, 90(1):217-224.
115. Baram Y., Sandell N.R. An information theoretic approach to dynamic systems modeling and identification. IEEE Trans. Automat. Control 1978, 23(1):61-66.
116. Baram Y., Beeri Y. Stochastic model simplification. IEEE Trans. Automat. Control 1981, 26(2):379-390.
117. Tugnait J.K. Continuous-time stochastic model simplification. IEEE Trans. Automat. Control 1982, 27(4):993-996.
118. Leland R. Reduced-order models and controllers for continuous-time stochastic systems: an information theory approach. IEEE Trans. Automat. Control 1999, 44(9):1714-1719.
119. Leland R. An approximate-predictor approach to reduced-order models and controllers for distributed-parameter systems. IEEE Trans. Automat. Control 1999, 44(3):623-627.
120. Chen B., Hu J., Zhu Y., Sun Z. Parameter identifiability with Kullback-Leibler information divergence criterion. Int. J. Adapt. Control Signal Process 2009, 23(10):940-960.
121. Weinstein E., Feder M., Oppenheim A.V. Sequential algorithms for parameter estimation based on the Kullback-Leibler information measure. IEEE. Trans. Acoust. 1990, 38(9):1652-1654.
122. Krishnamurthy V. Online estimation of dynamic shock-error models based on the Kullback-Leibler information measure. IEEE Trans. Automat. Control 1994, 39(5):1129-1135.
123. Stoorvogel A.A., Van Schuppen J.H. Approximation problems with the divergence criterion for Gaussian variables and Gaussian process. Syst. Control. Lett. 1998, 35:207-218.
124. Stoorvogel A.A., Van Schuppen J.H. System identification with information theoretic criteria. Identification, Adaptation, Learning 1996, Springer, Berlin. S. Bittanti, G. Picc (Eds.).
125. Pu L., Hu J., Chen B. Information theoretical approach to identification of hybrid systems. Hybrid Systems: Computation and Control 2008, Springer, Berlin Heidelberg, pp. 650-653.
126. Chen B., Zhu Y., Hu J., Sun Z. Adaptive filtering under minimum information divergence criterion. Int. J. Control Autom. Syst. 2009, 7(2):157-164.
127. Chandra S.A., Taniguchi M. Minimum α-divergence estimation for ARCH models. J. Time Series Anal. 2006, 27(1):19-39.
128. Pardo M.C. Estimation of parameters for a mixture of normal distributions on the basis of the Cressie and Read divergence. Commun. Stat-Simul. C. 1999, 28(1):115-130.
129. Cressie N., Pardo L. Minimum ë-divergence estimator and hierarchical testing in loglinear models. Stat. Sin. 2000, 10:867-884.
130. Pardo L. Statistical Inference Based on Divergence Measures 2006, Chapman & Hall/CRC, Boca Raton, FL.
131. Feng X., Loparo K.A., Fang Y. Optimal state estimation for stochastic systems: an information theoretic approach. IEEE Trans. Automat. Control 1997, 42(6):771-785.
132. Mustafa D., Glover K. Minimum entropy H∞ control. Lecture Notes in Control and Information Sciences 1990, 146. Springer-Verlag, Berlin.
133. Yang J.-M., Sakai H. A robust ICA-based adaptive filter algorithm for system identification. IEEE Trans. Circuits Syst. Express Briefs 2008, 55(12):1259-1263.
134. Durgaryan I.S., Pashchenko F.F. Identification of objects by the maximal information criterion. Autom. Remote Control 2001, 62(7):1104-1114.
135. Chen B., Hu J., Li H., Sun Z. Adaptive filtering under maximum mutual information criterion. Neurocomputing 2008, 71(16):3680-3684.
136. Chen B., Zhu Y., Hu J., Príncipe J.C. Stochastic gradient identification of Wiener system with maximum mutual information criterion. Signal Process., IET 2011, 5(6):589-597.
137. Liu W., Pokharel P.P., Principe J.C. Correntropy: properties and applications in non-Gaussian signal processing. IEEE Trans. Signal Process 2007, 55(11):5286-5298.
138. A. Singh, J.C. Principe, Using correntropy as a cost function in linear adaptive filters, in: International Joint Conference on Neural Networks (IJCNN'09), IEEE, 2009, pp. 2950-2955.
139. S. Zhao, B. Chen, J.C. Principe, Kernel adaptive filtering with maximum correntropy criterion, in: The 2011 International Joint Conference on Neural Networks (IJCNN), IEEE, 2011, pp. 2012-2017.
140. Bessa R.J., Miranda V., Gama J. Entropy and correntropy against minimum square error in offline and online three-day ahead wind power forecasting. IEEE Trans. Power Systems 2009, 24(4):1657-1666.
141. Xu J.W., Erdogmus D., Principe J.C. Minimizing Fisher information of the error in supervised adaptive filter training, in: Proc. ICASSP, 2004, pp.513-516.
142. Sakamoto Y., Ishiguro M., Kitagawa G. Akaike Information Criterion Statistics 1986, Reidel Publishing Company, Dordretcht, Netherlands.
143. Burnham K.P., Anderson D.R. Model Selection and Multimodel Inference: A Practical Information Theoretic Approach 2002, Springer-Verlag, New York. second ed.
144. Grunwald P.D. The Minimum Description Length Principle 2007, MIT Press, Cambridge, MA.
145. N. Tishby, F.C. Pereira, W. Bialek, The information bottleneck method. arXiv preprint physics/0004057, 2000.
146. Kolmogorov A.N. Three approaches to the quantitative definition of information. Probl. Inform. Transm. 1965, 1:4-7.
147. Johnson O., Johnson O.T. Information Theory and the Central Limit Theorem 2004, Imperial College Press, London.
148. Jaynes E.T. Information theory and statistical mechanics. Phys. Rev. 1957, 106:620-630.
149. Kapur J.N., Kesavan H.K. Entropy Optimization Principles with Applications 1992, Academic Press, Inc.
150. Ormoneit D., White H. An efficient algorithm to compute maximum entropy densities. Econom. Rev. 1999, 18(2):127-140.
151. Wu X. Calculation of maximum entropy densities with application to income distribution. J. Econom. 2003, 115(2):347-354.
152. A. Renyi, On measures of entropy and information. Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, vol. 1, 1961, pp. 547-561.
153. Havrda J., Charvat F. Concept of structural α-entropy. Kybernetika 1967, 3:30-35.
154. Varma R.S. Generalizations of Renyi's entropy of order α. J. Math. Sci. 1966, 1:34-48.
155. Arimoto S. Information-theoretic considerations on estimation problems. Inf. Control 1971, 19:181-194.
156. Salicru M., Menendez M.L., Morales D., Pardo L. Asymptotic distribution of (h,ë)-entropies. Commun. Stat. Theory Methods 1993, 22:2015-2031.
157. Rao M., Chen Y., Vemuri B.C., Wang F. Cumulative residual entropy: a new measure of information. IEEE Trans. Inf. Theory 2004, 50(6):1220-1228.
158. Zografos K., Nadarajah S. Survival exponential entropies. IEEE Trans. Inf. Theory 2005, 51(3):1239-1246.
159. Chen B., Zhu P., Príncipe J.C. Survival information potential: a new criterion for adaptive system training. IEEE Trans. Signal Process 2012, 60(3):1184-1194.
160. Mercer J. Functions of positive and negative type, and their connection with the theory of integral equations. Philos. Trans. R. Soc. London 1909, 209:415-446.
161. Vapnik V. The Nature of Statistical Learning Theory 1995, Springer, New York.
162. Scholkopf B., Smola A.J. Learning with Kernels, Support Vector Machines, Regularization, Optimization and Beyond 2002, MIT Press, Cambridge, MA, USA.
163. Whittle P. The analysis of multiple stationary time series. J. R. Stat. Soc. B 1953, 15(1):125-139.
164. Dempster A.P., Laird N.M., Rubin D.B. Maximum likelihood from incomplete data via the EM algorithm. J. R. Stat. Soc. B 1977, 39(1):1-38.
165. Mc Lachlan G.J., Krishnan T. The EM Algorithm and Extensions 2008, Wiley-Interscience, NJ.
166. Aiazzi B., Alparone L., Baronti S. Estimation based on entropy matching for generalized Gaussian PDF modeling. IEEE Signal Process. Lett. 1999, 6(6):138-140.
167. Pham D.T. Entropy of a variable slightly contaminated with another. IEEE Signal Process. Lett. 2005, 12:536-539.
168. Chen B., Hu J., Zhu Y., Sun Z. Information theoretic interpretation of error criteria. Acta Automatica Sin. 2009, 35(10):1302-1309.
169. Chen B., Príncipe J.C. Maximum correntropy estimation is a smoothed MAP estimation. IEEE Signal Process. Lett. 2012, 19(8):491-494.
170. Rubinstein R.Y. Simulation and the Monte Carlo Method 1981, Wiley, New York.
171. Styblinsli M.A., Tang T.S. Experiments in nonconvex optimization: stochastic approximation with function smoothing and simulated annealing. Neural Netw. 1990, 3:467-483.
172. Edmonson W., Srinivasan K., Wang C., Principe J. A global least mean square algorithm for adaptive IIR filtering. IEEE Trans. Circuits Syst 1998, 45:379-384.
173. Goodwin G.C., Payne R.L. Dynamic System Identification: Experiment Design and Data Analysis 1977, Academic Press, New York.
174. Moore E. On properly positive Hermitian matrices. Bull. Amer. Math. Soc. 1916, 23(59):66-67.
175. Aronszajn N. The theory of reproducing kernels and their applications. Cambridge Philos. Soc. Proc. 1943, 39:133-153.
176. Engel Y., Mannor S., Meir R. The kernel recursive least-squares algorithm. IEEE Trans. Signal Process 2004, 52:2275-2285.
177. Liu W., Pokharel P., Principe J. The kernel least mean square algorithm. IEEE Trans. Signal Process 2008, 56:543-554.
178. Liu W., Principe J. Kernel affine projection algorithm. EURASIP J. Adv. Signal Process 2008, Article ID 784292, 12 pages. 10.1155/2008/784292.
179. Platt J. A resource-allocating network for function interpolation. Neural. Comput. 1991, 3:213-225.
180. Richard C., Bermudez J.C.M., Honeine P. Online prediction of time series data with kernels. IEEE Trans. Signal Process 2009, 57:1058-1066.
181. Liu W., Park I.l., Principe J.C. An information theoretic approach of designing sparse kernel adaptive filters. IEEE Trans. Neural Netw. 2009, 20:1950-1961.
182. Chen B., Zhao S., Zhu P., Principe J.C. Quantized kernel least mean square algorithm. IEEE Trans. Neural Netw. Learn. Syst. 2012, 23(1):22-32.
183. Berilant J., Dudewicz E.J., Gyorfi L., van der Meulen E.C. Nonparametric entropy estimation: an overview. Int. J. Math. Statist. Sci. 1997, 6(1):17-39.
184. Vasicek O. A test for normality based on sample entropy. J. Roy. Statist. Soc. B 1976, 38(1):54-59.
185. Singh A., Príncipe J.C. Information theoretic learning with adaptive kernels. Signal Process 2011, 91(2):203-213.
186. D. Erdogmus, J.C. Principe, S.-P. Kim, J.C. Sanchez, A recursive Renyi's entropy estimator, in: Proceedings of the Twelfth IEEE Workshop on Neural Networks for Signal Processing, 2002, pp. 209-217.
187. Wu X., Stengos T. Partially adaptive estimation via the maximum entropy densities. J. Econom. 2005, 8:352-366.
188. Chen B., Zhu Y., Hu J., Zhang M. Stochastic information gradient algorithm based on maximum entropy density estimation. ICIC Exp. Lett. 2010, 4(3):1141-1145.
189. Zhu Y., Chen B., Hu J. Adaptive filtering with adaptive p-power error criterion. Int. J. Innov. Comput. Inf. Control 2011, 7(4):1725-1738.
190. Chen B., Principe J.C., Hu J., Zhu Y. Stochastic information gradient algorithm with generalized Gaussian distribution model. J. Circuit. Syst. Comput. 2012, 21(1).
191. Varanasi M.K., Aazhang B. Parametric generalized Gaussian density estimation. J. Acoust. Soc. Amer. 1989, 86(4):1404-1415.
192. Kokkinakis K., Nandi A.K. Exponent parameter estimation for generalized Gaussian probability density functions with application to speech modeling. Signal Process 2005, 85:1852-1858.
193. S. Han, J.C. Principe, A fixed-point minimum error entropy algorithm, in: Proceedings of the Sixteenth IEEE Signal Processing Society Workshop on Machine Learning for Signal Processing, 2006, pp.167-172.
194. S. Han, A family of minimum Renyi's error entropy algorithm for information processing, Doctoral dissertation, University of Florida, 2007.
195. Chen S., Billings S.A., Grant P.M. Recursive hybrid algorithm for non-linear system identification using radial basis function networks. Int. J. Control. 1992, 55:1051-1070.
196. Sayed A. Fundamentals of Adaptive Filtering 2003, Wiley, New York.
197. Clark G.A., Mitra S.K., Parker S.R. Block implementation of adaptive digital filters. IEEE Trans. Acoust. Speech Signal Process 1981, ASSP-29(3):744-752.
198. Bershad N.J., Bonnet M. Saturation effects in LMS adaptive echo cancellation for binary data. IEEE Trans. Acoust. Speech Signal Process 1990, 38(10):1687-1696.
199. T.Y. Al-Naffouri, A. Zerguine, M. Bettayeb, Convergence analysis of the LMS algorithm with a general error nonlinearity and an iid input, in: Proceedings of the Asilomar Conference on Signals, Systems, and Computers, vol. 1, 1998, pp. 556-559.
200. Chen B., Hu J., Pu L., Sun Z. Stochastic gradient algorithm under (h,φ)-entropy criterion. Circuits Syst. Signal Process 2007, 26(6):941-960.
201. Gibson J.D., Gray S.D. MVSE adaptive filtering subject to a constraint on MSE. IEEE Trans. Circuits Syst. 1988, 35(5):603-608.
202. Rao B.L.S.P. Asymptotic Theory of Statistical Inference 1987, Wiley, New York.
203. Kaplan D., Glass L. Understanding Nonlinear Dynamics 1995, Springer-Verlag, New York.
204. J.M. Kuo, Nonlinear dynamic modeling with artificial neural networks, Ph.D. dissertation, University of Florida, Gainesville, 1993.
205. Luenberger D.G. Linear and Nonlinear Programming 1973, Addison-Wesley, Reading, MA.
206. Wang L.Y., Zhang J.F., Yin G.G. System identification using binary sensors. IEEE Trans. Automat. Control 2003, 48(11):1892-1907.
207. Harvey A.C., Fernandez C. Time series for count data or qualitative observations. J. Bus. Econ. Stat. 1989, 7:407-417.
208. Al-Osh M., Alzaid A. First order integer-valued autoregressive INAR(1) process. J. Time Series Anal. 1987, 8(3):261-275.
209. Brannas K., Hall A. Estimation in integer-valued moving average models. Appl. Stoch. Model. Bus. Ind. 2001, 17(3):277-291.
210. Weis C.H. Thinning operations for modeling time series of counts-a survey. AStA Adv. Stat. Anal. 2008, 92(3):319-341.
211. Chen B., Zhu Y., Hu J., Principe J.C. Δ-Entropy: definition, properties and applications in system identification with quantized data. Inf. Sci. 2011, 181(7):1384-1402.
212. Janzura M., Koski T., Otahal A. Minimum entropy of error estimation for discrete random variables. IEEE Trans. Inf. Theory 1996, 42(4):1193-1201.
213. Silva L.M., Felgueiras C.S., Alexandre L.A., Marques J. Error entropy in classification problems: a univariate data analysis. Neural Comput. 2006, 18:2036-2061.
214. Ozertem U., Uysal I., Erdogmus D. Continuously differentiable sample-spacing entropy estimation. IEEE Trans. Neural Netw. 2008, 19:1978-1984.
215. Larranaga P., Lozano J.A. Estimation of Distribution Algorithms: A New Tool for Evolutionary Computation 2002, Kluwer Academic Publishers, Boston.
216. Rothenberg T.J. Identification in parametric models. Econometrica 1971, 39:577-591.
217. Grewal M.S., Glover K. Identifiability of linear and nonlinear dynamic systems. IEEE Trans. Automat. Control 1976, 21(6):833-837.
218. Tse E., Anton J.J. On the identifiability of parameters. IEEE Trans. Automat. Control 1972, 17(5):637-646.
219. Glover K., Willems J.C. Parameterizations of linear dynamical systems: canonical forms and identifiability. IEEE Trans. Automat. Control 1974, 19(6):640-646.
220. van der Vaart A.W. Asymptotic Statistics, Cambridge Series in Statistical and Probabilistic Mathematics 1998, Cambridge University Press, New York.
221. Doob J.L. Stochastic Processes 1953, John Wiley, New York.
222. Sara A., van de Geer Empirical Processes in M-estimation, Cambridge Series in Statistical and Probabilistic Mathematics 2000, Cambridge University Press, Cambridge.
223. Barron A.R., Gyorfi L., van der Meulen E.C. Distribution estimation consistent in total variation and in two types of information divergence. IEEE Trans. Inf. Theory 1992, 38(5):1437-1454.
224. Papoulis A., Pillai S.U. Probability, Random Variables, and Stochastic Processes 2002, McGraw-Hill Companies, Inc., New York. fourth ed.
225. Karny M. Towards fully probabilistic control design. Automatica 1996, 32(12):1719-1722.
226. Karny M., Guy T.V. Fully probabilistic control design. Syst. Control Lett. 2006, 55:259-265.
227. Wang H. Robust control of the output probability density functions for multivariable stochastic systems with guaranteed stability. IEEE Trans. Automat. Control 1999, 44(11):2103-2107.
228. Wang H. Bounded Dynamic Stochastic Systems: Modeling and Control 2000, Springer-Verlag, New York.
229. Wang H., Yue H. A rational spline model approximation and control of output probability density functions for dynamic stochastic systems. Trans. Inst. Meas. Control 2003, 25(2):93-105.
230. Sala-Alvarez J., Vázquez-Grau G. Statistical reference criteria for adaptive signal processing in digital communications. IEEE Trans. Signal Process 1997, 45(1):14-31.
231. Meyer M.E., Gokhale D.V. Kullback-Leibler information measure for studying convergence rates of densities and distributions. IEEE Trans. Inf. Theory 1993, 39(4):1401-1404.
232. R. Vidal, B. Anderson, Recursive identification of switched ARX hybrid models: exponential convergence and persistence of excitation, Proceedings of the Forty-Third IEEE Conference on Decision and Control (CDC), 2004.
233. C.-A. Lai, Global optimization algorithms for adaptive infinite impulse response filters, Ph.D. Dissertation, University of Florida, 2002.
234. B. Chen, J. Hu, H. Li, Z. Sun, Adaptive FIR filtering under minimum error/input information criterion. The Seventeenth IFAC Word Conference, Seoul, Korea, July 2008, pp. 3539-3543.
235. Kailath T., Hassibi B. Linear Estimation 2000, Prentice Hall, NJ.
236. Hyvarinen A., Karhunen J., Oja E. Independent Component Analysis 2001, Wiley, New York.
237. Cardoso J.F., Laheld B.H. Equivariant adaptive source separation. IEEE Trans. Signal Process 1996, 44(12):3017-3030.
238. Schetzen M. The Volterra and Wiener Theories of Nonlinear Systems 1980, Wiley, New York.
239. Bershad N.J., Celka P., Vesin J.M. Stochastic analysis of gradient adaptive identification of nonlinear systems with memory for Gaussian data and noisy input and output measurements. IEEE Trans. Signal Process 1999, 47:675-689.
240. Celka P., Bershad N.J., Vesin J.M. Stochastic gradient identification of polynomial Wiener systems: analysis and application. IEEE Trans. Signal Process 2001, 49:301-313.