A unifying information-theoretic framework for independent component analysis

Computers and Mathematics with Applications

Volumn 39, Issue 11, 2000, Pages 1-21

  Lee, Te Won ^a,b   Girolami, M ^c   Bell, A J ^a   Sejnowski, T J ^a,d  

^a HOWARD HUGHES MEDICAL INSTITUTE   (United States)

^b TECHNISCHE UNIVERSITÄT BERLIN   (Germany)

^c UNIVERSITY OF THE WEST OF SCOTLAND   (United Kingdom)

^d UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Blind source separation; Entropy; ICA; Information maximization; Maximum likelihood estimation

Indexed keywords

APPROXIMATION THEORY; ASYMPTOTIC STABILITY; CONVERGENCE OF NUMERICAL METHODS; INFORMATION THEORY; ITERATIVE METHODS; LEARNING ALGORITHMS; LEARNING SYSTEMS; MATRIX ALGEBRA; PROBABILITY DENSITY FUNCTION;

BLIND SOURCE SEPARATION; INDEPENDENT COMPONENT ANALYSIS; INFORMATION MAXIMIZATION; MULTICHANNEL BLIND DECONVOLUTION; NEGENTROPY MAXIMIZATION;

MAXIMUM LIKELIHOOD ESTIMATION;

EID: 0034207888     PISSN: 08981221     EISSN: None     Source Type: Journal    
DOI: 10.1016/S0898-1221(00)00101-2     Document Type: Article

References (91)

1. B. Pearlmutter and L. Parra, A context-sensitive generalization of ICA, In ICONIP'96, pp. 151-157, (1996).
2. J.-F. Cardoso, Infomax and maximum likelihood for blind source separation, IEEE Signal Processing Letters 4, 109-111 (1997).
3. A.J. Bell and T.J. Sejnowski, An information maximization approach to blind separation and blind deconvolution, Neural Computation 7, 1129-1159 (1995).
4. M. Girolami and C. Fyfe, Stochastic ICA contrast maximization using Oja's nonlinear PCA algorithm, International Journal of Neural Systems (to appear).
5. J. Karhunen and J. Joutsensalo, Representation and separation of signals using nonlinear PCA type learning, Neural Networks 7, 113-127 (1994).
6. E. Oja, The nonlinear PCA learning rule in independent component analysis, Neurocomputing 17, 25-45 (1997).
7. P. Comon, Independent component analysis - A new concept?, Signal Processing 36 (3), 287-314 (1994).
8. R. Lambert, Multichannel blind deconvolution: FIR matrix algebra and separation of multipath mixtures, Thesis, University of Southern California, Department of Electrical Engineering, (1996).
9. J. Herault and J. Jutten, Space or time adaptive signal processing by neural network models, In Neural Networks for Computing: AIP Conference Proceedings 151, (Edited by J.S. Denker), American Institute for Physics, New York, (1986).
10. C. Jutten and J. Herault, Blind separation of sources, Part I: An adaptive algorithm based on neuromimetic architecture, Signal Processing 24, 1-10 (1991).
11. A. Cichocki, R. Unbehauen and E. Rummert, Robust learning algorithm for blind separation of signals, Electronics Letters 30 (17), 1386-1387 (1994).
12. R. Linsker, Local synaptic learning rules suffice to maximize mutual information in a linear network, Neural Computation 4, 691-702 (1992).
13. H.B. Barlow, Possible principles underlying the transformation of sensory messages, In Sensory Communication, (Edited by W.A. Rosenblith), MIT Press, (1961).
14. J.J. Atick, Could information theory provide an ecological theory of sensory processing?, Networks 3, 213-251 (1992).
15. J.P. Nadal and N. Parga, Non-linear neurons in the low-noise limit: A factorial code maximizes information transfer, Network 4, 295-312 (1994).
16. Z. Roth and Y. Baram, Multidimensional density shaping by sigmoids, IEEE Trans. on Neural Networks 7 (5), 1291-1298 (1996).
17. J.-F. Cardoso and B. Laheld, Equivariant adaptive source separation, IEEE Trans. on Signal Processing 45 (2), 434-444 (1996).
18. M. Gaeta and J. Lacoume, Sources separation without a priori knowledge: The maximum likelihood solution, Eusipco 90, Barcelona, pp. 621-624 (1990).
19. M. Girolami and C. Fyfe, Generalised independent component analysis through unsupervised learning with emergent Bussgang properties, In Proc. International Conference on Neural Networks, Houston, TX, pp. 1788-1891, (1997).
20. M. Girolami and C. Fyfe, Extraction of independent signal sources using a deflationary exploratory projection pursuit network with lateral inhibition, I.E.E. Proceedings on Vision, Image and Signal Processing Journal 14 (5), 299-306 (1997).
21. T. Cover and J. Thomas, Elements of Information Theory, John Wiley and Sons, New York, (1991).
22. L. Xu, Least MSE reconstruction: A principle for self organizing nets, Neural Networks 6, 627-648 (1993).
23. T.-W. Lee, M. Girolami. and T. Sejnowski, Independent component analysis using an extended infomax algorithm for mixed sub-Gaussian and super-Gaussian sources, Neural Computation (submitted).
24. S. Amari, Natural gradient works efficiently in learning, Neural Computation (to appear).
25. L. Molgedey and H. Schuster, Separation of independent signals using time-delayed correlations. Physical Review Letters 72 (23), 3634-3637 (1994).
26. nd edition, McGraw-Hill, New York, (1984).
27. S. Amari and J.-F. Cardoso, Blind source separation - Semiparametric statistical approach, IEEE Trans. on Signal Processing 45, 2692-2700 (1997).
28. S. Amari, A. Cichocki and H. Yang, A new learning algorithm for blind signal separation, In Advances in Neural Information Processing Systems 8, pp. 757-763, (1996).
29. M. Girolami, An alternative perspective on adaptive independent component analysis algorithms, Technical Report, Computing and Information Systems, Paisley University, Scotland, ISSN 1461-6122, (1997).
30. J.-F. Cardoso, Blind signal processing: Statistical principles, In Proceedings of the IEEE, 86, pp. 2009-2025, (1998).
31. J.-F. Cardoso, Unsupervised Adaptive Filtering, Chapter "Entropic contrasts for source separation", (Edited by S. Haykin), First presented at the NIPS*96 on 'Blind Signal Processing' organized by A. Cichocki, (1998).
32. M. Girolami, Self-organizing artificial neural networks for signal separation, Ph.D. Thesis, Department of Computing and Information Systems, Paisley University, Scotland, (1997).
33. J.H. Friedman, Exploratory projection pursuit, Journal of the American Statistical Association 82 (397), 249-266 (1987).
34. B.S. Everitt, An Introduction to Latent Variable, Chapman and Hall, London, (1984).
35. M.C. Jones and R. Sibson, What is projection pursuit, The Royal Statistical Society A150, 1-36 (1987).
36. A. Stuart and J.K. Ord, Kendall's Advanced Theory of Statistic, 1, Distribution Theory, John Wiley, New York, (1987).
37. th Joint Symposium on Neural Computation, Vol. 7, pp. 132-140, Institute for Neural Computation, (1997).
38. T.-W. Lee, Independent Component Analysis: Theory and Applications, Kluwer Academic, Boston, (1998).
39. M. McKeown, T.-P. Jung, S. Makeig, G. Brown, S. Kindermann, T.-W. Lee and T. Sejnowski, Transiently time-locked fMRI activations revealed by independent component analysis, Proceedings of the National Academy of Sciences 95, 803-810 (1997).
40. E. Oja, The nonlinear PCA learning rule in independent component analysis, Neurocomputing 17, 25-45 (1997).
41. J. Karhunen, E. Oja, L. Wang, R. Vigario and J. Joutsensalo, A class of neural networks for independent component analysis, IEEE Trans. on Neural Networks 8, 487-504 (1997).
42. E. Oja and J. Karhunen, Signal separation by nonlinear Hebbian learning, In Proceedings IEEE ICNN 95, pp. 83-87, (1995).
43. J. Karhunen, L. Wang and R. Vigario, Nonlinear PCA type approach for source separation and independent component analysis, In Proc. ICNN, Perth, Australia, pp. 995-1000, (1995).
44. S. Bellini, Blind deconvolution, In Bussgang Techniques for Blind Deconvolution and Equalization, (Edited by Haykin), Prentice Hall, New York, (1994).
45. R. Lambert and A. Bell, Blind separation of multiple speakers in a multipath environment, In ICASSP, Munich, Germany, April 21-26, pp. 423-426, (1997).
46. T.-W. Lee, A. J. Bell and R. Orglmeister, Blind source separation of real-world signals, IEEE Proc. ICNN, Houston, TX, pp. 2129-2135 (1997).
47. S. Bellini, Bussgang techniques for blind deconvolution and equalization, In Blind Deconvolution, (Edited by S. Haykin), Prentice Hall, (1994).
48. E. Moulines, J.-F. Cardoso and E. Gassiat, Maximum likelihood for blind separation and deconvolution of noisy signals using mixture models, Proc. ICASSP'97, Munich, (5), pp. 3617-3620, (1997).
49. L. Xu, C. Cheung, H. Yang and S. Amari, Maximum equalization by entropy maximization and mixture of cumulative distribution functions, IEEE Proc. ICNN, Houston, TX, pp. 1821-1826, (1997).
50. S. Makeig, T. Jung, A.J. Bell, D. Ghahremani and T.J. Sejnowski, Blind separation of event-related brain response into spatial independent components, Proc. of the National Academy of Sciences 94, 10979-10984 (1997).
51. H. Yang and S. Amari, Adaptive on-line learning algorithms for blind separation - Maximum entropy and minimum mutual information, Neural Computation 9, 1457-1482 (1997).
52. D.T. Pham and P. Garrat, Blind separation of mixture of independent sources through a quasi-maximum likelihood approach, IEEE Trans. on Signal Proc. 45, 1712-1725 (1997).
53. S. Amari, T.-P. Chen and A. Cichocki, Stability analysis of adaptive blind source separation, Neural Networks 10 (8), 1345-1352 (1997).
54. S. Makeig, T.-P. Jung, A. Bell, D. Ghahramani and T. Sejnowski, Independent component analysis of electroencephalographic data, Proceedings of the National Academy of Sciences 94, 10979-10984 (1997).
55. T.-P. Jung, C. Humphries, T.-W. Lee, S. Makeig, M. McKeown, V. Iragui and T. Sejnowski, Extended ICA removes artifacts from electroencephalographic recordings, Advances in Neural Information Processing Systems (to appear).
56. S. Makeig, A. Bell, T.-P. Jung and T. Sejnowski, Independent component analysis of electroencephalographic data, In Advances in Neural Information Processing Systems 8, pp. 145-151, MIT Press, Cambridge, MA, (1996).
57. K. Torkkola, Blind separation of convolved sources based on information maximization, In IEEE Workshop on Neural Networks for Signal Processing, Kyoto, Japan, pp. 423-432, (1996).
58. D. Yellin and E. Weinstein, Multichannel signal separation: Methods and analysis, IEEE Transactions on Signal Processing 44 (1), 106-118 (1996).
59. M. Girolami and C. Fyfe, Extraction of independent signal sources using a deflationary exploratory projection pursuit network with lateral inhibition, I.E.E. Proceedings on Vision, Image and Signal Processing Journal 14 (5), 299-306 (1997).
60. A. Cichocki, S. Amari and J. Cao, Neural network models for blind separation of time delayed and convolved signals, IECE Trans. Fundamentals E82-A (1997).
61. H.-L. Nguyen Thi and C. Jutten, Blind source separation for convolutive mixtures, Signal Processing 45 (2) (1995).
62. S. Li and T.J. Sejnowski, Adaptive separation of mixed broadband sound sources with delays by a beam-forming Herault-Jutten network, IEEE Journal of Oceanic Engineering 20 (1), 73-79 (1994).
63. G. Burel, A non-linear neural algorithm, Neural Networks 5, 937-947 (1992).
64. H. Yang, S. Amari and A. Cichocki, Information back-propagation for blind separation of sources from non-Mor-linear mixtures, Proc. of ICNN'97, Houston, TX, pp. 2141-2146, (1997).
65. A. Taleb and C. Jutten, Nonlinear source separation: The post-non linear mixtures, ESANN'97, pp. 279-284, (1997).
66. T.-W. Lee, B. Köhler and R. Orglmeister, Blind source separation of nonlinear mixing models, IEEE Proc. NNSP, Florida, pp. 406-415, (1997).
67. P. Pajunen and J. Karhunen, A maximum likelihood approach to nonlinear blind source separation, In Proceedings of the 1997 Int. Conf. on Artificial Neural Networks (ICANN'97), Lausanne, pp. 541-546, (1997).
68. J. Lin and J. Cowan, Faithful representation of separable input distributions, Neural Computation 9 (6), 1305-1320 (1997).
69. C. Jutten and J.-F. Cardoso, Source separation: Really blind?, In Proc. NOLTA, pp. 79-84, (1995).
70. M. Lewicki and T. Sejnowski, Learning nonlinear overcomplete representations for efficient coding, Advances in Neural Information Processing Systems (to appear).
71. G. Hinton and Z. Ghahramani, Generative models for discovering sparse distributed representations, Philosophical Transactions Royal Society B 352, 1177-1190 (1997).
72. J.P. Nadal and N. Parga, Redundancy reduction and independent component analysis: Conditions on cumulants and adaptive approaches, Neural Computation 9, 1421-1456 (1997).
73. N. Murata, K.-R. Mueller, A. Ziehe and S. Amari, Adaptive on-line learning in changing environments, In Advances in Neural Information Processing Systems 9, pp. 599-605. MIT Press, (1997).
74. J. Hopfield, Olfactory computation and object perception, Proc. Natl. Acad. Sci. USA 88, 6462-6466 (1991).
75. Kauer and White, personal communication.
76. K. Torkkola, Blind separation of radio signals in fading channels, In Advances in Neural Information Processing Systems 10, MIT Press (to appear).
77. W. Lizhong and J. Moody, Multi-effect decompositions for financial data modeling, In Advances in Neural Information Processing Systems 9, pp. 995-1001, MIT Press, (1997).
78. M. Girolami, A. Cichocki and S. Amari, A common neural network model for exploratory data analysis and independent component analysis, Technical Report, bip-97-001, Brain Information Processing Group, RIKEN, Japan, (1997).
79. R. Linsker, A local learning rule that enables information maximization for arbitrary input distributions, Neural Computation 9, 1661-1665 (1997).
80. M. Girolami and C. Fyfe, A temporal model of linear anti-Hebbian learning, Neural Processing Letters Journal 4 (3), 1-10 (1997).
81. F. Field, What is the goal of sensory coding?, Neural Computation 6, 559-601 (1994).
82. A.J. Bell and T.J. Sejnowski, The 'independent components' of natural scenes are edge filters, Vision Research 37 (23), 3327-3338 (1997).
83. M. Bartlett and T.J. Sejnowski, Viewpoint invariant face recognition using independent component analysis and attractor networks, In Advances in Neural Information Processing Systems 9, pp. 817-823, MIT Press, (1997).
84. M.H. Cohen and A.G. Andreou, Current-mode subthreshold MOS implementation of the Herault-Jutten autoadaptive network, IEEE J. Solid-State Circuits 27 (5), 714-727 (1992).
85. ndedition, Prentice-Hall, (1991).
86. A. Hyvärinen and E. Oja, A fast fixed-point algorithm for independent component analysis, Neural Computation 9, 1483-1492 (1997).
87. th European Symposium on Artificial Neural Networks, Bruges, Belgium, pp. 249-266, (1996).
88. P. Pajunen, Blind source separation of binary sources with less sensors than sources, Proc. of ICNN'97, Houston, TX, pp. 1994-1997, (1997).
89. D.T. Pham, P. Garrat and C. Jutten, Separation of a mixture of independent sources through a maximum likelihood approach, Proc. EUSIPCO, pp. 771-774, (1992).
90. D.T. Pham, Blind separation of instantaneous mixture of sources via an independent component analysis, IEEE Trans. on Signal Proc. 44 (11), 2768-2779 (1996).
91. R. Vigario, A. Hyvärinen and E. Oja, ICA fixed-point algorithm in extraction of artifacts from EEG, In Proc. 1996 IEEE Nordic Signal Processing Symposium NORSIG'96, Espoo, Finland, pp. 383-386, (1996).