Causality detection based on information-theoretic approaches in time series analysis

Physics Reports

Volumn 441, Issue 1, 2007, Pages 1-46

  Hlaváčková Schindler, Katerina ^a   Paluš, Milan ^b   Vejmelka, Martin ^b   Bhattacharya, Joydeep ^a,c  

^a ERICH SCHMID INSTITUTE OF MATERIALS SCIENCE   (Austria)

^b INSTITUTE OF COMPUTER SCIENCE   (Czech Republic)

^c UNIVERSITY OF LONDON   (United Kingdom)

Author keywords

Causality; Entropy; Estimation; Mutual information

Indexed keywords

EID: 33947524701     PISSN: 03701573     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.physrep.2006.12.004     Document Type: Review

References (203)

1. H.D.I. Abarbanel, Introduction to Nonlinear Dynamics for Physicists, Lecture Notes in Physics, World Scientific, Singapore, 1993.
2. M. Abramowitz, I.A. Stegun, Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables, 9th printing, Dover, New York, 1972, pp. 946-949.
3. Ahmad I.A., and Lin P.E. A nonparametric estimation of the entropy for absolutely continuous distributions. IEEE Trans. Inform. Theory 22 (1976) 372-375
4. Ahmed N.A., and Gokhale D.V. Entropy expressions and their estimators for multivariate distributions. IEEE Trans. Inform. Theory 35 (1989) 688-692
5. S. Akaho, Y. Kiuchi, S. Umeyama, MICA: multimodal independent component analysis, Proc. IJCNN (1999) 927-932.
6. Amari S., Cichocki A., and Yang H.H. A new learning algorithm for blind signal separation. In: Touretzky D.S., Mozer M.C., and Hasselmo M.E. (Eds). Advances in Neural Information Processing Systems vol. 8 (1996), MIT Press, Cambridge, MA 757-763
7. Ancona N., Marinazzo D., and Stramaglia S. Radial basis function approach to nonlinear Granger causality of time series. Phys. Rev. E 70 (2004) 056221
8. Ancona N., and Stramaglia S. An invariance property of predictors in kernel-induced hypothesis spaces. Neural Comput. 18 (2006) 749-759
9. Antos A., and Kontoyiannis I. Convergence properties of functional estimates for discrete distributions, Random Structures and Algorithms. Special issue: Average-Case Analysis of Algorithms 19 (2002) 163-193
10. Aparicio F.M., and Escribano A. Information-theoretic analysis of serial dependence and cointegration. Stud. Nonlinear Dynam. Econometr. 3 (1998) 119-140
11. Arnold B.C. Pareto Distributions (1985), International Co-Operative Publishing House, Burtonsvile, MD
12. Arnhold J., Grassberger P., Lehnertz K., and Elger C.E. A robust method for detecting interdependences: application to intracranially recorded EEG. Physica D 134 (1999) 419-430
13. E.G. Baek, W.A. Brock, A general test for nonlinear Granger causality: Bivariate model, Working paper, Iowa State University and University of Wisconsin, Madison, 1992.
14. Baghli M. A model-free characterization of causality. Econ. Lett. 91 (2006) 380-388
15. Basharin G.P. On a statistical estimate for the entropy of the sequence of independent random variables. Theory Probab. Appl. 4 (1959) 333-338
16. Beirlant J., and van Zuijlen M.C.A. The empirical distribution function and strong laws for functions of order statistics of uniform spacings. J. Multivar. Anal. 16 (1985) 300-317
17. Beirlant J. Limit theory for spacing statistics from general univariate distributions. Pub. Inst. Stat. Univ. Paris XXXI Fasc. 1 (1986) 27-57
18. Beirlant J., Dudewitz E.J., Györfi L., and van der Meulen E.C. Nonparametric entropy estimation: an overview. Int. J. Math. Statist. Sci. 6 (1997) 17-39
19. Bell D., Kay J., and Malley J. A non-parametric approach to non-linear causality testing. Econ. Lett. 51 (1996) 7-18
20. A. Berger, The improved iterative scaling algorithm: a gentle introduction 〈http://www.cs.cmu.edu/afs/cs/user/aberger/www/ps/scaling.ps〉, 1997.
21. Bercher J.F., and Vignat C. Estimating the entropy of a signal with applications. IEEE Trans. Signal Process. 48 (2000) 1687-1694
22. Bickel P., and Breiman L. Sums of functions of nearest neighbor distances, moment bounds, limit theorems and a goodness of fit test. Ann. Statist. 11 (1983) 185-214
23. Bishop C.M. Neural Networks for Pattern Recognition (1995), Oxford University Press, New York
24. Blinnikov S., and Moessner R. Expansions for nearly Gaussian distributions. Astronom. Astrophys. Suppl. Ser. 130 (1998) 193-205
25. Blinowska K.J., Kuś R., and Kamiński M. Granger causality and information flow in multivariate processes. Phys. Rev. E 70 (2004) 050902R
26. Boccaletti S., Kurths J., Osipov G., Valladares D.L., and Zhou C.S. The synchronization of chaotic systems. Phys. Rep. 366 (2002) 1-101
27. Bračič Lotrič M., and Stefanovska A. Synchronization and modulation in the human cardiorespiratory system. Physica A 283 (2000) 451-461
28. Brea J., Russell D.F., and Neiman A.B. AB Measuring direction in the coupling of biological oscillators: a case study for electroreceptors of paddlefish. Chaos 16 (2006) 026111
29. Brooks C. Predicting stock index volatility: can market volume help?. J. Forecast. 17 (1998) 59-80
30. Broomhead D.S., and Lowe D. Multivariate functional interpolation and adaptive networks. Complex Systems 2 (1988) 321-355
31. A.J. Butte, I.S. Kohane, Mutual information relevance networks: functional genomic clustering using pairwise entropy measurements, Pac. Symp. Biocomput. (2000) 418-429.
32. Cellucci C.J., Albano A.M., and Rapp P.E. Statistical validation of mutual information calculations: comparison of alternative numerical algorithm. Phys. Rev. E 71 (2005) 066208
33. Chávez M., Martinerie J., and Le Van Quyen M. Statistical assessment of nonlinear causality: application to epileptic EEG signals. J. Neurosci. Methods 124 (2003) 113-128
34. Chen Y., Rangarajan G., Feng J., and Ding M. Analyzing multiple nonlinear time series with extended Granger causality. Phys. Lett. A 324 (2004) 26-35
35. Cornfeld I.P., Fomin S.V., and Sinai Y.G. Ergodic Theory (1982), Springer, New York
36. Cover T., and Thomas J. Elements of Information Theory (1991), Wiley, New York (Chapter 9)
37. Cover T., and Thomas J. Elements of Information Theory (1991), Wiley, New York (Chapter 5)
38. Cramer H. On the composition of elementary errors. Skand. Aktuarietidskr. 11 (1928) 13-14, 141-180
39. Cressie N. On the logarithm of higher order spacings. Biometrika 63 (1976) 343-355
40. Darbellay G., and Vajda I. Estimation of the information by an adaptive partitioning of the observation space. IEEE Trans. Inform. Theory 45 (1999) 1315-1321
41. Darbellay G. An estimator of the mutual information based on a criterion of independence. Comput. Statist. Data Anal. 32 (1999) 1-17
42. Daub C., Steuer R., Selbig J., and Kloska S. Estimating mutual information using B-spline functions-an improved similarity measure for analysing gene expression data. BMC Bioinformatics 5 118 (2004) 1-12
43. De Boor C. A Practical Guide to Splines (1978), Springer, New York
44. Dempster A., Laird N., and Rubin D. Maximum likelihood from incomplete data via the EM algorithm. J. Roy. Statist. Soc. B 39 (1977) 138
45. Diks C., and DeGoede J. A general nonparametric bootstrap test for Granger causality. In: Broer, Krauskopf, and Vegter (Eds). Global Analysis of Dynamical Systems (2001) 391-403 (Chapter 16)
46. Diks C., and Mudelsee M. Redundancies in the Earth's climatological time series. Phys. Lett. A 275 (2000) 407-414
47. Diks C., and Panchenko V. A note on the Hiemstra-Jones test for Granger non-causality. Stud. Nonlinear Dynam. Econometr. 9 (2005) 4/1-7
48. Diks C., and Panchenko V. A new statistic and practical guidelines for nonparametric Granger causality testing. J. Econom. Dynam. Control 30 (2006) 1647-1669
49. Dmitriev Y.G., and Tarasenko F.P. On the estimation functions of the probability density and its derivatives. Theory Probab. Appl. 18 (1973) 628-633
50. Dobrushin R.L. A simplified method of experimentally evaluating the entropy of a stationary sequence. Teoriya Veroyatnostei i ee Primeneniya 3 (1958) 462-464
51. Dudewitz E.J., and Van der Meulen E.C. Entropy-based test of uniformity. J. Amer. Statist. Assoc. 76 (1981) 967-974
52. Efron B., and Stein C. The jackknife estimate of variance. Ann. Statist. 9 (1981) 586-596
53. D. Erdogmus, Information theoretic learning: Renyi's Entropy and its application to adaptive system training, Ph.D. Thesis, University of Florida, 2002.
54. Erdogmus D., and Principe J. An error-entropy minimization algorithm for supervised training of nonlinear adaptive systems. IEEE Trans. Signal Processing 50 (2002) 1780-1786
55. Fisher R.A. On an absolute criterion for fitting frequency curves. Messenger Math. 41 (1912) 155-160
56. Fisher R.A. On the mathematical foundations of theoretical statistics. Philos. Trans. Roy. Soc. London Ser. A 222 (1922) 309-368
57. Fraser A., and Swinney H. Independent coordinates for strange attractors from mutual information. Phys. Rev. A 33 (1986) 1134-1140
58. Fraser A. Information and entropy in strange attractors. IEEE Trans. Inform. Theory 35 (1989) 245-262
59. Gaztanga E., Fosalba P., and Elizande E. Gravitational evolution of the large-scale probability density distribution: the Edgeworth and Gamma expansions. Astrophys. J. 539 1 (2000) 522-531
60. A. German, J.B. Carlin, H.S. Stern, D.B. Rubin, Bayesian Data Analysis, Chapman & Hall/A CRC Press Company, Texts in Statistical Science Series, 2004.
61. Geweke J. Inference and causality in economic time series models. In: Griliches Z., and Intriligator M.D. (Eds). Handbook of Econometrics vol. 2 (1984), North-Holland 1101-1144
62. Gelfand I.M., and Yaglom A.M. Calculation of the amount of information about a random function contained in another such function. Amer. Math. Soc. Transl. 12 (1959) 199-236
63. Goria M.N., Leonenko N.N., Mergel V.V., and Novi Inverardi P.L. A new class of random vector entropy estimators and its applications in testing statistical hypotheses. Nonparametr. Statist. 17 (2005) 277-297
64. Granger C.W.J. Investigating causal relations by econometric and cross-spectral methods. Econometrica 37 (1969) 424-438
65. Granger C.W.J., and Newbold P. Forecasting Economic Time Series (1977), Academic Press, New York
66. Granger C.W.J. Testing for causality: a personal viewpoint. J. Econ. Dynam. Control 2 (1980) 329-352
67. C.W.J. Granger, Time series analysis, cointegration, and applications. Nobel Lecture, December 8, 2003, in: Les Prix Nobel. The Nobel Prizes 2003, ed. Tore Frängsmyr, [Nobel Foundation] (Stockholm, 2004) pp. 360-366. http://nobelprize.org/nobel_prizes/economics/laureates/2003/granger-lect ure.pdf.
68. Grassberger P. An optimized box-assisted algorithm for fractal dimensions. Phys. Lett. A 148 (1990) 63-68
69. P. Grassberger, I. Procaccia, Measuring of strangeness of strange attractors, Physica D (1983) 189-208.
70. Grassberger P. Finite sample corrections to entropy and dimension estimates. Phys. Lett. A 128 (1988) 369-373
71. P. Grassberger, Entropy Estimates from Insufficient Samplings, Arxiv preprint physics/0307138, (2003), arxiv.org.
72. Green M.L., and Savit R. Dependent variables in broad band continuous time series. Physica D 50 (1991) 521-544
73. Györfi L., and Van der Meulen E.C. Density-free convergence properties of various estimators of entropy. Comput. Statist. Data Anal. 5 (1987) 425-436
74. Györfi L., and Van der Meulen E.C. An entropy estimate based on a kernel density estimation. In: Berkes I., Csaki E., and Revesz P. (Eds). Limit Theorems in Probability and Statistics (1989), North-Holland, Amsterdam 229-240
75. Györfi L., and Van der Meulen E.C. On nonparametric estimation of entropy functionals. In: Roussas G. (Ed). Nonparametric Functional Estimation and Related Topics (1990), Kluwer Academic Publisher, Amsterdam 81-95
76. Hall P. Limit theorems for sums of general functions of m-spacings. Math. Proc. Camb. Philos. Soc. 96 (1984) 517-532
77. Hall P. On Kullback-Leibler loss and density estimation. Ann. Statist. 15 (1987) 1491-1519
78. Hall P. The Bootstrap and Edgeworth Expansion (1992), Springer, New York
79. Hall P., and Morton S.C. On the estimation of entropy. Ann. Inst. Statist. Math. 45 (1993) 69-88
80. Halsey T.C., Jensen M.H., Kadanoff L.P., Procaccia I., and Shraimann B. Fractal measures and their singularities: the characterization of strange sets. Phys. Rev. A 33 (1986) 1141
81. Haykin S. Neural Networks: A Comprehensive Foundation. second ed. (1998), Prentice-Hall, Englewood Cliffs, NJ
82. Hentschel H.G.E., and Procaccia I. The infinite number of dimensions of probabilistic fractals and strange attractors. Physica D 8 (1983) 435
83. Herzel H. Complexity of symbol sequences. Syst. Anal. Model. Simul. 5 (1988) 435-444
84. Hiemstra C., and Jones J.D. Testing for linear and nonlinear Granger causality in the stock price-volume relation. J. Finance 49 (1994) 1639-1664
85. Hinrichs H., Heinze H.J., and Schoenfeld M.A. Causal visual interactions as revealed by an information theoretic measure and fMRI. NeuroImage 31 (2006) 1051-1060
86. Hinton G., and Sejnowski T. Learning and relearning in Boltzmann machines. In: Rumelhart D., and McClelland J. (Eds). Parallel Distributed Processing vol. 1 (1986), MIT Press, Cambridge 282-317 (Chapter 7)
87. K. Hlaváčková-Schindler, Some comparisons of the complexity of neural network models and linear methods by means of metric entropy, in: Andrýsek, Kárný, Kracík (Eds.), Multiple Participant Decision Making, Int. Ser. Adv. Intell. 9 (2004) 149-160.
88. Ho M.-Ch., Hung Y.-Ch., and Jiang I.-M. Phase synchronization in inhomogeneous globally coupled map lattices. Phys. Lett. A 324 (2004) 450-457
89. Hoekstra B.P.T., Diks C.G.H., Allessie M.A., and DeGoede J. Non-linear time series analysis: methods and applications to atrial fibrillation. Ann. Ist. Super. Sanita 37 (2003) 325-333
90. Hua-Yang H., and Amari S. Adaptive on-line learning algorithms for blind separation-maximum entropy and minimum mutual information. Neural Comput. 9 (1997) 1457-1482
91. Ihara S. Information Theory for Continuous Systems (1993), World Scientific, Singapore
92. Ivanov A.V., and Rozhkova A. Properties of the statistical estimate of the entropy of a random vector with a probability density. Probl. Inform. Transm. 10 (1981) 171-178
93. Jamšek J., Stefanovska A., and McClintock P.V.E. Nonlinear cardio-respiratory interactions revealed by time-phase bispectral analysis. Phys. Med. Biol. 49 (2004) 4407-4425
94. Joe H. On the estimation of entropy and other functionals of a multivariable density. Ann. Inst. Statist. Math. 41 (1989) 638-697
95. Jones M.C., and Sibson R. What is projection pursuit?. J. Roy. Statist. Soc. London Ser. A 150 (1987) 1-36
96. Kagan A.M., Linnik Y.V., and Rao C.R. Characterization Problems in Mathematical Statistics (1973), Wiley, New York
97. Kamiński M., Ding M., Truccolo W.A., and Bressler S.L. Evaluating causal relations in neural systems: Granger causality, directed transfer function and statistical assessment of significance. Biol. Cybern. 85 (2001) 145-157
98. Katura T., Tanaka N., Obata A., Sato H., and Maki A. Quantitative evaluation of interrelations between spontaneous low-frequency oscillations in cerebral hemodynamics and systemic cardiovascular dynamics. Neuroimage 31 (2006) 1592-1600
99. K.H. Knuth, A. Gotera, C.T. Curry, K.A. Huyser, K.R. Wheeler, W.B. Rossow, Revealing relationships among relevant climate variables with information theory, Earth-Sun System Technology Conference 2005, Adelphi, MD, 2005.
100. Kolmogorov A.N. Entropy per unit time as a metric invariant of automorphism. Dokl. Akad. Nauk SSSR 124 (1959) 754-755
101. Kozachenko L.F., and Leonenko N.N. Sample estimate of the entropy of a random vector. Probl. Inform. Transm. 23 (1987) 95-100
102. Kraskov A., Stögbauer H., and Grassberger P. Estimation mutual information. Phys. Rev. E 69 (2004) 066138
103. Kullback S., and Leibler R.A. On information and sufficiency. Ann. Math. Statist. 22 (1951) 79-86
104. N. Leonenko, L. Pronzato, V. Savani, A class of Rényi information estimators for multidimensional densities, Laboratoire I3S, CNRS-Université de Nice-Sophia Antipolis, Technical Report I3S/RR-2005-14-FR, 2005.
105. J.J. Lin, N. Saito, R.A. Levine, Edgeworth approximations of the Kullbach-Leibler distance towards problems in image analysis, 2001, 〈http://www.math.ucdavis.edu/∼saito/publications/saito_ekld2.pd f〉.
106. Ma Y., and Kanas A. Testing for nonlinear Granger causality from fundamentals to exchange rates in the ERM. J. Int. Finance Markets, Inst. Money 10 (2000) 69-82
107. Marinazzo D., Pellicoro M., and Stramaglia S. Nonlinear parametric model for Granger causality of time series. Phys. Rev. E 73 (2006) 066216
108. Marschinski R., and Kantz H. Analysing the information flow between financial time series-an improved estimator for transfer entropy. Eur. Phys. J. B 30 (2002) 275-281
109. G. Miller, W. Madow, On the maximum likelihood estimate of the Shannon-Wiener measure of information, Air Force Cambridge Research Center Technical Report, vol. 75, 1954, pp. 54-75.
110. Miller G. Note on the bias of information estimates. In: Quastler H. (Ed). Information Theory in Psychology II-B (1955), Free Press, Glencoe, IL 95-100
111. Moddemeijer R. On estimation of entropy and mutual information of continuous distribution. Signal Processing 16 (1989) 233-246
112. Mokhov I.I., and Smirnov D.A. El Nino-Southern Oscillation drives North Atlantic Oscillation as revealed with nonlinear techniques from climatic indices. Geophys. Res. Lett. 33 (2006) L03708
113. Mokkadem A. Estimation of the entropy and information for absolutely continuous random variables. IEEE Trans. Inform. Theory 35 (1989) 195-196
114. Moon Y., Rajagopalan B., and Lall U. Estimation of mutual information using kernel density estimators. Phys. Rev. E 52 (1995) 2318-2321
115. Neal R., and Hinton G. A view of the EM algorithm that justifies incremental. In: Jordan M.I. (Ed). Learning in Graphical Models (1999), MIT Press, Cambridge, MA 355-368
116. Nemenman I., Shafee F., and Bialek W. Entropy and inference revisited. In: Dietterich T.G., Becker S., and Ghahramani Z. (Eds). Advances in Neural Information Processing Systems vol. 14 (2002), MIT Press, Cambridge, MA
117. Nemenman I., Bialek W., and de Ruyter van Stevenick R. Entropy and information in neural spike trains: progress on sampling problem. Phys. Rev. E 69 (2004) 056111
118. Nemenman I., and Bialek W. Occam factors and model-independent Bayesian learning of continuous distributions. Phys. Rev. E 65 (2002) 026137
119. Otsuka K., Miyasaka Y., and Kubota T. Formation of an information network in a self-pulsating multimode laser. Phys. Rev. E 69 (2004) 046201
120. Ottes R. A critique of Suppe's theory of probabilistic causality. Synthese 48 (1981) 167-189
121. Paluš M., and Stefanovska A. Direction of coupling from phases of interacting oscillators: an information-theoretic approach. Phys. Rev. E 67 (2003) 055201R
122. Paluš M., Komárek V., Procházka T., Hrnčíř Z., and Štěrbová K. Synchronization and information flow in EEG of epileptic patients. IEEE Eng. Med. Biol. Mag. 20 (2001) 65-71
123. Paluš M., Komárek V., Hrnčíř Z., and Štěrbová K. Synchronization as adjustment of information rates: detection from bivariate time series. Phys. Rev. E 63 (2001) 046211
124. Paluš M. Testing for nonlinearity using redundancies: quantitative and qualitative aspects. Physica D 80 (1995) 186-205
125. Paluš M. Coarse-grained entropy rates for characterization of complex time series. Physica D 93 (1996) 64-77
126. Paluš M. Kolmogorov entropy from time series using information-theoretic functional. Neural Network World 7 3 (1997) 269-292. http://www.uivt.cas.cz/mp/papers/rd1a.pdf 〈http://www.uivt.cas.cz/~mp/papers/rd1a.pdf〉
127. M. Paluš, Identifying and quantifying chaos by using information-theoretic functionals, in: A.S. Weigend, N.A. Gershenfeld (Eds.), Time Series Prediction: Forecasting the Future and Understanding the Past, Santa Fe Institute Studies in the Sciences of Complexity, Proceedings of vol. XV, Addison-Wesley, Reading, MA, 1993, pp. 387-413.
128. Paluš M. Detecting nonlinearity in multivariate time series. Phys. Lett. A 213 (1996) 138-147
129. Paluš M., and Hoyer D. Detecting nonlinearity and phase synchronization with surrogate data. IEEE Eng. Med. Biol. 17 (1998) 40-45
130. Paninski L. Estimation of entropy and mutual information. Neural Comput. 15 (2003) 1191-1253
131. Paninski L. Estimating entropy on m bins given fewer than m samples. IEEE Trans. Inform. Theory 50 (2004) 2200-2203
132. Parzen E. On estimation of a probability density function and mode. Time Series Analysis Papers (1967), Holden-Day Inc., San Diego, California
133. Pearl J. Causality: Models, Reasoning and Inference (2000), Cambridge University Press, New York
134. Penrose M. Central limit theorems for k-nearest neighbor distances. Stochastic Processes Appl. 85 (2000) 295-320
135. Petersen K. Ergodic Theory (1983), Cambridge University Press, Cambridge
136. Peters M.A., and Iglesias P.A. Minimum entropy control for discrete-time varying systems. Automatica 33 (1997) 591-605
137. Pflieger M.E., and Greenblatt R.E. Using conditional mutual information to approximate causality for multivariate physiological time series. Int. J. Bioelectromagn. 7 (2005) 285-288
138. Pikovsky A., Rosenblum M., and Kurths J. Synchronization. A Universal Concept in Nonlinear Sciences (2001), Cambridge University Press, Cambridge
139. Pinsker M.S. Information and Information Stability of Random Processes (1964), Holden Day, San Francisco
140. Pompe B. Measuring statistical dependencies in a time series. J. Statist. Phys. 73 (1993) 587-610
141. Poschel T., Ebeling W., and Rose H. Dynamic entropies, long-range correlations and fluctuations in complex linear structures. J. Statist. Phys. 80 (1995) 1443-1452
142. Prasaka Rao B.L.S. Nonparametric Functional Estimation (1983), Academic Press, New York
143. Prichard D., and Theiler J. Generalized redundancies for time series analysis. Physica D 84 (1995) 476-493
144. Principe J.C., Fischer J.W., and Xu D. Information theoretic learning. In: Haykin S. (Ed). Unsupervised Adaptive Filtering (2000), Wiley, New York 265-319
145. J.C. Principe, D. Xu, Q. Zhao, J.W. Fischer, Learning from examples with information theoretic criteria, J. VLSI Signal Processing Systems, Special Issue on Neural Networks (2000) 61-77.
146. Pronzato L., Wynn H.P., and Zhigljavsky A.A. Using Renyi entropies to measure: uncertainty in search problems. Lect. Appl. Math. AMS 33 (1997) 253-268
147. Le Van Quyen M., Martinerie J., Adam C., and Varela F.J. Nonlinear analyses of interictal EEG map the brain interdependences in human focal epilepsy. Physica D 127 (1999) 250-266
148. Quian Quiroga R., Arnhold J., and Grassberger P. Learning driver-response relationships from synchronization patterns. Phys. Rev. E 61 5 (2000) 5142-5148
149. Reid N. Asymptotic expansions. In: Kotz S., Read C.B., and Banks D.L. (Eds). Encyclopedia for Statistical Sciences, Update Volume (1996), Wiley, New York 32-39
150. A. Rényi, On measures of entropy and information, in: Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, vol. 1, University of California Press, Berkeley, CA, 1961, pp. 547-561.
151. A. Rényi, Some fundamental questions of information theory, Selected Papers of Alfred Rényi, vol. 2, Akademia Kiado, Budapest, 1976, pp. 526-552.
152. Rigoll G. Maximum mutual information neural networks for hybrid connectionist-HMM speech recognition systems. IEEE Trans. Speech Audio Processing, Special Issue on Neural Networks for Speech 2 (1994) 175-184
153. Rissanen J., Speed T.P., and Yu B. Density estimation by stochastic complexity. IEEE Trans. Inform. Theory 38 (1992) 315-323
154. F. Rossi, A. Lendasse, D. Francois, V. Wertz, M. Verleysen, Mutual information for the selection of relevant variables in spectrometric nonlinear modeling, in: Chemometrics and Intelligent Laboratory Systems, Lille, France, November 30-December 1, 2005, pp. 52-55.
155. Russel B. On the notion of cause. Proc. Aristotelian Soc. New Ser. 13 (1913) 1-26
156. Schäfer C., Rosenblum M.G., Kurths J., and Abel H.H. Heartbeat synchronized with ventilation. Nature 392 (1998) 239-240
157. Schäfer C., Rosenblum M.G., Kurths J., and Abel H.H. Synchronization in the human cardiorespiratory system. Phys. Rev. E 60 (1999) 857-870
158. Schiff S.J., So P., Chang T., Burke R.E., and Sauer T. Detecting dynamical interdependence and generalized synchrony through mutual prediction in a neural ensemble. Phys. Rev. E 54 (1996) 6708-6724
159. Schmitt A.O., Herzel H., and Ebeling W. A new method to calculate higher-order entropies from finite samples. Europhys. Lett. 23 (1993) 303-309
160. Schmitz A. Measuring statistical dependence and coupling of subsystems. Phys. Rev. E 62 (2000) 7508-7511
161. N. Schraudolph, Optimization of entropy with neural networks, Ph.D. Thesis, University of California, San Diego, USA, 1995.
162. Schraudolph N. Gradient-based manipulation of non-parametric entropy estimates. IEEE Trans. Neural Networks 14 (2004) 828-837
163. Schreiber T. Measuring information transfer. Phys. Rev. Lett. 85 (2000) 461-464
164. Schürmann T. Bias analysis in entropy estimation. J. Phys. A Math. Gen. 37 (2004) L295-L301
165. Selltiz C., Wrightsman L.S., and Cook S.W. Research Methods in Social Relations (1959), Holt, Rinehart and Winston, New York
166. Shannon C.E. A mathematical theory of communication. Bell System Tech. J. 27 (1948) 379-423
167. Shaw R. Strange attractors, chaotic behavior and information flow. Z. Naturforsch 36A (1981) 80-112
168. Shaw R.S. The Dripping Faucet as a Model Chaotic System (1985), Aerial, Santa Cruz
169. Silvapulle P., and Choi J.S. Testing for linear and nonlinear Granger causality in the stock price-volume relation. Korean evidence. Quart. Rev. Econ. Finan. 39 (1999) 59-76
170. Silverman B.W. Density Estimation (1986), Chapman & Hall, London
171. Sinai Y.G. On the concept of entropy for a dynamic system. Dokl. Akad. Nauk SSSR 124 (1959) 768-771
172. Sinai Y.G. Introduction to Ergodic Theory (1976), Princeton University Press, Princeton, NJ
173. Sorjamaa A., Hao J., and Lendasse A. Mutual information and k-nearest neighbors approximator for time series prediction. Proceedings of ICANN 2005, Lecture Notes in Computer Science vol. 3697 (2005) 553-558
174. Stefanovska A., Haken H., McClintock P.V.E., Hožič M., Bajrović F., and Ribarič S. Reversible transitions between synchronization states of the cardiorespiratory system. Phys. Rev. Lett. 85 (2000) 4831-4834
175. Steuer R., Kurths J., Daub C.O., Weise J., and Selbig J. The mutual information. detecting and evaluating dependencies between variables. Bioinformatics 18 Suppl. 2 (2002) S231-S240
176. Strong S.P., Koberle R., de Ruyter van Steveninck R.R., and Bialek W. Entropy and information in neural spike trains. Phys. Rev. Lett. 80 (1998) 197-202
177. H.J. Su, H. White, A nonparametric Hellinger metric test for conditional independence, Technical Report, Department of Economics, University of California, San Diego, 2003.
178. F. Takens, Detecting strange attractors in turbulence, in: D.A. Rand, L.S. Young (Eds.), Dynamical Systems and Turbulence, Springer, Berlin, 1981, pp. 366-381.
179. Takens F. Invariants related to dimension and entropy. Atas do 13 Coloquio Brasileiro de Mathematica (1983), Rio de Janeiro, Brasil
180. Taleb A., and Jutten C. Entropy optimization-application to source separation. Lecture Notes in Computer Science vol. 1327 (1997) 529-534
181. Tarasenko F.P. On the evaluation of an unknown probability density function, the direct estimation of the entropy from independent observations of a continuous random variable, and the distribution-free entropy test of goodness-of-fit. Proc. IEEE 56 (1968) 2052-2053
182. Tass P., Rosenblum M.G., Weule J., Kurths J., Pikovsky A., Volkmann J., Schnitzler A., and Freund H.-J. Detection of n:m phase locking from noisy data: application to magnetoencephalography. Phys. Rev. Lett. 81 (1998) 3291-3294
183. Theiler J. Estimating fractal dimension. J. Opt. Soc. Amer. A 7 (1990) 1055-1071
184. Theodoridis S., and Koutroumbas K. Pattern Recognition (1999), Academic Press, San Diego
185. Torkkola K. Feature extraction by non-parametric mutual information maximization. J. Mach. Learn. Res. 3 (2003) 1415-1438
186. Trappenberg T., Ouyang J., and Back A. Input variable selection. mutual information and linear mixing measures. IEEE Trans. Knowledge Data Eng. 18 (2006) 37-46
187. Triacca U. Is Granger causality analysis appropriate to investigate the relationships between atmospheric concentration of carbon dioxide and global surface air temperature?. Theor. Appl. Climatol. 81 (2005) 133-135
188. Tsybakov A.B., and van Meulen E.C. Root-n consistent estimators of entropy for densities with unbounded support. Scand. J. Statist. 23 (1994) 75-83
189. 〈http://mathworld.wolfram.com/UlamMap.html〉.
190. Van Es B. Estimating functional related to a density by a class of statistics based on spacings. Scand. J. Statist. 19 (1992) 61-72
191. Van Hulle M.M. Edgeworth approximation of multivariate differential entropy. Neural Comput. 17 (2005) 1903-1910
192. M.M. Van Hulle, Multivariate Edgeworth-based entropy estimation, in: Proceedings of the IEEE Workshop on Machine Learning for Signal Processing, Mystic, Connecticut, USA, September 28-30, 2005.
193. Vašíček O. A test for normality based on sample entropy. J. Roy. Statist. Soc Ser. B Methodol. 38 (1976) 54-59
194. M. Vejmelka, K. Hlaváčková-Schindler, Mutual Information Estimation in Higher Dimensions: A Speed-Up of a k-Nearest Neighbor Based Estimator, submitted to ICANNGA'07.
195. Verdes P.F. Assessing causality from multivariate time series. Phys. Rev. E 72 (2005) 026222
196. Verdugo Lazo A.C.G., and Rathie P.N. On the entropy of continuous probability distributions. IEEE Trans. Inform. Theory 24 (1978) 120-122
197. Victor J. Binless strategies for estimation of information from neural data. Phys. Rev. E 66 (2002) 051903
198. P. Viola, Alignment by maximization of mutual information, Ph.D. Thesis, MIT, 1995.
199. Viola P., Schraudolph N., and Sejnowski T. Empirical entropy manipulation for real-world problems. Advances in Neural Information Processing Systems (NIPS 8) (1996), The MIT Press, Cambridge, MA 851-857
200. Wiener N. The theory of prediction. In: Beckenbach E.F. (Ed). Modern Mathematics for Engineers (1956), McGraw-Hill, New York
201. 〈http://en.wikipedia.org/wiki/Causality〉.
202. Xu L., Cheung C., Yang H., and Amari S. Maximum equalization by entropy maximization and mixture of cumulative distribution functions. Proceedings of ICNN'97, Houston (1997) 1821-1826
203. Yang J., and Grigolini P. On the time evolution of the entropic index. Phys. Lett. A 263 (1999) 323-330