An introductory review of information theory in the context of computational neuroscience

Biological Cybernetics

Volumn 105, Issue 1, 2011, Pages 55-70

  McDonnell, Mark D ^a   Ikeda, Shiro ^b   Manton, Jonathan H ^c  

^a UNIVERSITY OF SOUTH AUSTRALIA   (Australia)

^b INSTITUTE OF STATISTICAL MATHEMATICS   (Japan)

^c UNIVERSITY OF MELBOURNE   (Australia)

Author keywords

Channel capacity; Information theory; Neural coding; Neuroscience

Indexed keywords

BIOLOGICAL PROCESS; COMPUTATIONAL NEUROSCIENCE; FUNDAMENTAL CONCEPTS; NEURAL CODING; NEURONAL NETWORKS; NEUROSCIENCE;

DATA PROCESSING; NEURAL NETWORKS;

INFORMATION THEORY;

BRAIN; COGNITION; HUMAN; INFORMATION SCIENCE; INTERPERSONAL COMMUNICATION; MATHEMATICS; NERVE CELL; NEUROSCIENCE; PHYSIOLOGY; REVIEW;

BRAIN; COGNITION; COMMUNICATION; HUMANS; INFORMATION THEORY; MATHEMATICS; NEURONS; NEUROSCIENCES;

EID: 79961169265     PISSN: 03401200     EISSN: 14320770     Source Type: Journal    
DOI: 10.1007/s00422-011-0451-9     Document Type: Review

References (64)

1. PO Amblard OJJ Michel 2011 On directed information theory and Granger causality graphs J Comput Neurosci 30 7 16 20333542 10.1007/s10827-010-0231-x
2. T Berger 2003 Living information theory: The 2002 Shannon lecture IEEE Inf Theory Soc Newsl 53 16 19
3. T Berger JD Gibson 1998 Lossy source coding IEEE Trans Inf Theory 44 2693 2723 10.1109/18.720552 (Pubitemid 128741904)
4. T Berger WB Levy 2010 A mathematical theory of energy efficient neural computation and communication IEEE Trans Inf Theory 56 852 874 10.1109/TIT.2009.2037089
5. A Borst FE Theunissen 1999 Information theory and neural coding Nat Neurosci 2 947 957 10526332 10.1038/14731 1:CAS:528:DyaK1MXmvFGlsr4%3D (Pubitemid 30599556)
6. E Burdet R Osu DW Franklin TE Milner M Kawato 2001 The central nervous system stabilizes unstable dynamics by learning optimal impedance Nature 414 446 449 11719805 10.1038/35106566 1:CAS:528:DC%2BD3MXovFamsLY%3D (Pubitemid 34173725)
7. E Burdet K Tee I Mareels T Milner C Chew D Franklin R Osu M Kawato 2006 Stability and motor adaptation in human arm movements Biol Cybern 94 20 32 16283374 10.1007/s00422-005-0025-9 1:STN:280:DC%2BD2MnptFKrsg%3D%3D (Pubitemid 43007064)
8. Buzsáki G (2006) Rhythms of the brain. Oxford University Press, Oxford
9. JE Cohen 2004 Mathematics is biology's next microscope, only better; biology is mathematics' next physics, only better PLoS Biol 2 e439 15597117 10.1371/journal.pbio.0020439 (Pubitemid 40280507)
10. Cover TM, Thomas JA (2006) Elements of information theory, 2nd edn. Wiley, New York
11. Dayan P, Abbott LF (2001) Theoretical neuroscience: computational and mathematical modeling of neural systems. The MIT Press, Cambridge
12. M Eichler 2006 On the evaluation of information flow in multivariate systems by the directed transfer function Biol Cybern 94 469 482 16544165 10.1007/s00422-006-0062-z
13. IR Fiete Y Burak T Brookings 2008 What grid cells convey about rat location J Neurosci 28 6858 18596161 10.1523/JNEUROSCI.5684-07.2008 1:CAS:528:DC%2BD1cXot1Oqt7o%3D
14. GD Fischbach 1992 Mind and brain Sci Am 267 3 24 33 10.1038/ scientificamerican0992-48
15. K Peng Tee R Osu CM Chew TE Milner M Kawato 2008 CNS learns stable, accurate, and efficient movements using a simple algorithm J Neurosci 28 44 11165 11173 10.1523/JNEUROSCI.3099-08.2008
16. K Friston 2010 The free-energy principle: a unified brain theory? Nat Rev Neurosci 11 127 138 20068583 10.1038/nrn2787 1:CAS:528:DC%2BC3cXksFGktw%3D%3D
17. M Gastpar B Rimoldi M Vetterli 2003 To code, or not to code: Lossy source-channel communication revisited IEEE Trans Inf Theory 49 1147 1158 10.1109/TIT.2003.810631
18. D George J Hawkins 2009 Towards a mathematical theory of cortical micro-circuits PLoS Comput Biol 5 e1000532 19816557 10.1371/journal.pcbi.1000532
19. CWJ Granger 1969 Investigating causal relations by econometric models and cross-spectral methods Econometrica 37 424 438 10.2307/1912791
20. W Hesse E Mõller M Arnold B Schack 2003 The use of time-variant EEG Granger causality for inspecting directed interdependencies of neural assemblies J Neurosci Methods 124 27 44 12648763 10.1016/S0165-0270(02)00366-7 (Pubitemid 36315806)
21. J Huang SP Meyn 2005 Characterization and computation of optimal distributions for channel coding IEEE Trans Inf Theory 51 2336 2351 10.1109/TIT.2005.850108 (Pubitemid 41136396)
22. S Ikeda JH Manton 2009 Capacity of a single spiking neuron channel Neural Comput 21 1714 1748 19191592 10.1162/neco.2009.05-08-792
23. Jacobs AL, Fridman G, Douglas RM, Alam NM, Latham PE, Prusky GT, Nirenberg S (2009) Ruling out and ruling in neural codes. Proceedings of the National Academy of Sciences of the USA Early Edition, pp 1-6
24. Johnson DH (2008) Information theory and neuroscience: why is the intersection so small? In: Proc. Information Theory Workshop, 5-9 May 2008, pp 104-108
25. DH Johnson 2010 Information theory and neural information processing IEEE Trans Inf Theory 56 653 666 10.1109/TIT.2009.2037047
26. DH Johnson IN Goodman 2008 Inferring the capacity of the vector Poisson channel with a Bernoulli model Netw Comput Neural Syst 19 13 33 10.1080/09548980701656798
27. DC Knill A Pouget 2004 The Bayesian brain: the role of uncertainty in neural coding and computation Trends Neurosci 27 712 719 15541511 10.1016/j.tins.2004.10.007 1:CAS:528:DC%2BD2cXpslGit7Y%3D (Pubitemid 39488074)
28. L Kostal 2010 Information capacity in the weak-signal approximation Phys Rev E 82 026115 10.1103/PhysRevE.82.026115
29. L Kostal P Lansky 2010 Information transfer for small-amplitude signals Phys Rev E 81 050901 10.1103/PhysRevE.81.050901
30. TK Landauer 1986 How much do people remember? Some estimates of the quantity of learned information in long-term memory Cogn Sci 10 477 493 10.1207/s15516709cog1004
31. P Lansky R Rodriguez L Sacerdote 2004 Mean instantaneous firing frequency is always higher than the firing rate Neural Comput 16 477 489 15022676 10.1162/089976604772744875 (Pubitemid 38157012)
32. P Lansky O Pokora JP Rospars 2008 Classification of stimuli based on stimulus-response curves and their variability Brain Res 1225 57 66 18538308 10.1016/j.brainres.2008.04.058 1:CAS:528:DC%2BD1cXps1GgtLs%3D
33. SB Laughlin TJ Sejnowski 2003 Communication in neuronal networks Science 301 1870 1874 14512617 10.1126/science.1089662 1:CAS:528:DC%2BD3sXnsFSgt7k%3D (Pubitemid 37221384)
34. SB Laughlin RR de Ruytervan Steveninck JC Anderson 1998 The metabolic cost of neural information Nat Neurosci 1 36 41 10195106 10.1038/236 1:CAS:528:DyaK1MXitVCnt70%3D (Pubitemid 128652639)
35. TS Lee D Mumford 2003 Hierarchical Bayesian inference in the visual cortex JOSA A 20 1434 1448 12868647 10.1364/JOSAA.20.001434
36. WB Levy RA Baxter 2002 Energy-efficient neuronal computation via quantal synaptic failures J Neurosci 22 4746 4755 12040082 1:CAS:528: DC%2BD38XksFWns70%3D (Pubitemid 37465690)
37. DM MacKay WS McCulloch 1952 The limiting information capacity of a neuronal link Bull Math Biophys 14 127 135 10.1007/BF02477711
38. ZF Mainen TJ Sejnowski 1995 Reliability of spike timing in neocortical neurons Science 268 1503 1506 7770778 10.1126/science.7770778 1:CAS:528:DyaK2MXmtFKku74%3D
39. H Marko 1967 Information theory and cybernetics Spectr IEEE 4 11 75 83 10.1109/MSPEC.1967.5217173
40. H Marko 1973 The bidirectional communication theory-a generalization of information theory IEEE Trans Commun 21 1345 1351 10.1109/TCOM.1973.1091610
41. Massey J (1990) Causality, feedback and directed information. In: Proceedings 1990 Inernational Symposium on Information Theory and its Applications, pp 27-30
42. MD McDonnell NG Stocks 2008 Maximally informative stimuli and tuning curves for sigmoidal rate-coding neurons and populations Phys Rev Lett 101 058103 18764432 10.1103/PhysRevLett.101.058103
43. Mitter SK (2010) Towards a unified view of inference, communication and control. In: 2010 international conference on signal processing and communications (SPCOM), pp 1-2
44. SK Mitter NJ Newton 2005 Information and entropy flow in the Kalman-Bucy filter J Stat Phys 118 145 176 10.1007/s10955-004-8781-9
45. AP Nikitin NG Stocks RP Morse MD McDonnell 2009 Neural population coding is optimized by discrete tuning curves Phys Rev Lett 103 138101 19905542 10.1103/PhysRevLett.103.138101
46. S Panzeri R Senatore MA Montemurro RS Petersen 2007 Correcting for the sampling bias problem in spike train information measures J Neurophysiol 98 1064 1072 17615128 10.1152/jn.00559.2007 (Pubitemid 47435863)
47. G Poggio 1964 Time series analysis of impulse sequences of thalamic somatic sensory neurons J Neurophysiol 27 517 545 14194957 1:STN:280: DyaF2c7lt12ruw%3D%3D
48. CJ Quinn TP Coleman N Kiyavash NG Hatsopoulos 2011 Estimating the directed information to infer causal relationships in ensemble neural spike train recordings J Comput Neurosci 30 17 44 20582566 10.1007/s10827-010-0247-2
49. RPN Rao DH Ballard 1999 Predictive coding in the visual cortex: a functional interpretation of some extra-classical receptive-field effects Nat Neurosci 2 79 87 10195184 10.1038/4580 1:CAS:528:DyaK1MXhsl2ns7k%3D (Pubitemid 30488944)
50. Rieke F, Warland D, de Ruytervan Steveninck R, Bialek W (1997) Spikes: exploring the neural code. MIT Press, Cambridge
51. J Rissanen M Wax 1987 Measures of mutual and causal dependence between two time series IEEE Trans Inf Theory 33 598 601 10.1109/TIT.1987.1057325 (Pubitemid 17651020)
52. R Sarpeshkar 1998 Analog versus digital: extrapolating from electronics to neurobiology Neural Comput 10 7 1601 1638 9744889 10.1162/089976698300017052 1:STN:280:DyaK1cvis1eluw%3D%3D (Pubitemid 128463675)
53. Shannon CE (1948) A mathematical theory of communication. Bell Syst Tech J 27:379-423 and 623-656
54. Shannon CE, Weaver W (1949) The mathematical theory of communication. University of Illinois Press, Urbana
55. J Smith 1971 The information capacity of amplitude- and variance-constrained scalar Gaussian channels Inf Control 18 203 219 10.1016/S0019-9958(71)90346-9
56. RB Stein 1967 The information capacity of nerve cells using a frequency code Biophys J 7 797 826 19210999 10.1016/S0006-3495(67)86623-2 1:STN:280:DC%2BD1M7jsFGrtg%3D%3D
57. P Suksompong T Berger 2010 Capacity analysis for integrate-and-fire neurons with descending action potential thresholds IEEE Trans Inf Theory 56 838 851 10.1109/TIT.2009.2037042
58. S Tatikonda S Mitter 2009 The capacity of channels with feedback IEEE Trans Inf Theory 55 323 349 10.1109/TIT.2008.2008147
59. E Todorov MI Jordan 2002 Optimal feedback control as a theory of motor coordination Nat Neurosci 5 11 1226 1235 12404008 10.1038/nn963 1:CAS:528:DC%2BD38Xot1ansb4%3D (Pubitemid 35247183)
60. LR Varshney PJ Sjöström DB Chklovskii 2006 Optimal information storage in noisy synapses under resource constraints Neuron 52 409 423 17088208 10.1016/j.neuron.2006.10.017 1:CAS:528:DC%2BD28Xht1KhtbjK (Pubitemid 44667440)
61. von Neumann J (2000) The computer and the brain, 2nd edn. Yale University Press, New Haven
62. VQ Vu B Yu RE Kass 2009 Information in the nonstationary case Neural Comput 21 688 703 18928371 10.1162/neco.2008.01-08-700
63. J Waddell R Dzakpasu V Booth B Riley J Reasor G Poe M Zochowski 2007 Causal entropies-a measure for determining changes in the temporal organization of neural systems J Neurosci Methods 162 320 332 17275095 10.1016/j.jneumeth. 2006.12.008 (Pubitemid 46527912)
64. Wiener N (1948) Cybernetics: or control and communication in the animal and the machine, 2nd edn. The Massachusetts Institute of Technology, New York