Reward-modulated Hebbian learning of decision making

Neural Computation

Volumn 22, Issue 6, 2010, Pages 1399-1444

  Pfeiffer, Michael ^a   Nessler, Bernhard ^a   Douglas, Rodney J ^b   Maass, Wolfgang ^a  

^a GRAZ UNIVERSITY OF TECHNOLOGY   (Austria)

^b UNIVERSITY OF ZURICH   (Switzerland)

Author keywords

[No Author keywords available]

Indexed keywords

ACTION POTENTIAL; ALGORITHM; ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; BAYES THEOREM; BRAIN; COMPUTER SIMULATION; DECISION MAKING; MATHEMATICAL PHENOMENA; NERVE CELL; NERVE CELL NETWORK; PHYSIOLOGY; REWARD; SYNAPTIC TRANSMISSION;

ACTION POTENTIALS; ALGORITHMS; ARTIFICIAL INTELLIGENCE; BAYES THEOREM; BRAIN; COMPUTER SIMULATION; DECISION MAKING; MATHEMATICAL CONCEPTS; NERVE NET; NEURAL NETWORKS (COMPUTER); NEURONS; REWARD; SYNAPTIC TRANSMISSION;

EID: 77953483366     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/neco.2010.03-09-980     Document Type: Article

References (66)

1. Abbott, L. F., & Nelson, S. B. (2000). Synaptic plasticity: Taming the beast. Nature Neuroscience, 3, 1178-1183.
2. Abraham,W. C. (2008). Metaplasiticity: Tuning synapses and networks for plasticity. Nature Reviews Neuroscience, 9, 387-399.
3. Attias, H. (2003). Planning by probabilistic inference. In Proc. of the 9th Int.Workshop on Artificial Intelligence and Statistics. N.p.: Society for Artificial Intelligence and Statistics.
4. Audibert, J.-Y., Munos, R., & Szepesvari, C. (2007). Tuning bandit problems in stochastic environments. In Proc. of the 18th International Conference on Algorithmic Learning Theory (pp. 150-165).
5. Auer, P., Cesa-Bianchi,N.,&Fischer, P. (2002). Finite time analysis of the multi-armed bandit problem. Machine Learning, 47(2/3), 235-256.
6. Auer, P., Jaksch, T.,&Ortner, R. (2009). Near-optimal regret bounds for reinforcement learning. In D. Koller, D. Schuurmans, Y. Bengio, & L. Bottou (Eds.), Advances in neural information processing systems, 21 (pp. 89-96). Cambridge, MA: MIT Press.
7. Bailey, C. H., Giustetto, M., Huang, Y.-Y., Hawkins, R. D., & Kandel, E. R. (2000). Is heterosynaptic modulation essential for stabilizing Hebbian plasticity and memory? Nature Reviews Neuroscience, 1, 11-20.
8. Bertsekas, D. P., & Tsitsiklis, J. (1996). Neuro-dynamic programming. Belmont, MA: Athena Scientific.
9. Bishop, C. M. (2006). Pattern recognition and machine learning. New York: Springer.
10. Caporale, N., & Dan, Y. (2008). Spike timing-dependent plasticity: A Hebbian learning rule. Annu Rev Neuroscience, 31, 25-46.
11. Dayan, P., & Abbott, L. F. (2001). Theoretical neuroscience: Computational and mathematical modeling of neural systems. Cambridge, MA: MIT Press.
12. Dayan, P., & Daw, N. (2008). Decision theory, reinforcement learning, and the brain. Cognitive, Affective, and Behavioral Neuroscience, 8, 429-453.
13. Dayan, P., & Kakade, S. (2001). Explaining away in weight space. In T. K. Leen, T. G. Dietterich, & V. Tresp (Eds.), Advances in neural information processing systems, 13 (pp. 451-457). Cambridge, MA: MIT Press.
14. Dayan, P., & Yu, A. (2003). Uncertainty and learning. IETE Journal of Research, 49, 171-182.
15. Deneve, S. (2008). Bayesian spiking neurons I: Inference. Neural Computation, 20(1), 91-117.
16. Domingos, P., & Pazzani, M. (1997). On the optimality of the simple Bayesian classifier under zero-one loss. Machine Learning, 275(29), 103-130.
17. Douglas, R. J., & Martin, K. A. (2004). Neuronal circuits of the neocortex. Annu. Rev. Neurosci., 27, 419-451.
18. Doya, K. (2002). Metalearning and neuromodulation. Neural Networks, 15, 495-506.
19. Dragalin, V., Tartakovsky, A., & Veeravalli, V. (1999). Multihypothesis sequential probability ratio tests-Part I: Asymptotic optimality. IEEE Transactions on Information Theory, 45(7), 2448-2461.
20. Farries,M.A.,&Fairhall, A. L. (2007). Reinforcement learning with modulated spike timing-dependent synaptic plasticity. Journal of Neurophysiology, 98, 3648-3665.
21. Frégnac, Y. (2003). Hebbian synaptic plasticity. In M. A. Arbib (Ed.), The handbook of brain theory and neural networks (pp. 515-522). Cambridge, MA: MIT Press.
22. Georgopoulos, A. P., Schwartz, A. P., & Kettner, R. E. (1986). Neuronal population coding of movement direction. Science, 233, 1416-1419.
23. Gittins, J. (1979). Bandit processes and dynamic allocation indices. Journal of the Royal Statistical Society, 41, 148-177.
24. Gold, J. I., & Shadlen, M. N. (2007). The neural basis of decision making. Annu. Rev. Neuroscience, 30, 535-574.
25. Griffiths, T., & Tenenbaum, J. (2005). Structure and strength in causal induction. Cognitive Psychology, 51, 334-384.
26. Gurney, K., & Bogacz, R. (2006). The basal ganglia and cortex implement optimal decision making between alternative actions. Neural Computation, 19, 442-477.
27. Hahnloser, R. H. R., Sarpeshkar, R., Mahowald, M. A., Douglas, R. J., & Seung, H. S. (2000). Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit. Nature, 405, 947-951.
28. Hebb, D. O. (1949). The organization of behavior. Hoboken, NJ: Wiley.
29. Ide, J. S., & Cozman, F. G. (2002). Random generation of Bayesian networks. In Proc. of the 16th Brazilian Symposium on Artificial Intelligence: Advances in Artificial Intelligence (pp. 366-375). Berlin: Springer-Verlag.
30. Jensen, F. V., & Nielsen, T. D. (2007). Bayesian networks and decision graphs (2nd ed.). Berlin: Springer.
31. Kearns,M.,&Singh, S. (1998). Near-optimal performance for reinforcement learning in polynomial time. In Proc. of the 15th International Conference on Machine Learning (ICML) (pp. 260-268). San Francisco: Morgan Kaufmann.
32. Kononenko, I. (1998). Bayesian neural networks. Biol. Cybernetics, 61, 361-370.
33. Kschischang, F. R., Frey, B. J., & Loeliger, H. A. (2001). Factor graphs and the sumproduct algorithm. IEEE Transactions on Information Theory, 47(2), 498-519.
34. Lai, T., & Robbins, H. (1985). Asymptotically efficient adaptive allocation rules. Advances in Applied Mathematics, 6, 4-22.
35. Lansner, A., & Ekeberg, O. (1998). A one-layer feedback artificial neural network with a Bayesian learning rule. International Journal of Neural Systems, 1, 77-87.
36. Lansner, A.,&Holst, A. (1996). A higher order Bayesian neural networkwith spiking units. International Journal of Neural Systems, 7(2), 115-128.
37. Legenstein, R., Pecevski, D., & Maass, W. (2008). A learning theory for rewardmodulated spike-timing-dependent plasticity with application to biofeedback. PLoS Computational Biology, 4(10), 1-27.
38. Lohmiller, W., & Slotine, J. J. (1998). Contraction analysis for nonlinear systems. Automatica, 34(6), 683-696.
39. Maass, W. (2000). On the computational power of winner-take-all. Neural Computation, 12(11), 2519-2536.
40. Montague, P., Dayan, P., Person, C., & Sejnowski, T. (1995). Bee foraging in uncertain environments using predictive Hebbian learning. Nature, 377, 725-728.
41. Neapolitan, R. (2004). Learning Bayesian networks. Upper Saddle River, NJ: Prentice Hall.
42. Neftci, E., Chicca, E., Indiveri, G., Slotine, J., & Douglas, R. (2008). Contraction properties of VLSI cooperative competitive neural networks of spiking neurons. In J. C. Platt, D. Koller, Y. Singer, & S. Roweis (Eds.), Advances in neural information processing systems, 20. Cambridge, MA: MIT Press.
43. Nessler, B., Pfeiffer, M., & Maass, W. (2009). Hebbian learning of Bayes optimal decisions. In D. Koller, D. Schuurmans, Y. Bengio, & L. Bottou (Eds.), Advances in neural information processing systems, 21. Cambridge, MA: MIT Press.
44. O'Keefe, J., Burgess, N., Donnett, J., Jeffery, K., & Maguire, E. (1998). Place cells, navigational accuracy, and the human hippocampus. Philosophical Transactions of the Royal Society of London, 353(1373), 1333-1340.
45. Olshausen, B. A., & Field, D. J. (1996). Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature, 381, 607-609.
46. Pouget, A., & Latham, P. (2002). Population codes. In M. A. Arbib (Ed.), The handbook of brain theory and neural networks (2nd ed., pp. 893-897). Cambridge, MA: MIT Press.
47. Rao, R. P. N. (2007). Neural models of Bayesian belief propagation. In K. Doya, S. Ishii, A. Pouget, & R. P. N. Rao (Eds.), Bayesian brain (pp. 239-267). Cambridge, MA: MIT Press.
48. Rescorla, R. A., & Wagner, A. R. (1972). Classical conditioning II. In A. H. Black & W. F., Prokasy (Eds.),Atheory of Pavlovian conditioning:Variations in the effectiveness of reinforcement and nonreinforcement (pp. 64-99). New York: Appleton-Century-Crofts.
49. Reynolds, J. N., Hyland, B. I., & Wickens, J. R. (2001). A cellular mechanism of reward-related learning. Nature, 413, 67-70.
50. Riesenhuber, M., & Poggio, T. (1999). Models of object recognition. Nature Neuroscience, 2, 1019-1025.
51. Roth, D. (1999). Learning in natural language. In T. Dean (Ed.), Proc. of the International Joint Conference on Artificial Intelligence (pp. 898-904). San Francisco: Morgan Kaufmann.
52. Sakai, Y., Okamoto, H., & Fukai, T. (2006). Computational algorithms and neuronal network models underlying decision processes. Neural Networks, 19(8), 1091-1105.
53. Sandberg, A., Lansner, A., Petersson, K. M., & Ekeberg, O. (2002). A Bayesian attractor network with incremental learning. Network: Computation in Neural Systems, 13, 179-194.
54. Schultz, W., Dayan, P., & Montague, P. (1997). A neural substrate of prediction and reward. Science, 275, 1593-1599.
55. Steimer, A., Maass, W., & Douglas, R. (2009). Belief-propagation in networks of spiking neurons. Neural Computation, 21, 2502-2523.
56. Sugrue, L. P., Corrado, G. S., & Newsome, W. T. (2004). Matching behavior and the representation of value in the parietal cortex. Science, 304, 1782-1787.
57. Sugrue, L. P., Corrado, G. S., & Newsome, W. T. (2005). Choosing the greater of two goods: Neural currencies for valuation and decision making. Nature Reviews Neuroscience, 6(5), 363-375.
58. Sutton, R. S. (1992). Gain adaptation beats least squares. In Proceedings of the 7th Yale Workshop on Adaptive and Learning Systems (pp. 161-166). New Haven, CT: Yale University.
59. Sutton,R. S.,&Barto, A. G. (1998). Reinforcement learning:Anintroduction. Cambridge, MA: MIT Press.
60. Verma, D., & Rao, R. (2006). Goal-based imitation as probabilistic inference over graphical models. In Y. Weiss, B. Schölkopf, & J. Platt (Eds.), Advances in neural information processing systems, 18 (pp. 1393-1400). Cambridge, MA: MIT Press.
61. Wald, A., & Wolfowitz, J. (1948). Optimal character of the sequential probability ratio test. Ann. Math. Statist., 19, 326-339.
62. Wang, X. J. (2002). Probabilistic decision making by slow reverberation in cortical circuits. Neuron, 36, 955-968.
63. Yang, T., & Shadlen, M. N. (2007). Probabilistic reasoning by neurons. Nature, 447, 1075-1080.
64. Yu, A., & Dayan, P. (2003). Expected and unexpected uncertainty: ACh and NE in the neocortex. In S. Becker, S. Thrün, & K. Obermayer (Eds.), Advances in neural information processing systems, 15 (pp. 157-164). Cambridge, MA: MIT Press.
65. Yuille, A. L. (2006). Augmented Rescorla-Wagner and maximum likelihood estimation. In Y. Weiss, B. Schölkopf, & J. Platt (Eds.), Advances in neural information processing systems, 18 (pp. 1561-1568). Cambridge, MA: MIT Press.
66. Yuille, A. L., & Geiger, D. (2003). Winner-take-all networks. In M. A. Arbib (Ed.), The handbook of brain theory and neural networks (pp. 1228-1231). Cambridge, MA: MIT Press.