Solving the distal reward problem with rare correlations

Neural Computation

Volumn 25, Issue 4, 2013, Pages 940-978

  Soltoggio, Andrea ^a   Steil, Jochen J ^a  

^a BIELEFELD UNIVERSITY   (Germany)

Author keywords

[No Author keywords available]

Indexed keywords

ACTION POTENTIAL; BIOLOGICAL MODEL; COMPUTER SIMULATION; LEARNING; LETTER; MEMORY; NERVE CELL; NERVE CELL PLASTICITY; PHYSIOLOGY; REINFORCEMENT; REWARD; SYNAPSE;

ACTION POTENTIALS; COMPUTER SIMULATION; LEARNING; MEMORY; MODELS, NEUROLOGICAL; NEURONAL PLASTICITY; NEURONS; REINFORCEMENT (PSYCHOLOGY); REWARD; SYNAPSES;

EID: 84877812921     PISSN: 08997667     EISSN: 1530888X     Source Type: Journal    
DOI: 10.1162/NECO_a_00419     Document Type: Letter

References (66)

1. Abbott, L. F. (1990).Modulation of function and gated learning in a networkmemory. Proceedings of the National Academy of Science of the United States of America, 87, 9241-9245.
2. Alexander, W. H., & Sporns, O. (2002). An embodied model of learning, plasticity, and reward. Adaptive Behavior, 10, 143-159.
3. Arbuthnott, G.W., &Wickens, J. (2007). Space, time and dopamine. Trends in Neurosciences, 30(2), 62-69.
4. Bailey, C. H., Giustetto, M., Huang, Y.-Y., Hawkins, R. D., & Kandel, E. R. (2000). Is heterosynaptic modulation essential for stabilizing Hebbian plasticity and memory? Nature Reviews Neuroscience, 1(1), 11-20.
5. Barco, A., Lopez de Armentia, M., & Alarcon, J. M. (2008). Synapse-specific stabilization of plasticity processes: The synaptic tagging and capture hypothesis revisited 10 years later. Neuroscience and Biobehavioral Reviews, 32, 831-851.
6. Bear,M. F., Connors, B.W.,&Paradiso, M. A. (2005). Neuroscience: Exploring the brain. Baltimore, MD:Williams & Wilkins.
7. Bi, G.-q., & Poo, M.-m. (1998). Synaptic modifications in cultured hippocampal neurons: Dependence on spike timing, synaptic strength, and postsynaptic cell type. Journal of Neuroscience, 18(24), 10464-10472.
8. Bi, G.-q., & Poo, M.-m. (2001). Synaptic modification by correlated activity: Hebb's postulate revisited. Annual Review of Neuroscience, 24, 139-166.
9. Cass, W. A., & Gerhardt, G. A. (1995). In vivo assessment of dopamine uptake in rat medial prefrontal cortex: Comparison with dorsal striatum and nucleus accumbens. Journal of Neurochemistry, 65, 201-207.
10. Cooper, S. J. (2005). Donald O. Hebb's synapse and learning rule: A history and commentary. Neuroscience and Biobehavioral Reviews, 28(8), 851-874.
11. Dale, H. H. (1935). Pharmacology and nerve-endings. Proc. R. Soc. Med., 28, 319-332.
12. Deco, G., & Rolls, E. T. (2005). Synaptic and spiking dynamics underlying reward reversal in the orbitofrontal cortex. Cerebral Cortex, 15, 15-30.
13. Farries,M. A., & Fairhall, A. L. (2007). Reinforcement learning with modulated spike timing-dependent synaptic plasticity. Journal of Neurophysiology, 98, 3648-3665.
14. Fellous, J.-M., & Linster, C. (1998). Computational models of neuromodulation. Neural Computation, 10, 771-805.
15. Florian, R. V. (2007). Reinforcement learning through modulation of spike-timingdependent synaptic plasticity. Neural Computation, 19, 1468-1502.
16. Foster, D. J., & Wilson, M. A. (2006). Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature, 440, 683-680.
17. Frey, U., & Morris, R. G. M. (1997). Synaptic tagging and long-term potentiation. Nature, 385, 533-536.
18. Garris, P., Ciolkowski, E., Pastore, P., & Wighmann, R. (1994). Efflux of dopamine from the synaptic cleft in the nucleus accumbens of the rat brain. Journal of Neuroscience, 14, 6084-6093.
19. Gerstner, W., & Kistler, M. W. (2002). Mathematical formulations of Hebbian learning. Biological Cybernetics, 87, 404-415.
20. Harris-Warrick, R. M., & Marder, E. (1991). Modulation of neural networks for behavior. Annual Review of Neuroscience, 14, 39-57.
21. Hasselmo,M. E. (1995). Neuromodulation and cortical function:Modeling the physiological basis of behavior. Behavioural Brain Research, 67, 1-27.
22. Hasselmo, M. E. (2005). Expecting the unexpected: Modeling of neuromodulation. Neuron, 46, 426-528.
23. Hebb, D. O. (1949). The organization of behavior: A neuropsychological theory. NewYork: Wiley.
24. Histed, M. H., Pasupathy, A., & Miller, E. K. (2009). Learning substrates in the primate prefrontal cortex and striatum: Sustained activity related to successful actions. Neuron, 63, 146-148.
25. Hull, C. L. (1943). Principles of behavior. New York: Appleton Century.
26. Izhikevich, E. M. (2006). Polychonization: Computation with spikes. Neural Computation, 18, 245-282.
27. Izhikevich, E.M. (2007). Solving the distal reward problem through linkage of STDP and dopamine signaling. Cerebral Cortex, 17, 2443-2452.
28. Kandel, E. R., & Tauc, L. (1965). Heterosynaptic facilitation in neurones of the abdominal ganglion of Aplysia depilans. Journal of Physiology, 181, 1-27.
29. Legenstein, R., Pecevski, D., & Maass, W. (2008). A learning theory for rewardmodulated spike-timing-dependent plasticity with application to biofeedback. PLoS Comput. Biol., 4(10), e1000180.
30. Magee, J. C., & Johnston, D. (1997). A synaptically controlled, associative signal for Hebbian plasticity in hippocampal neurons. Science, 275, 209-213.
31. Marder, E. (1996). Neuralmodulation: Following your own rhythm. Current Biology, 6(2), 119-121.
32. Marder, E., & Thirumalai, V. (2002). Cellular, synaptic and network effects of neuromodulation. Neural Networks, 15, 479-493.
33. Markram, H., Lübke, J., Frotscher, M., & Sakmann, B. (1997). Regulation of synaptic efficacy by coincidence of postsynaptic APs and EPSPs. Science, 275, 213-215.
34. Montague, P. R., Dayan, P., Person, C., & Sejnowski, T. J. (1995). Bee foraging in uncertain environments using predictive Hebbian learning. Nature, 377, 725-728.
35. O'Doherty, J. P., Kringelbach, M. L., Rolls, E. T., & Andrews, C. (2001). Abstract reward and punishment representations in the humanorbitofrontal cortex. Nature Neuroscience, 4(1), 95-102.
36. Pan, W.-X., Schmidt, R., Wickens, J. R., & Hyland, B. I. (2005). Dopamine cells respond to predicted events during classical conditioning: Evidence for eligibility traces in the reward-learning Network. Journal of Neuroscience, 25(26), 6235- 6242.
37. Päpper, M., Kempter, R., & Leibold, C. (2011). Synaptic tagging, evaluation of memories, and the distal reward problem. Learning and Memory, 18, 58-70.
38. Pavlov, I. P. (1927). Conditioned reflexes. New York: Oxford University Press.
39. Pawlak,V.,Wickens, J. R.,Kirkwood,A.,&Kerr, J. N. (2010). Timing is not everything: Neuromodulation opens the STDP gate. Frontiers in Synaptic Neuroscience, 2.
40. Pfeiffer, M., Nessler, B., Douglas, R. J., & Maass, W. (2010). Reward-modulated Hebbian learning of decision making. Neural Computation, 22, 1-46.
41. Porr, B., & Wörgötter, F. (2007). Learning with relevance: Using a third factor to stabilize Hebbian learning. Neural Computation, 19(10), 2694-2719.
42. Potjans,W., Diesmann, M., & Morrison, A. (2011). An imperfect dopaminergic error signal can drive temporal-difference learning. PLoS Computational Biology, 7(5), 1-20.
43. Potjans, W., Morrison, A., & Diesmann, M. (2009). A spiking neural network model of an actor-critic learning agent. Neural Computation, 21(2), 301-339.
44. Redgrave, P., Gurney, K., & Reynolds, J. (2008). What is reinforced by phasic dopamine signals? Brain Research Reviews, 58, 322-339.
45. Redondo, R. L., & Morris, R. G. M. (2011). Making memories last: The synaptic tagging and capture hypothesis. Nature Reviews Neuroscience, 12, 17-30.
46. Rolls, E. T. (2009). Handbook of reward and decision Making. Orlando, FL: Academic Press.
47. Rolls, E T., McCabe, C., & Redoute, J. (2008). Expected value, reward outcome, and temporal difference error representations in a probabilistic decision task. Cerebral Cortex, 18, 652-663.
48. Sarkisov, D. V. P., & Wang, S. S. H. (2008). Order-dependent coincidence detection in cerebellar purkinje neurons at the inositol ttrisphosphate receptor. Journal of Neuroscience, 28(1), 133-142.
49. Schultz, W. (1998). Predictive reward signal of dopamine neurons. Journal of Neurophysiology, 80, 1-27.
50. Schultz,W. (2002). Getting formal with dopamine and reward. Neuron, 36, 241-263.
51. Schultz,W. (2006). Behavioural theories and the neurophysiology of reward. Annual Review of Psychology, 57, 87-115.
52. Schultz, W., Apicella, P., & Ljungberg, T. (1993). Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. Journal of Neuroscience, 13, 900-913.
53. Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate for prediction and reward. Science, 275, 1593-1598.
54. Skinner, B. F. (1953). Science and human behavior. New York: Macmillan.
55. Soltoggio, A., Bullinaria, J. A., Mattiussi, C., Dürr, P., & Floreano, D. (2008). Evolutionary advantages of neuromodulated plasticity in dynamic, reward-based scenarios. In Artificial Life XI: Proceedings of the Eleventh International Conference on the Simulation and Synthesis of Living Systems. Cambridge, MA: MIT Press.
56. Soltoggio, A, & Stanley, K. O. (2012). From modulated Hebbian plasticity to simple behavior learning through noise and weight saturation. Neural Networks, 34, 28-41.
57. Soula, H., Alwan, A., & Beslon, G. (2005). Learning at the edge of chaos: Temporal coupling of spiking neurons controller for autonomous robotic. In Proceedings of the AAAI Spring Symposia on Developmental Robotics. Palo Alto, CA: AAAI.
58. Staddon, J. E. R. (1983). Adaptive behaviour and learning. Cambridge: Cambridge University Press.
59. Strata, P., & Harvey, R. (1999). Dale's principle. Brain Research Bulletin, 59, 349-350.
60. Sutton,R. S.,&Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
61. Thorndike, E. L. (1911). Animal intelligence. New York: Macmillan.
62. Timberlake, W., & Lucas, G. A. (1985). The basis of superstitious behavior: Chance contingency, stimulus substitution, or appetitive behavior?Journal of Experimental Analysis of Behaviour, 44(3), 279-299.
63. Vasilaki, E., Frémaux, N., Urbanczik, R., Senn, W., & Gerstner, W. (2009). Spikebased reinforcement learning in continuous state and action space: When policy gradient methods fail. PLoS Computational Biology, 5(12).
64. Wang, S. S. H., Denk, W., & Häusser, M. (2000). Coincidence detection in single dendritic spines mediated by calcium release. Nature Neuroscience, 3(12), 1266-1273.
65. Wighmann, R., & Zimmerman, J. (1990). Control of dopamine extracellular concentration in rat striatum by impulse flow and uptake. Brain Res. Brain Res. Rev., 15(2), 135-144.
66. Wise, R. A. (2004). Dopamine, learning and motivation. Nature Reviews Neuroscience, 5, 1-12.