Rare neural correlations implement robotic conditioning with delayed rewards and disturbances

Frontiers in Neurorobotics

Volumn 7, Issue APR, 2013, Pages

  Soltoggio, Andrea ^a   Lemme, Andre ^a   Reinhart, Felix ^a   Steil, Jochen J ^a  

^a BIELEFELD UNIVERSITY   (Germany)

Author keywords

Classical conditioning; Distal reward; Instrumental conditioning; Neuromodulation; Robotics

Indexed keywords

ARTICLE; ASSOCIATION; CLASSICAL CONDITIONING; CONDITIONING; HUMAN; INSTRUMENTAL CONDITIONING; LEARNING; NERVE CELL PLASTICITY; NEUROMODULATION; NEUROTRANSMISSION; NONHUMAN; REWARD; ROBOTICS; STIMULUS; SYNAPSE;

EID: 84904917211     PISSN: None     EISSN: 16625218     Source Type: Journal    
DOI: 10.3389/fnbot.2013.00006     Document Type: Article

References (57)

1. Alexander, W. H., and Sporns, O. (2002). An embodied model of learning, plasticity, and reward. Adapt. Behav. 10, 143-159.
2. Anlezark, G. M., Crow, T. J., and Greenway, A. P. (1973). Impaired learning and decreased cortical norepinephrine after bilateral locus coeruleus lesions. Science 181, 682-684.
3. Asada, M., MacDormanb, K. F., Ishigurob, H., and Kuniyoshic, Y. (2001). Cognitive developmental robotics as a new paradigm for the design of humanoid robots. Rob. Auton. Syst. 37, 185-193.
4. Aston-Jones, G., and Cohen, J. D. (2005). Adaptive gain and the role of the locus coeruleus-norepinephrine system in optimal performance. J. Comp. Neurol. 493, 99-110.
5. Avery, M. C., Nitz, D. A., Chiba, A. A., and Krichmar, J. L. (2012). Simulation of cholinergic and noradrenergic modulation of behavior in uncertain environments. Front. Comput. Neurosci. 6:5. doi:10.3389/fncom.2012.00005.
6. Bailey, C. H., Giustetto, M., Zhu, H., Chen, M., and Kandel, E. R. (2000). A novel function for serotonin-mediated short-term facilitation in Aplysia: conversion of a transient, cell-wide homosynaptic Hebbian plasticity into a persistent, protein synthesis-independent synapse-specific enhancement. Proc. Natl. Acad. Sci. U.S.A. 97, 11581-11586.
7. Brembs, B., Lorenzetti, F. D., Reyes, F. D., Baxter, D. A., and Byrne, J. H. (2002). Operant reward learning in aplysia: neuronal correlates and mechanisms. Science 296, 1706-1709.
8. Carew, T. J., Walters, E. T., and Kandel, E. R. (1981). Classical conditioning in a simple withdrawal reflex in Aplysia californica. J. Neurosci. 1, 1426-1437.
9. Cox, R. B., and Krichmar, J. L. (2009). Neuromodulation as a robot controller: a brain inspired strategy for controlling autonomous robots. IEEE Robot. Autom. Mag. 16, 72-80.
10. Doya, K. (1999). What are the computations of the cerebellum, the basal ganglia and the cerebral cortex? Neural. Netw. 12, 961-974.
11. Doya, K. (2002). Metalearning and neuromodulation. Neural. Netw. 15, 495-506.
12. Farries, M. A., and Fairhall, A. L. (2007). Reinforcement learning with modulated spike timing-dependent synaptic plasticity. J. Neurophysiol. 98, 3648-3665.
13. Fellous, J.-M., and Linster, C. (1998). Computational models of neuromodulation. Neural. Comput. 10, 771-805.
14. Florian, R. V. (2007). Reinforcement learning through modulation of spike-timing-dependent synaptic plasticity. Neural. Comput. 19, 1468-1502.
15. Frey, U., and Morris, R. G. M. (1997). Synaptic tagging and long-term potentiation. Nature 385, 533-536.
16. Gallistel, C. R. (1993). The Organization of Learning. Cambridge: MIT Press.
17. Hammer, M. (1993). An identified neuron mediates the unconditioned stimulus in associative olfactory learning in honeybees. Nature 366, 59-63.
18. Hasselmo, M. E. (1994). Runaway synaptic modification in models of cortex: implications for Alzheimer's disease. Neural. Netw. 7, 13-40.
19. Hasselmo, M. E. (1999). Neuromodulation: acetylcholine and memory consolidation. Trends Cogn. Sci. (Regul. Ed.) 3, 351-359.
20. Hasselmo, M. E. (1995). Neuromodulation and cortical function: modeling the physiological basis of behavior. Behav. Brain Res. 67, 1-27.
21. Hull, C. L. (1943). Principles of Behavior. New-York: Appleton Century.
22. Izhikevich, E. M. (2007). Solving the distal reward problem through linkage of STDP and dopamine signaling. Cereb. Cortex 17, 2443-2452.
23. Kamin, L. J. (1969). Punishment and Aversive Behavior, Chapter Predictability, Surprise, Attention and Conditioning. New York: Appleton-Century-Crofts, 279-296.
24. Kandel, E. R., and Tauc, L. (1965). Heterosynaptic facilitation in neurones of the abdominal ganglion of Aplysia depilans. J. Physiol. (Lond.) 181, 1-27.
25. Kaski, S., and Kohonen, T. (1994). Winner-take-all networks for physiological models of competitive learning. Neural. Netw. 7, 973-984.
26. Krichmar, J. L. (2008). The neuromodulatory system: a framework for survival and adaptive behavior in a challenging world. Adapt. Behav. 16, 385-399.
27. Legenstein, R., Pecevski, D., and Maass, W. (2008). A learning theory for reward-modulated spike-timing-dependent plasticity with application to biofeedback. PLoS Comput. Biol. 4:e1000180. doi:10.1371/journal.pcbi.1000180.
28. Marder, E., and Thirumalai, V. (2002). Cellular, synaptic and network effects of neuromodulation. Neural. Netw. 15, 479-493.
29. McGill, R., Turkey, J. W., and Larsen, W. A. (1978). Variations of box plots. Am. Stat. 32, 12-16.
30. Menzel, R., and Müller, U. (1996). Learning and memory in honeybees: from behavior to natural substrates. Annu. Rev. Neurosci. 19, 379-404.
31. Montague, P. R., Dayan, P., and Sejnowski, T. J. (1996). A framework for mesencephalic dopamine systems based on predictive Hebbian learning. J. Neurosci. 16, 1936-1947.
32. Osherson, D., Stob, M., and Weinstein, S. (1990). Systems that Learn: An Introduction to Learning Theory for Cognitive and Computer Scientists (Learning, Development, and Conceptual Change). Cambridge: MIT Press.
33. Päpper, M., Kempter, R., and Leibold, C. (2011). Synaptic tagging, evaluation of memories, and the distal reward problem. Learn. Mem. 18, 58-70.
34. Pavlov, I. P. (1927). Conditioned Reflexes. Oxford: Oxford University Press.
35. Pfeiffer, M., Nessler, B., Douglas, R. J., and Maass, W. (2010). Reward-modulated Hebbian learning of decision making. Neural. Comput. 22, 1-46.
36. Porr, B., and Wörgötter, F. (2007). Learning with relevance: using a third factor to stabilize Hebbian learning. Neural. Comput. 19, 2694-2719.
37. Potjans, W., Diesmann, M., and Morrison, A. (2011). An imperfect dopaminergic error signal can drive temporal-difference learning. PLoS Comput. Biol. 7:e1001133. doi:10.1371/journal.pcbi.1001133.
38. Potjans, W., Morrison, A., and Diesmann, M. (2009). A spiking neural network model of an actor-critic learning agent. Neural. Comput. 21, 301-339.
39. Redondo, R. L., and Morris, R. G. M. (2011). Making memories last: the synaptic tagging and capture hypothesis. Nat. Rev. Neurosci. 12, 17-30.
40. Reynolds, J. N., and Wickens, J. R. (2002). Dopamine-dependent plasticity of corticostriatal synapses. Neural. Netw. 15, 507-521.
41. Sarkisov, D. V., and Wang, S. S. H. (2008). Order-dependent coincidence detection in cerebellar Purkinje neurons at the inositol trisphosphate receptor. J. Neurosci. 28, 133-142.
42. Schultz, W., Apicella, P., and Ljungberg, T. (1993). Responses of monkey dopamine neurons to reward and conditioned stimuli during successive steps of learning a delayed response task. J. Neurosci. 13, 900-913.
43. Schultz, W., Dayan, P., and Montague, P. R. (1997). A neural substrate for prediction and reward. Science 275, 1593-1598.
44. Skinner, B. F. (1948). "Superstition" in the pigeon. J. Exp. Psychol. 38, 168-172.
45. Skinner, B. F. (1953). Science and Human Behavior. New York: MacMillan.
46. Soltoggio, A., Bullinaria, J. A., Mattiussi, C., Dürr, P., and Floreano, D. (2008). "Evolutionary advantages of neuromodulated plasticity in dynamic, reward-based scenarios," in Artificial Life XI: Proceedings of the Eleventh International Conference on the Simulation and Synthesis of Living Systems. Cambridge: MIT Press.
47. Soltoggio, A., and Stanley, K. O. (2012). From modulated Hebbian plasticity to simple behavior learning through noise and weight saturation. Neural. Netw. 34, 28-41.
48. Soltoggio, A., and Steil, J. J. (2013). Solving the distal reward problem with rare correlations. Neural. Comput. 25, 940-978.
49. Soula, H., Alwan, A., and Beslon, G. (2005). "Learning at the edge of chaos: temporal coupling of spiking neurons controller for autonomous robotic," in Proceedings of the AAAI Spring Symposia on Developmental Robotics. Stanford, CA: AAAI Spring Symposium Series.
50. Sporns, O., and Alexander, W. H. (2002). Neuromodulation and plasticity in an autonomous robot. Neural Netw. 15, 761-774.
51. Sutton, R. S., and Barto, A. G. (1998). Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
52. Thorndike, E. L. (1911). Animal Intelligence. New York: Macmillan.
53. Tsakarakis, N., Metta, G., Sandini, G., Vernon, D., Beira, R., Becchi, F., et al. (2007). iCub - the design and realization of an open humanoid platform for cognitive and neuroscience research. Adv. Robot. 21, 1151-1175.
54. Vasilaki, E., Frémaux, N., Urbanczik, R., Senn, W., and Gerstner, W. (2009). Spike-based reinforcement learning in continuous state and action space: when policy gradient methods fail. PLoS Comput. Biol. 5:e1000586. doi:10.1371/journal.pcbi.1000586.
55. Wang, S. S. H., Denk, W., and Häusser, M. (2000). Coincidence detection in single dendritic spines mediated by calcium release. Nat. Neurosci. 3, 1266-1273.
56. Wixted, J. T. (2004). The psychology and neuroscience of forgetting. Annu. Rev. Psychol. 55, 235-269.
57. Ziemke, T., and Thieme, M. (2002). Neuromodulation of reactive sensorimotor mappings as short-term memory mechanism in delayed response tasks. Adapt. Behav. 10, 185-199.