Accounting for negative automaintenance in pigeons: A dual learning systems approach and factored representations

PLoS ONE

Volumn 9, Issue 10, 2014, Pages

  Lesaint, Florian ^a,b   Sigaud, Olivier ^a,b   Khamassi, Mehdi ^a,b  

^a SORBONNE UNIVERSITÉ   (France)

^b UNIVERSITÉ PIERRE ET MARIE CURIE   (France)

Author keywords

[No Author keywords available]

Indexed keywords

ANIMAL BEHAVIOR; ANIMAL EXPERIMENT; ARTICLE; AUTOSHAPING; CONDITIONING; CONTROLLED STUDY; INSTRUMENTAL CONDITIONING; MALADJUSTMENT; MATHEMATICAL COMPUTING; NEGATIVE FEEDBACK; NONHUMAN; PAVLOVIAN CONDITIONING; PIGEON; REINFORCEMENT; REWARD; SOCIAL INTERACTION; STIMULUS RESPONSE; TASK PERFORMANCE; ALGORITHM; ANIMAL; LEARNING; PIGEONS AND DOVES; THEORETICAL MODEL;

COLUMBA;

ALGORITHMS; ANIMALS; BEHAVIOR, ANIMAL; COLUMBIDAE; LEARNING; MODELS, THEORETICAL;

EID: 84908587620     PISSN: None     EISSN: 19326203     Source Type: Journal    
DOI: 10.1371/journal.pone.0111050     Document Type: Article

References (71)

1. Breland K, Breland M (1961) The misbehavior of organisms. Am Psychol 16: 681.
2. Hershberger WA (1986) An approach through the looking-glass. Anim Learn Behav 14: 443-451.
3. Guitart-Masip M, Huys QJM, Fuentemilla L, Dayan P, Duzel E, et al. (2012) Go and no-go learning in reward and punishment: interactions between affect and effect. Neuroimage 62: 154-166.
4. Dayan P, Niv Y, Seymour B, Daw ND (2006) The misbehavior of value and the discipline of the will. Neural Netw 19: 1153-1160.
5. Redish AD, Jensen S, Johnson A (2008) A unified framework for addiction: vulnerabilities in the decision process. Behav Brain Sci 31: 415-437.
6. Beierholm UR, Dayan P (2010) Pavlovian-instrumental interaction in observing behavior. PLoS Comput Biol 6: e1000903.
7. Clark JJ, Hollon NG, Phillips PEM (2012) Pavlovian valuation systems in learning and decision making. Curr Opin Neurobiol 22: 1054-1061.
8. Williams DR, Williams H (1969) Auto-maintenance in the pigeon: Sustained pecking despite contingent non-reinforcement. J Exp Anal Behav.
9. Skinner BF (1938) The behavior of organisms: An experimental analysis. Appleton-Century-Crofts New York, 82-82 pp.
10. Brown PL, Jenkins HM (1968) Auto-shaping of the pigeon's key peck. J Exp Anal Behav 11: 1-8.
11. Deich JD, Wasserman EA (1977) Rate and temporal pattern of key pecking under autoshaping and omission schedules of reinforcement. J Exp Anal Behav 27: 399-405.
12. Griffin RW, Rashotte ME (1973) A note on the negative automaintenance procedure. Bull Psychon Soc 2: 402-404.
13. Killeen PR (2003) Complex dynamic processes in sign tracking with an omission contingency (negative automaintenance). J Exp Psychol Anim Behav Process 29: 49.
14. Woodard WT, Ballinger JC, Bitterman M (1974) Autoshaping: further study of "negative automaintenance". J Exp Anal Behav 22: 47-51.
15. Locurto C, Terrace H, Gibbon J (1976) Autoshaping, random control, and omission training in the rat. J Exp Anal Behav 26: 451-462.
16. Locurto C, Terrace H, Gibbon J (1978) Omission training (negative automaintenance) in the rat: Effects of trial offset. Bull Psychon Soc 12: 11-14.
17. O'Connell MF (1979) Temporal distributions of responding during discrete-trial omission training in rats. J Exp Anal Behav 31: 31.
18. Gormezano I, Hiller GW (1972) Omission training of the jaw-movement response of the rabbit to a water us. Psychon Sci 29: 276-278.
19. Sanabria F, Sitomer MT, Killeen PR (2006) Negative automaintenance omission training is effective. J Exp Anal Behav 86: 1-10.
20. Lesaint F, Sigaud O, Flagel SB, Robinson TE, Khamassi M (2014) Modelling individual differences in the form of pavlovian conditioned approach responses: A dual learning systems approach with factored representations. PLoS Comput Biol 10: e1003466.
21. Flagel SB, Clark JJ, Robinson TE, Mayo L, Czuj A, et al. (2011) A selective role for dopamine in stimulus-reward learning. Nature 469: 53-57.
22. Boakes R (1977) Performance on learning to associate a stimulus with positive reinforcement. Operant-Pavlovian interactions: 67-97.
23. Dickinson A, Balleine B (1994) Motivational control of goal-directed action. Anim Learn Behav 22: 1-18.
24. Graybiel AM (2008) Habits, rituals, and the evaluative brain. Annu Rev Neurosci 31: 359-387.
25. Dolan RJ, Dayan P (2013) Goals and habits in the brain. Neuron 80: 312-325.
26. Sutton RS, Barto AG (1998) Reinforcement learning: An introduction. The MIT Press.
27. Schultz W (1998) Predictive reward signal of dopamine neurons. J Neurophysiol 80: 1-27.
28. Niv Y (2009) Reinforcement learning in the brain. J Math Psychol 53: 139-154.
29. Mahler SV, Berridge KC (2009) Which cue to "want?" Central amygdala opioid activation enhances and focuses incentive salience on a prepotent reward cue. J Neurosci 29: 6500-13.
30. DiFeliceantonio AG, Berridge KC (2012) Which cue to 'want'? Opioid stimulation of central amygdala makes goal-trackers show stronger goal-tracking, just as sign-trackers show stronger sign-tracking. Behav Brain Res 230: 399-408.
31. Berridge KC (2007) The debate over dopamines role in reward: the case for incentive salience. Psychopharmacology 191: 391-431.
32. Guitart-Masip M, Duzel E, Dolan R, Dayan P (2014) Action versus valence in decision making. Trends Cogin Sci 18: 194-202.
33. Lesaint F, Sigaud O, Clark JJ, Flagel SB, Khamassi M (2014) Experimental predictions drawn from a computational model of sign-trackers and goal-trackers. J Physiol Paris: in press.
34. Dayan P, Berridge KC (2014) Model-based and model-free pavlovian reward learning: Revaluation, revision, and revelation. Cogn Affect Behav Neurosci: 1-20.
35. Dayan P, Balleine BW (2002) Reward, motivation, and reinforcement learning. Neuron 36: 285-298.
36. Nicola SM (2010) The flexible approach hypothesis: unification of effort and cue-responding hypotheses for the role of nucleus accumbens dopamine in the activation of reward-seeking behavior. J Neurosci 30: 16585-16600.
37. Huys QJM, Cools R, Gölzer M, Friedel E, Heinz A, et al. (2011) Disentangling the roles of approach, activation and valence in instrumental and pavlovian responding. PLoS Comput Biol 7: e1002028.
38. Geurts DE, Huys QJ, den Ouden HE, Cools R (2013) Aversive pavlovian control of instrumental behavior in humans. J Cogn Neurosci 25: 1428-1441.
39. Yin HH, Ostlund SB, Balleine BW (2008) Reward-guided learning beyond dopamine in the nucleus accumbens: the integrative functions of cortico-basal ganglia networks. Eur J neurosci 28: 1437-1448.
40. Alloy LB, Ehrman RN (1981) Instrumental to pavlovian transfer: Learning about response-reinforcer contingencies affects subsequent learning about stimulus-reinforcer contingencies. Learn Motiv 12: 109-132.
41. Prévost C, McNamee D, Jessup RK, Bossaerts P, O'Doherty JP (2013) Evidence for model-based computations in the human amygdala during pavlovian conditioning. PLoS Comput Biol 9: e1002918.
42. Balleine BW, O'Doherty JP (2009) Human and rodent homologies in action control: corticostriatal determinants of goal-directed and habitual action. Neuropsychopharmacology 35: 48-69.
43. Daw ND, Niv Y, Dayan P (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8: 1704-1711.
44. Keramati M, Dezfouli A, Piray P (2011) Speed/Accuracy trade-off between the habitual and the goal-directed processes. PLoS Comput Biol 7: e1002055.
45. Pezzulo G, Rigoli F, Chersi F (2013) The mixed instrumental controller: using value of information to combine habitual choice and mental simulation. Front Psychol 4.
46. Meyer PJ, Lovic V, Saunders BT, Yager LM, Flagel SB, et al. (2012) Quantifying individual variation in the propensity to attribute incentive salience to reward cues. PLoS ONE 7: e38987.
47. Corbit LH, Balleine BW (2005) Double dissociation of basolateral and central amygdala lesions on the general and outcome-specific forms of pavlovian-instrumental transfer. J Neurosci 25: 962-970.
48. Schmajuk NA, Lam YW, Gray JA (1996) Latent inhibition: A neural network approach. J Exp Psychol Anim Behav Process 22: 321-349.
49. Balkenius C (1999) Dynamics of a classical conditioning model. Auton Robots 7: 41-56.
50. Redish AD, Jensen S, Johnson A, Kurth-Nelson Z (2007) Reconciling reinforcement learning models with behavioral extinction and renewal: Implications for addiction, relapse, and problem gambling. Psychol Rev 114: 784-805.
51. Stout SC, Miller RR (2007) Sometimes-competing retrieval (SOCR): A formalization of the comparator hypothesis. Psychol Rev 114: 759-783.
52. Courville AC, Daw ND, Touretzky DS (2006) Bayesian theories of conditioning in a changing world. Trends Cogn Sci 10: 294-300.
53. Gershman SJ, Niv Y (2012) Exploring a latent cause theory of classical conditioning. Anim Learn Behav 40: 255-268.
54. Kamin LJ (1967) Predictability, surprise, attention, and conditioning. In: Campbell BA, Church RMa, editors, Punishment and aversive behavior, New York: Appleton-Century-Crofts. pp.279-296.
55. Lattal KM, Nakajima S (1998) Overexpectation in appetitive pavlovian and instrumental conditioning. Anim Learn Behav 26: 351-360.
56. Doya K, Samejima K, Katagiri Ki, Kawato M (2002) Multiple model-based reinforcement learning. Neural Comput 14: 1347-1369.
57. Bellman R (1957) Dynamic programming. Princeton University Press.
58. Khamassi M, Martinet LE, Guillot A (2006) Combining self-organizing maps with mixtures of experts: application to an actor-critic model of reinforcement learning in the basal ganglia. In: From Animals to Animats 9, Springer. pp.394-405.
59. Elfwing S, Uchibe E, Doya K (2013) Scaled free-energy based reinforcement learning for robust and efficient learning in high-dimensional state spaces. Front Neurorobot 7.
60. Boutilier C, Dearden R, Goldszmidt M (2000) Stochastic dynamic programming with factored representations. Artif Intell 121: 49-107.
61. Degris T, Sigaud O, Wuillemin PH (2006) Learning the structure of factored markov decision processes in reinforcement learning problems. In: Proceedings of the 23rd international conference on Machine learning. ACM, pp.257-264.
62. Vigorito CM, Barto AG (2008) Autonomous hierarchical skill acquisition in factored mdps. In: Yale Workshop on Adaptive and Learning Systems, New Haven, Connecticut.
63. Fitzpatrick CJ, Gopalakrishnan S, Cogan ES, Yager LM, Meyer PJ, et al. (2013) Variation in the form of pavlovian conditioned approach behavior among outbred male sprague-dawley rats from different vendors and colonies: Signtracking vs. goal-tracking. PloS ONE 8: e75042.
64. Poling A, Poling T (1978) Automaintenance in guinea pigs: Effects of feeding regimen and omission training. J Exp Anal Behav 30: 37-46.
65. Gamzu E, Schwam E (1974) Autoshaping and automaintenance of a key-press response in squirrel monkeys. J Exp Anal Behav 21: 361-371.
66. Meyer PJ, Cogan ES, Robinson TE (2014) The form of a conditioned stimulus can influence the degree to which it acquires incentive motivational properties. PloS ONE 9: e98163.
67. Fiorillo CD, Tobler PN, Schultz W (2003) Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299: 1898-1902.
68. Gargiulo PA, Acerbo MJ, Krug I, Delius J (2005) Cognitive effects of dopaminergic and glutamatergic blockade in nucleus accumbens in pigeons. Pharmacology Biochemistry and Behavior 81: 732-739.
69. Niv Y, Duff MO, Dayan P (2005) Dopamine, uncertainty and td learning. Behavioral and Brain Functions 1: 1-9.
70. Miyata H, Fujita K (2008) Pigeons (columba livia) plan future moves on computerized maze tasks. Anim Cogn 11: 505-516.
71. Daw ND (2011) Trial-by-trial data analysis using computational models. In: Delgado MR, Phelps EA, Robbins TW, editors, Decision Making, Affect, and Learning: Attention and Performance XXIII, Oxford University Press, volume 23, chapter 1.