Pavlovian-instrumental interaction in 'observing behavior'

PLoS Computational Biology

Volumn 6, Issue 9, 2010, Pages

  Beierholm, Ulrik R ^a   Dayan, Peter ^a  

^a UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

DOPAMINE NEURONS; EXPERIMENTAL CONDITIONS; MIDBRAIN; PREDICTION ERRORS; WORKING MEMORY;

ACCURACY; ANIMAL BEHAVIOR; ANIMAL EXPERIMENT; ARTICLE; BRAIN FUNCTION; CALCULATION; CONTROLLED STUDY; DISCRIMINATION LEARNING; DOPAMINERGIC NERVE CELL; MESENCEPHALON; NONHUMAN; PREDICTION; STATE DEPENDENT LEARNING; STIMULUS RESPONSE; TASK PERFORMANCE; WORKING MEMORY;

EID: 78049420686     PISSN: 1553734X     EISSN: 15537358     Source Type: Journal    
DOI: 10.1371/journal.pcbi.1000903     Document Type: Article

References (59)

1. Breland K, Breland M (1961) The misbehavior of organisms. Am Psychol 16: 681-84.
2. Williams DR, Williams H (1969) Auto-maintenance in the pigeon: sustained pecking despite contingent non-reinforcement. J Exp Anal Behav 12: 511-520.
3. Sheffield F (1965) Relation between classical conditioning and instrumental learning. In: Prokasy W, ed. Classical conditioning. New York, NY: Appelton- Century-Crofts. pp 302-322.
4. Dayan P, Niv Y, Seymour B, D Daw N (2006) The misbehavior of value and the discipline of the will. Neural Netw 19: 1153-1160.
5. Balleine B (2005) Neural bases of food-seeking: Affect arousal and reward in corticostriatolimbic circuits. Physiol Behav 86: 717-730.
6. Daw N, Niv Y, Dayan P (2005) Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nat Neurosci 8: 1704-1711.
7. Dayan P (2008) The role of value systems in decision-making. In: Engel C, Singer W, eds. Better than Conscious. Cambridge, MA: Ernst Strüngmann Forum, MIT Press. pp 51-70.
8. Wyckoff LB (1952) The role of observing responses in discrimination learning. Part I. Psychol Rev 59: 431-442.
9. Dinsmoor J (1983) Observing and conditioned reinforcement. Behav Brain Sci 6: 693-728.
10. Bromberg-Martin ES, Hikosaka O (2009) Midbrain dopamine neurons signal preference for advance information about upcoming rewards. Neuron 63: 119-126.
11. Prokasy W (1956) The acquisition of observing responses in the absence of differential external reinforcement. J Comp Physiol Psychol 49: 131-134.
12. Roper KL, Zentall TR (1999) Observing Behavior in Pigeons: The Effect of Reinforcement Probability and Response Cost Using a Symmetrical Choice Procedure. Learn Motiv 220: 201-220.
13. Daly H (1992) Preference for unpredictability is reversed when unpredictable nonreward is aversive. In: Gormezano I, Wasserman E, eds. Learning and memory: The behavioral and biological substrates. Hillsdale, NJ: Erlbaum. pp 81-104.
14. Kreps D, Porteus E (1978) Temporal resolution of uncertainty and dynamic choice theory. Econometrica 46: 185-200.
15. Caplin A, Leahy J (2001) Psychological Expected Utility Theory and Anticipatory Feelings? Q J Econ 116: 55-79.
16. Loewenstein G (1987) Anticipation and the valuation of delayed consumption. Econ J (London) 97: 666-684.
17. Lovallo D, Kahneman D (2000) Living with uncertainty: attractiveness and resolution timing. J Behav Decis Mak 13: 179-190.
18. Shannon C, Weaver W (1949) The mathematical theory of information, volume 97. Urbana, IL: University of Illinois Press.
19. Montague PR, Dayan P, Sejnowski TJ (1996) A framework for mesencephalic dopamine systems based on predictive hebbian learning. J Neurosci 16: 1936-47.
20. Schultz W, Dayan P, Montague P (1997) A neural substrate of prediction and reward. Science 275: 1593.
21. Sutton RS, Barto AG (1998) Reinforcement Learning: An Introduction. Cambridge, MA: MIT Press.
22. Sutton RS (1988) Learning to predict by the methods of temporal differences. Mach Learn 3: 9-44.
23. Sutton R, Barto A (1990) Time-derivative models of Pavlovian reinforcement. Cambridge, MA: MIT press. pp 497-538.
24. Kehoe E, Schreurs B, Amodei N (1981) Blocking acquisition of the rabbit's nictitating membrane response to serial conditioned stimuli. Learn Motiv 12: 92-108.
25. Suri RE, Schultz W (1999) A neural network model with dopamine-like reinforcement signal that learns a spatial delayed response task. Neuroscience 91: 871-890.
26. Grossberg S, Schmajuk N (1988) Neural dynamics of adaptive timing and temporal discrimination during associative learning. Neural Netw 1: 98.
27. Ludvig EA, Sutton RS, Kehoe EJ (2008) Stimulus representation and the timing of reward-prediction errors in models of the dopamine system. Neural Comput 20: 3034-3054.
28. Mauk MD, Buonomano DV (2004) The neural basis of temporal processing. Annu Rev Neurosci 27: 307-340.
29. O'Reilly R, Frank M (2006) Making working memory work: A computational model of learning in the prefrontal cortex and basal ganglia. Neural Comput 18: 283-328.
30. Frank M, Loughry B, O'Reilly R (2001) Interactions between frontal cortex and basal ganglia in working memory: a computational model. Cogn Affect Behav Neurosci 1: 137-160.
31. Williams GV, Goldman-Rakic PS (1995) Modulation of memory fields by dopamine d1 receptors in prefrontal cortex. Nature 376: 572-575.
32. Tobler PN, Dickinson A, Schultz W (2003) Coding of predicted reward omission by dopamine neurons in a conditioned inhibition paradigm. The Journal of neuroscience: the official journal of the Society for Neuroscience 23: 10402-10.
33. Fiorillo CD, Tobler PN, Schultz W (2003) Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299: 1898-1902.
34. Bayer HM, Glimcher PW (2005) Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47: 129-141.
35. Badia P, Harsh J, Abbott B (1979) Choosing between predictable and unpredictable shock conditions: Data and theory. Psychol Bull 86: 1107-1131.
36. Mitchell KM, Perkins NP, Perkins CC (1965) Conditions affecting acquisition of observing responses in the absence of differential reward. J Comp Physiol Psychol 60: 435-7.
37. Levis DJ, Perkins CC (1965) Acquisition of observing responses (RO) with water reward. Psychol Rep 16: 114.
38. Daly HB (1989) Preference for unpredictable food rewards occurs with high proportion of reinforced trials or alcohol if rewards are not delayed. J Exp Psychol Anim Behav Process 15: 3-13.
39. Tobler PN, Fiorillo CD, Schultz W (2005) Adaptive coding of reward value by dopamine neurons. Science 307: 1642-1645.
40. Shidara M, Aigner TG, Richmond BJ (1998) Neuronal signals in the monkey ventral striatum related to progress through a predictable series of trials. J Neurosci 18: 2613-2625.
41. Mackintosh NJA (1975) theory of attention: Variations in the associability of stimuli with reinforcement. Psychol Rev 2: 276-298.
42. Daly HB, Daly JT (1982) A Mathematical Model of Reward and Aversive Nonreward: Its Application in Over 30 Appetitive Learning Situations. New York 11: 441-480.
43. Rescorla R, Wagner A (1972) Variations in the Effectiveness of Reinforcement and Nonreinforcement. New York: Classical Conditioning II: Current Research and Theory, Appleton-Century-Crofts.
44. Widrow B, Hoff MJ (1960) Adaptive switching circuits. IRE WESCON Convention Record. pp 96-104.
45. Sutton R, Barto A (1987) A temporal-difference model of classical conditioning. Proc Annu Conf Cogn Sci Soc. pp 355-378.
46. Berlyne D (1957) Uncertainty and conflict - a point of contact between information-theory and behavior-theory concepts. Psychol Rev 64: 329-339.
47. Dayan P (2009) Prospective and retrospective temporal difference learning. Network 20: 32-46.
48. Niv Y, Duff MO, Dayan P (2005) Dopamine, uncertainty and TD learning. Behavioral Brain Function 1: 6.
49. Niv Y, Joel D, Dayan P (2006) A normative perspective on motivation. Trends Cogn Sci 10: 375-381.
50. Salamone JD, Correa M (2002) Motivational views of reinforcement: implications for understanding the behavioral functions of nucleus accumbens dopamine. Behav Brain Res 137: 3-25.
51. Bunzeck N, Dayan P, Dolan R, Düzel E (2010) A common mechanism for adaptive scaling of reward and novelty. Human Brain Mapping, In press.
52. Spetch ML, Belke TW, Barnet RC, Dunn R, Pierce WD (1990) Suboptimal choice in a percentage-reinforcement procedure: effects of signal condition and terminal-link length. J Exp Anal Behav 53: 219-234.
53. Gipson C, Alessandri J, Miller H, Zentall TR (2009) Preference for 50% reinforcement over 75% reinforcement by pigeons. Learn Behav 37: 289-298.
54. Aston-Jones G, Cohen JD (2005) Adaptive gain and the role of the locus coeruleus-norepinephrine system in optimal performance. J Comp Neurol 493: 99-110.
55. Lieberman DA, Cathro JS, Nichol K, Watson E (1997) The role of S- in human observing behavior: bad news is sometimes better than no news. Learn Motiv 28: 20-42.
56. Fantino E, Case DA (1983) Human observing:maintaned by stimuli correlated with reinforcement but not extinction. Journal of the experimental analysis of behavior 40: 193-210.
57. Wyckoff L (1959) Toward a quantitative theory of secondary reinforcement. Psychol Rev 66: 68-78.
58. Perone M, Baron A (1980) Reinforcement of human observing behavior by a stimulue correlated with extinction or increased effort. J Exp Anal Behav 34: 239-261.
59. Luce RD (1959) On the possible psychophysical laws. Psychol Rev 66: 81-95.