The dynamics of operant conditioning

Psychological Review

Volumn 106, Issue 1, 1999, Pages 20-61

  Dragoi, Valentin ^a   Staddon, J E R ^b  

^a MASSACHUSETTS INSTITUTE OF TECHNOLOGY   (United States)

^b NONE

Author keywords

[No Author keywords available]

Indexed keywords

ANIMAL; ARTICLE; HUMAN; INSTRUMENTAL CONDITIONING; MEMORY; PSYCHOLOGICAL MODEL; REINFORCEMENT;

ANIMALS; CONDITIONING, OPERANT; HUMANS; MEMORY; MODELS, PSYCHOLOGICAL; REINFORCEMENT (PSYCHOLOGY);

EID: 0032602208     PISSN: 0033295X     EISSN: None     Source Type: Journal    
DOI: 10.1037/0033-295X.106.1.20     Document Type: Article

References (133)

1. Adams, C. D. (1982). Variations in the sensitivity of instrumental responding to reinforcer devaluation. Quarterly Journal of Experimental Psychology, 34B, 77-98.
2. Ainslie, G. (1975). Specious reward: A behavioral theory of impulsiveness and impulse control. Psychological Bulletin, 82, 463-496.
3. Ainslie, G., & Herrnstein, R. J. (1981). Preference reversal and delayed reinforcement. Animal Learning and Behavior, 9, 476-482.
4. Amsel, A. (1962). Frustrative nonreward in partial reinforcement and discrimination learning. Psychological Review, 69, 306-328.
5. Amsel, A., Rashotte, M. E., & MacKinnon, J. R. (1966). Partial reinforcement effects within subjects and between subjects. Psychological Monographs, 80 (20, Whole No. 628).
6. Bacon, W. E. (1962). Partial-reinforcement extinction effect following different amounts of training. Journal of Comparative and Physiological Psychology, 55, 998-1003.
7. Baum, W. M. (1979). Matching, undermatching, and overmatching in studies of choice. Journal of the Experimental Analysis of Behavior, 32, 269-281.
8. Belke, T. W. (1992). Stimulus preference and the transitivity of preference. Animal Learning and Behavior, 20, 401-406.
9. Benefield, R., Oscos, A., & Ehrefreund, D. (1974). Role of frustration in successive positive contrast effect. Journal of Comparative and Physiological Psychology, 86, 648-651.
10. Bernheim, J. W., & Williams, D. R. (1967). Time-dependent contrast effects in a multiple schedule of food reinforcement. Journal of the Experimental Analysis of Behavior, 10, 243-249.
11. Bitterman, M. E., & McConnell, J. V. (1954). The role of set in successive discrimination. American Journal of Psychology, 67, 129-132.
12. Black, R. W. (1968). Shifts in magnitude of reward and contrast effects in instrumental conditioning. Psychological Review, 75, 114-126.
13. Bloomfield, T. M. (1967). Some temporal properties of behavioral contrast. Journal of the Experimental Analysis of Behavior, 10, 159-164.
14. Bolles, R. C. (1972). Reinforcement, expectancy, and learning. Psychological Review, 79, 394-409.
15. Bower, G. H. (1961). A contrast effect in differential conditioning. Journal of Experimental Psychology, 62, 196-199.
16. Bush, R. R., & Mosteller, F. (1955). Stochastic models for learning. New York: Wiley.
17. Capaldi, E. J. (1967). Sequential versus nonsequential variables in partial delay of reward. Journal of Experimental Psychology, 74, 161-166.
18. Capaldi, E. J. (1971). Memory and learning: A sequential viewpoint. In W. K. Honig & P. H. R. James (Eds.), Animal memory (pp. 111-154). New York: Academic Press.
19. Capaldi, E. J. (1972). Successive negative contrast effect: Intertrial interval, type of shift, and four sources of generalization decrement. Journal of Experimental Psychology, 96, 433-438.
20. Capaldi, E. J., & Lynch, D. (1967). Repeated shifts in reward magnitude: Evidence in favor of an associational and absolute (noncontextual) interpretation. Journal of Experimental Psychology, 75, 226-235.
21. Capaldi, E. J., & Stevenson, H. W. (1957). Response reversal following different amounts of training. Journal of Comparative and Physiological Psychology, 50, 195-198.
22. Chung, S. H., & Herrnstein, R. J. (1967). Choice and delay of reinforcement Journal of the Experimental Analysis of Behavior, 10, 67-74.
23. Cohen, S. L., Riley, D. S., & Weigle, P. A. (1993). Tests of behavior momentum in simple and multiple schedules with rats and pigeons. Journal of the Experimental Analysis of Behavior, 60, 255-291.
24. Colwill, R. M., & Rescorla, R. A. (1985a). Instrumental responding remains sensitive to reinforcer devaluation after extensive training. Journal of Experimental Psychology: Animal Behavior Processes, 11, 520-536.
25. Colwill, R. M., & Rescorla, R. A. (1985b). Postconditioning devaluation of a reinforcer affects instrumental responding. Journal of Experimental Psychology: Animal Behavior Processes, 11, 120-132.
26. Colwill, R. M., & Rescorla, R. A. (1986). Associative structures in instrumental learning. In G. H. Bower (Ed.), The psychology of learning and motivation (Vol. 20, pp. 55-104). New York: Academic Press.
27. Colwill, R. M., & Rescorla, R. A. (1990). Effect of reinforcer devaluation on discriminative control of instrumental behavior. Journal of Experimental Psychology: Animal Behavior Processes, 16, 40-47.
28. Cox, W. M. (1975). A review of recent incentive contrast studies involving discrete trial procedures. Psychological Record, 25, 373-393.
29. Crespi, L. P. (1952). Quantitative variation of incentive and performance in the white rat. American Journal of Psychology, 55, 467-517.
30. Crum, J., Brown, W. L., & Bitterman, M. E. (1951). The effect of partial and delayed reinforcement on resistance to extinction. American Journal of Psychology, 64, 228-237.
31. Daly, H. B., & Daly, J. T. (1982). A mathematical model of reward and aversive nonreward: Its application in over 30 appetitive learning situations. Journal of Experimental Psychology: General, 111, 441-480.
32. Davenport, J. W. (1962). The interaction of magnitude and delay of reinforcement in spatial discrimination. Journal of Comparative and Physiological Psychology, 55, 267-273.
33. Davis, D. G. S., & Staddon, J. E. R. (1990). Memory for reward in probabilistic choice: Markovian and non-Markovian properties. Behaviour, 114, 37-64.
34. Davis, D. G. S., Staddon, J. E. R., Machado, A., & Palmer, R. (1993). The process of recurrent choice. Psychological Review, 100, 320-341.
35. Davison, M., & McCarthy, D. (1988). The matching law. Hillsdale, NJ: Erlbaum.
36. Dickinson, A., & Mulatero, C. W. (1989). Reinforcer specificity of the suppression of instrumental performance on a non-contingent schedule. Behavioural Processes, 19, 167-180.
37. Di Lollo, V., & Beez, V. (1966). Negative contrast effect as a function of magnitude of reward decrement. Psychonomic Science, 5, 99-100.
38. Donahoe, J. W., & Palmer, D. C. (1994). Learning and complex behavior. Boston: Allyn & Bacon.
39. Douglas, R. J., Koch, C., Mahowald, M., Martin, K. A. C., & Suarez, H. H. (1995). Recurrent excitation in neocortical circuits. Science, 269, 981-985.
40. Dragoi, V. (1997). A dynamic theory of acquisition and extinction in operant learning. Neural Networks, 10, 201-229.
41. Dragoi, V., & Staddon, J. E. R. (1993). A competitive neural network model for the process of recurrent choice. In M. C. Mozer, P. Smolensky, D. S. Touretzky, J. L. Elman, & A. S. Weigend (Eds.), Proceedings of the 1993 Connectionist Models Summer School (pp. 65-73). Hillsdale, NJ: Erlbaum.
42. Dreyfus, L. R. (1991). Local shifts in relative reinforcement rate and time allocation on concurrent schedules. Journal of Experimental Psychology: Animal Behavior Processes, 17, 486-502.
43. Estes, W. K. (1955). Statistical theory of spontaneous recovery and regression Psychological Review, 62, 145-154.
44. Ettinger, R. H., & Staddon, J. E. R. (1982). Behavioral competition, component duration and multiple-schedule contrast. Behaviour Analysis Letter, 2, 31-38.
45. Franchina, J. J., & Brown, T. S. (1971). Reward magnitude shifts effects in rats with hippocampal lesions. Journal of Comparative and Physiological Psychology, 76, 365-370.
46. Gallistel, C. R. (1990). The organization of learning. Cambridge, MA: MIT Press.
47. Gibbon, J. (1995). Dynamics of time matching: Arousal makes better seem worse Psychonomic Bulletin and Review, 2, 208-215.
48. Gibbon, J., Church, R. M., Fairhurst, S., & Kacelnik, A. (1988). Scalar expectancy theory and choice between delayed rewards. Psychological Review, 95, 102-114.
49. Gonzales, R. C., & Bitterman, M. E. (1969). Spaced-trials partial reinforcement effect as a function of contrast. Journal of Comparative and Physiological Psychology, 67, 94-103.
50. Gonzales, R. C., & Shepp, B. E. (1961). Simultaneous and successive discrimination-reversal in the rat. American Journal of Psychology, 74, 584-589.
51. Gray, J. A. (1979). The neuropsychology of anxiety. Oxford: Oxford University Press.
52. Gray, J. A. (1982). The neuropsychology of anxiety: An enquiry into the functions of the septo-hippocampal system. Oxford: Oxford University Press.
53. Grossberg, S. (1981). Psychophysiological substrates of schedule interactions and behavioral contrast. SIAM-AMS Proceedings, 13, 157-186.
54. Grossberg, S. (1982). Processing of expected and unexpected events during conditioning and attention: A psychophysiological theory. Psychological Review, 89, 529-572.
55. Halliday, M. S., & Boakes, R. A. (1971). Behavioral contrast and response independent reinforcement. Journal of the Experimental Analysis of Behavior, 16, 429-434.
56. Hammond, L. J., & Weinberg, M. (1984). Signaling unearned reinforcers removes the suppression produced by a zero correlation in an operant paradigm. Animal Learning and Behavior, 12, 371-377.
57. Harlow, H. F. (1949). The formation of learning sets. Psychological Review, 56, 51-65.
58. Herrnstein, R. J. (1961). Relative and absolute strength of response as a function of frequency of reinforcement. Journal of the Experimental Analysis of Behavior, 4, 267-272.
59. Herrnstein, R. J. (1970). On the law of effect. Journal of the Experimental Analysis of Behavior, 13, 243-266.
60. Herrnstein, R. J., & Vaughan, W. (1980). Melioration and behavioral allocation. In J. E. R. Staddon (Ed.), Limits to action: The allocation of individual behavior (pp. 143-146). San Diego, CA: Academic Press.
61. Hinson, J. M., & Staddon, J. E. R. (1978). Behavioral competition: A mechanism for schedule interactions. Science, 202, 432-434.
62. Hinson, J. M., & Staddon, J. E. R. (1983). Matching, maximizing and hill climbing. Journal of the Experimental Analysis of Behavior, 40, 321-331.
63. Hooper, R. (1967). Variables controlling the overlearning reversal effect (ORE). Journal of Experimental Psychology, 73, 612-619.
64. Horner, J. M., & Staddon, J. E. R. (1987). Probabilistic choice: A simple invariance. Behavioural Processes, 15, 59-92.
65. Hull, C. L. (1943). Principles of behavior. New York: Appleton-Century-Crofts.
66. Jenkins, H. M. (1962). Resistance to extinction when partial reinforcement is followed by regular reinforcement. Journal of the Experimental Analysis of Behavior, 64, 441-450.
67. Kacelnik, A., Krebs, J. R., & Ens, B. (1987). Foraging in a changing environment: An experiment with starlings (Sturnus vulgaris). In M. L. Commons, A. Kacelnik, & S. J. Shettleworth (Eds.), Quantitative analyses of behavior VI: Foraging (pp. 63-87). Hillsdale, NJ: Erlbaum.
68. Killeen, P. (1968). On the measurement of reinforcement frequency in the study of preference. Journal of the Experimental Analysis of Behavior, 11, 263-269.
69. Killeen, P. (1970). Preference for fixed-interval schedules of reinforcement. Journal of the Experimental Analysis of Behavior, 14, 127-131.
70. Killeen, P. (1994). Mathematical principles of reinforcement. Behavioral and Brain Sciences, 17, 105-172.
71. Klopf, A. H. (1988). A neuronal model of classical conditioning. Psychobiology, 16, 85-125.
72. Lea, S. E. G., & Dow, S. M. (1984). The integration of reinforcements over time. Annals of the New York Academy of Sciences, 423, 269-277.
73. Lewis, D. J. (1956). Acquisition, extinction, and spontaneous recovery as a function of percentage of reinforcement and intertrial intervals. Journal of Experimental Psychology, 51, 45-53.
74. Logan, F. A. (1960). Incentive. New Haven, CT: Yale University Press.
75. Luce, R. D. (1959). Individual choice behavior: A theoretical analysis. New York: Wiley.
76. Mackintosh, N. J. (1969). Further analysis of the overtraining reversal effect. Journal of Comparative and Physiological Psychology Monographs, 67 (2, Pt. 2).
77. Mackintosh, N. J. (1970). Distribution of trials and the partial reinforcement effect in the rat. Journal of Comparative and Physiological Psychology, 73, 341-348.
78. Mackintosh, N. J. (1974). The psychology of animal learning. San Diego, CA: Academic Press.
79. Mackintosh, N. J., & Dickinson, A. (1979). Instrumental (Type II) conditioning. In A. Dickinson & R. A. Boakes (Eds.), Mechanisms of learning and motivation (pp. 143-167). Hillsdale, NJ: Erlbaum.
80. Mark, T. A., & Gallistel, C. R. (1994). Kinetics of matching. Journal of Experimental Psychology: Animal Behavior Processes, 20, 79-95.
81. Mark, T. A., & Gallistel, C. R. (1995, November). The microstructure of matching. Paper presented at the Psychonomic Society Meeting, Washington, DC.
82. Maxwell, F. R., Calef, R. S., Murray, D. W., Shepard, D. C., & Norville, R. A. (1976). Positive and negative successive contrast effects following multiple shifts in reward magnitude under high drive and immediate reinforcement. Animal Learning and Behavior, 4, 480-484.
83. Mazur, J. E. (1992). Choice behavior in transition: Development of preference with ratio and interval schedules. Journal of Experimental Psychology: Animal Behavior Processes, 18, 364-378.
84. Mazur, J. E. (1995). Development of preference and spontaneous recovery in choice behavior with concurrent variable-interval schedules. Animal Learning and Behavior, 23, 93-103.
85. Mazur, J. E., & Ratti, T. A. (1991). Choice behavior in transition: Development of preference in a free-operant procedure. Animal Learning and Behavior, 19, 241-248.
86. McClelland, J. L., & Rumelhart, D. E. (1985). Distributed memory and the representation of general and specific information. Journal of Experimental Psychology: General, 114, 159-188.
87. McEwen, D. (1972). The effects of terminal-link fixed-interval and variable-interval schedules on responding under concurrent chained schedules. Journal of the Experimental Analysis of Behavior, 18, 253-261.
88. McHose, J. H. (1970). Relative reinforcement effects: S1/S2 and S1/S1 paradigms in instrumental conditioning. Psychological Review, 77, 135-146.
89. Meyer, D. R. (1951). The effects of differential rewards on discrimination reversal learning by monkeys. Journal of Experimental Psychology, 41, 268-274.
90. Mikulka, P. J., Lehr, R., & Pawlik, W. B. (1967). Effect of reinforcement schedules on reward shifts. Journal of Experimental Psychology, 74, 57-61.
91. Myerson, J., & Miezin, F. M. (1980). The kinetics of choice: An operant systems analysis. Psychological Review, 87, 160-174.
92. Nevin, J. A. (1988). Behavioral momentum and the partial reinforcement effect. Psychological Bulletin, 103, 44-56.
93. Nevin, J. A., & Shettleworth, S. J. (1966). An analysis of contrast effects in multiple schedules. Journal of the Experimental Analysis of Behavior, 9, 305-315.
94. Pavlik, W. B., Carlton, P. L., & Hughes, R. A. (1965). Partial reinforcement effects in a runaway: Between-and within-Ss. Psychonomic Science, 3, 203-204.
95. Pavlik, W. B., & Collier, A. A. (1977). Magnitude and schedule of reinforcement in rats' resistance to extinction - Within subjects. American Journal of Psychology, 90, 195-205.
96. Pavlov, I. P. (1927). Conditioned reflexes. Oxford, England: Oxford University Press.
97. Pearce, J. M., & Hall, G. (1980). A model for Pavlovian learning: Variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychological Review, 87, 532-552.
98. Rashotte, M. E., & Amsel, A. (1968). The generalized PRE: Within-S PRF and CRF training in different runaways, at different times of day by different experimenters. Psychonomic Science, 11, 315-316.
99. Reid, L. S. (1953). The development of noncontinuity behavior through continuity learning. Journal of Experimental Psychology, 46, 107-112.
100. Renner, K. E. (1963). Influence of deprivation and availability of goal box cues on the temporal gradient of reinforcement. Journal of Comparative Psychology, 56, 101-104.
101. Rescorla, R. A., & Wagner, A. R. (1972). A theory of Pavlovian conditioning: Variations in the effectiveness of reinforcement and nonreinforcement. In A. H. Black & W. F. Prokasy (Eds.), Classical conditioning II: Current research and theory (pp. 64-99). New York: Appleton-Century-Crofts.
102. Reynolds, B. (1961). Behavioral contrast. Journal of the Experimental Analysis of Behavior, 4, 57-71.
103. Robbins, S. J. (1990). Mechanisms underlying spontaneous recovery in autoshaping. Journal of Experimental Psychology: Animal Behavior Processes, 16, 235-249.
104. Rozin, P. (1965). Temperature independence of an arbitrary temporal discrimination in the goldfish. Science, 149, 561-563.
105. Russell, W. R. (1971). The traumatic amnesia. London: Oxford University Press.
106. Schmajuk, N. A., & DiCarlo, J. J. (1992). Stimulus configuration, classical conditioning, and hippocampal function. Psychological Review, 99, 268-305.
107. Sgro, J. A., & Weinstock, S. (1963). Effects of delay on subsequent running under immediate reinforcement. Journal of Experimental Psychology, 66, 260-263.
108. Sheffield, V. F. (1949). Extinction as a function of partial reinforcement and distribution of practice. Journal of Experimental Psychology, 39, 511-526.
109. Shimp, C. P., & Wheatley, K. L. (1971). Matching to relative reinforcement frequency in multiple schedules with a short component duration. Journal of the Experimental Analysis of Behavior, 15, 205-210.
110. Spence, K. W. (1952). The nature of the response in discrimination learning. Psychological Review, 59, 89-93.
111. Spence, K. W. (1956). Behavior theory and conditioning. New Haven, CT: Yale University Press.
112. Sperling, S. E. (1970). ORE in simultaneous and differential reversal: The acquisition task, the acquisition criterion, and the reversal task. Journal of Experimental Psychology, 84, 349-360.
113. Squire, L. R., Slater, P. C., & Chace, P. M. (1975). Retrograde amnesia: Temporal gradient in very long term following electroconvulsive therapy. Science, 187, 77-79.
114. Staddon, J. E. R. (1988). Quasi-dynamic choice models: Melioration and ratio invariance. Journal of the Experimental Analysis of Behavior, 49, 303-320.
115. Staddon, J. E. R., & Simmelhag, V. G. (1971). The "superstition" experiment: A reexamination of its implications for the principles of adaptive behavior. Psychological Review, 78, 3-43.
116. Staddon, J. E. R., & Zhang, Y. (1991). On the assignment-of-credit problem in operant learning. In M. L. Commons, S. Grossberg, & J. E. R. Staddon (Eds.), Neural network models of conditioning and action (pp. 279-293). Hillsdale, NJ: Erlbaum.
117. Stemmler, M., Usher, M., & Niebur, E. (1995). Lateral interactions in primary visual cortex: A model bridging physiology and psychophysics. Science, 269, 1877-1880.
118. Sutherland, N. S., & Mackintosh, N. J. (1971). Mechanisms of animal discrimination learning. New York: Academic Press.
119. Sutton, R. S., & Barto, A. G. (1981). Toward a modern theory of adaptive networks: Expectation and prediction. Psychological Review, 88, 135-170.
120. Sutton, R. S., & Barto, A. G. (1990). Time-derivative models of Pavlovian reinforcement. In M. Gabriel & J. W. Moore (Eds.), Learning and computational neuroscience: Foundations of adaptive networks (pp. 497-537). Cambridge, MA: MIT Press.
121. Tolman, E. C. (1932). Purposive behavior in animals and men. New York: Appleton-Century-Crofts.
122. Tolman, E. C., & Honzik, C. H. (1930). Introduction and removal of reward, and maze performance in rats. University of California Publications in Psychology, 4, 257-275.
123. Vaughan, W. (1982). Choice and the Rescorla-Wagner model. In M. L. Commons, R. J. Herrnstein, & H. Rachlin (Eds.), Quantitative analyses of behavior: Vol. 2. Matching and maximizing accounts. Cambridge, MA: Ballinger.
124. Vogel, J. R., Mikulka, P. J., & Spear, N. E. (1966). Effect of interpolated extinction and level of training on the "depression" effect. Journal of Experimental Psychology, 72, 51-60.
125. Weinstock, S. (1954). Resistance to extinction of a running response following partial reinforcement under widely spaced trials. Journal of Comparative and Physiological Psychology, 47, 318-322.
126. Weinstock, S. (1958). Acquisition and extinction of a partially reinforced running response at a 24-hour intertrial interval. Journal of Experimental Psychology, 46, 151-158.
127. Weisman, R. G. (1969). Some determinants of inhibitory stimulus control. Journal of the Experimental Analysis of Behavior, 12, 443-450.
128. Williams, B. A. (1981). The following schedule of reinforcement as a fundamental determinant of steady state contrast in multiple schedules. Journal of the Experimental Analysis of Behavior, 35, 293-310.
129. Williams, B. A. (1983). Another look at contrast in multiple schedules. Journal of the Experimental Analysis of Behavior, 39, 345-384.
130. Williams, B. A. (1994). The role of probability of reinforcement in models of choice. Psychological Review, 101, 704-707.
131. Williams, B. A., & Royalty, P. (1989). A test of the melioration theory of matching. Journal of Experimental Psychology: Animal Behavior Processes, 15, 99-113.
132. Wilson, W., Weiss, E. J., & Amsel, A. (1955). Two tests of the Sheffield hypothesis concerning resistance to extinction, partial reinforcement, and distribution of practice. Journal of Experimental Psychology, 50, 51-60.
133. Wilton, R. N., & Clements, R. O. (1971). Behavioral contrast as a function of the duration of an immediately preceding period of extinction. Journal of the Experimental Analysis of Behavior, 16, 425-428.