Models of trace decay, eligibility for reinforcement, and delay of reinforcement gradients, from exponential to hyperboloid

Behavioural Processes

Volumn 87, Issue 1, 2011, Pages 57-63

  Killeen, Peter R ^a  

^a ARIZONA STATE UNIVERSITY   (United States)

Author keywords

Delay of reinforcement gradients; Discounting; Forced choice paradigms; Magnitude effect; Matching paradigms; Reinforcement learning; Trace decay gradients

Indexed keywords

DATA SET; LINEARITY; MAGNITUDE; NUMERICAL MODEL; PARADIGM SHIFT; PERCEPTION; PROVENANCE;

ANIMAL BEHAVIOR; ARTICLE; CONDITIONING; DECISION MAKING; HUMAN; MATHEMATICAL ANALYSIS; NONHUMAN; REINFORCEMENT; THEORY;

ANIMALS; CHOICE BEHAVIOR; CONDITIONING (PSYCHOLOGY); HUMANS; LEARNING; MODELS, PSYCHOLOGICAL; MOTIVATION; REINFORCEMENT (PSYCHOLOGY); TIME FACTORS;

EID: 79954735886     PISSN: 03766357     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.beproc.2010.12.016     Document Type: Article

References (81)

1. Adriani W., Caprioli A., Granstrem O., Carli M., Laviola G. The spontaneously hypertensive-rat as an animal model of ADHD: evidence for impulsive and non-impulsive subpopulations. Neurosci. Biobehav. Rev. 2003, 27:639-651.
2. Baum W.M. Matching, undermatching, and overmatching in studies of choice. J. Exp. Anal. Behav. 1979, 32:269-281.
3. Baum W.M. Understanding Behaviorism: Behavior, Culture, and Evolution 2005, Blackwell, Malden, MA, p. 312.
4. Bezzina G., Cheung T.H.C., Asgari K., Hampson C.L., Body S., Bradshaw C.M., Szabadi E., Deakin J.F.W., Anderson I.M. Effects of quinolinic acid-induced lesions of the nucleus accumbens core on inter-temporal choice: a quantitative analysis. Psychopharmacology 2007, 195:71-84.
5. Boutros N., Elliffe D., Davison M. Time versus response indices affect conclusions about preference pulses. Behav. Processes 2009, 84:450-454.
6. Bower G.H. A turning point in mathematical learning theory. Psychol. Rev. 1994, 101:290-300.
7. Brannon E.M., Wusthoff C.J., Gallistel C.R., Gibbon J. Numerical subtraction in the pigeon: evidence for a linear subjective number scale. Psychol. Sci. 2001, 12:238-243.
8. Christensen D.R., Grace R.G. A decision model for steady-state choice in concurrent chains. J. Exp. Anal. Behav. 2010, 94:227-240.
9. Crowley M.A., Donahoe J.W. Matching: its acquisition and generalization. J. Exp. Anal. Behav. 2004, 82:143-159.
10. da Costa Araújo S., Body S., Hampson C.L., Langley R.W., Deakin J.F.W., Anderson I.M., Bradshaw C.M., Szabadi E. Effects of lesions of the nucleus accumbens core on inter-temporal choice: further observations with an adjusting-delay procedure. Behav. Brain Res. 2009, 202:272-277.
11. Davison M. Behavior-centric versus reinforcer-centric descriptions of behavior. PsyCrit 2006, 12(November):1-3.
12. Davison M., Baum W.M. Every reinforcer counts: reinforcer magnitude and local preference. J. Exp. Anal. Behav. 2003, 80:95-129.
13. Elliffe D., Davison M., Landon J. Relative reinforcer rates and magnitudes do not control concurrent choice independently. J. Exp. Anal. Behav. 2008, 90:169-185.
14. Escobar R., Bruner C.A. Response induction during the acquisition and maintenance of lever pressing with delayed reinforcement. J. Exp. Anal. Behav. 2007, 88:29-49.
15. Estes W.K. Toward a statistical theory of learning. Psychol. Rev. 1950, 57:94-107.
16. Estes W.K., Suppes P. Foundations of stimulus sampling theory. Contemporary Developments in Mathematical Psychology 1974.
17. Farell B., Pelli D.G. Psychophysical methods, or how to measure a threshold and why. Vision Research: A Practical Guide to Laboratory Methods 1999, Oxford Univ. Press, New York. R.H.S. Carpenter, J.G. Robson (Eds.).
18. Fox A.T., Hand D.J., Reilly M.P. Impulsive choice in a rodent model of attention-deficit/hyperactivity disorder. Behav. Brain Res. 2008, 187:146-152.
19. Grace R.C., Berg M.E., Kyonka E.G.E. Choice and timing in concurrent chains: effects of initial-link duration. Behav. Processes 2006, 71:188-200.
20. Green L., Myerson J. A discounting framework for choice with delayed and probabilistic rewards. Psychol. Bull. 2004, 130:769-792.
21. Green L., Myerson J., Holt D.D., Slevin J.R., Estle S.J. Discounting of delayed food rewards in pigeons and rats: is there a magnitude effect?. J. Exp. Anal. Behav. 2004, 81:39-50.
22. Hand D.J. Measurement Theory and Practice 2004, Oxford University Press, Inc., New York, p. 320.
23. Hinvest N.S., Anderson I.M. The effects of real versus hypothetical reward on delay and probability discounting. Q. J. Exp. Psychol. 2010, 63:1072-1084.
24. Ho M.Y., Mobini S., Chiang T.J., Bradshaw C.M., Szabadi E. Theory and method in the quantitative analysis of "impulsive choice" behaviour: implications for psychopharmacology. Psychopharmacology 1999, 146:362-372.
25. Horney J., Fantino E. Choice for conditioned reinforcers in the signaled absence of primary reinforcement. J. Exp. Anal. Behav. 1984, 41:193-201.
26. Ito M., Asaki K. Choice behavior of rats in a concurrent-chains schedule: amount and delay of reinforcement. J. Exp. Anal. Behav. 1982, 37:383-392.
27. Jenkins H.M., Boakes R.A. Observing stimulus sources that signal food or no food. J. Exp. Anal. Behav. 1973, 20:197-207.
28. Johansen E.B., Killeen P.R., Russell V.A., Tripp G., Wickens J.R., Tannock R., Williams J., Sagvolden T. Origins of altered reinforcement effects in ADHD. Behav. Brain Funct. 2009, 5:7.
29. Killeen P.R. The matching law. J. Exp. Anal. Behav. 1972, 17:489-495.
30. Killeen P.R. Incentive theory. Nebraska Symposium on Motivation 1982, vol. 1981. Response Structure and Organization, University of Nebraska Press, Lincoln.
31. Killeen P.R. Incentive theory II: models for choice. J. Exp. Anal. Behav. 1982, 38:217-232.
32. Killeen P.R. Incentive theory IV: magnitude of reward. J. Exp. Anal. Behav. 1985, 43:407-417.
33. Killeen P.R. Mathematical principles of reinforcement. Behav. Brain Sci. 1994, 17:105-172.
34. Killeen P.R. Modeling games from the 20th century. Behav. Processes 2001, 54:33-52.
35. Killeen P.R. Writing and overwriting short-term memory. Psychon. Bull. Rev. 2001, 8:18-43.
36. Killeen P.R. Gradus ad parnassum: ascending strength gradients or descending memory traces?. Behav. Brain Sci. 2005, 28:432-434.
37. Killeen P.R. An additive-utility model of delay discounting. Psychol. Rev. 2009, 116:602-619.
38. Killeen P.R., Bizo L.A. The mechanics of reinforcement. Psychon. Bull. Rev 1998, 221-238.
39. Killeen P.R., Fantino E. A unified theory of choice. J. Exp. Anal. Behav. 1990, 53:189-200.
40. Lattal K.A. Signal functions in delayed reinforcement. J. Exp. Anal. Behav. 1984, 42:239-253.
41. Lattal K.A. Delayed reinforcement of operant behavior. J. Exp. Anal. Behav. 2010, 93:129-139.
42. Liang C.H., Ho M.Y., Yang Y.Y., Tsai C.T. Testing the applicability of a multiplicative hyperbolic model of inter-temporal and risky choice in human volunteers. Chin. J. Psychol. 2010, 52:189-204.
43. Lieberman D.A., McIntosh D.C., Thomas G.V. Learning when reward is delayed: a marking hypothesis. J. Exp. Psychol. Anim. Behav. Process 1979, 5:224-242.
44. MacDonall J.S. Synthesizing concurrent interval performances. J. Exp. Anal. Behav. 2000, 74:189-206.
45. MacDonall J.S. Earning and obtaining reinforcers under concurrent interval scheduling. J. Exp. Anal. Behav. 2005, 84:167-183.
46. Maguire D.R., Rodewald A.M., Hughes C.E., Pitts R.C. Rapid acquisition of preference in concurrent schedules: effects of D-amphetamine on sensitivity to reinforcement amount. Behav. Processes 2009, 81:238-243.
47. Marcattilio A.J.M., Richards R.W. Preference for signaled versus unsignaled reinforcement delay in concurrent-chain schedules. J. Exp. Anal. Behav. 1981, 36:221-229.
48. Mazur J.E. Hyperbolic value addition and general models of animal choice. Psychol. Rev. 2001, 108:96-112.
49. Mazur J.E. Distributed versus exclusive preference in discrete-trial choice. J. Exp. Psychol. Anim. Behav. Process 2010, 36:321-333.
50. Miczek K.A., Grossman S.P. Positive conditioned suppression: effects of CS duration. J. Exp. Anal. Behav. 1971, 15:243-247.
51. Moore J., Fantino E. Choice and response contingencies. J. Exp. Anal. Behav. 1975, 23:339-347.
52. Neimark E.D., Estes W.K. Stimulus Sampling Theory 1967, Holden-Day, San Francisco.
53. Nickerson R.S. Mathematical Reasoning 2009, Patterns, Problems, Conjectures, and Proofs. Psychology Press, London.
54. Niv Y., Joel D., Meilijson I., Ruppin E. Evolution of reinforcement learning in uncertain environments: a simple explanation for complex foraging behaviors. Adapt. Behav. 2002, 10:5-24.
55. Ong E.L., White K.G. Amount-dependent temporal discounting?. Behav. Processes 2004, 66:201-212.
56. Orduña V., Hong E., Bouzas A. Interval bisection in spontaneously hypertensive rats. Behav. Processes 2007, 74:107-111.
57. Pearce J.M., Hall G. Overshadowing the instrumental conditioning of a lever-press response by a more valid predictor of the reinforcer. J. Exp. Psychol. Anim. Behav. Process 1978, 4:356-367.
58. Pitts R.C., Febbo S.M. Quantitative analyses of methamphetamine's effects on self-control choices: implications for elucidating behavioral mechanisms of drug action. Behav. Processes 2004, 66:213-233.
59. Rachlin H. On the tautology of the matching law. J. Exp. Anal. Behav. 1971, 15:249-251.
60. Rachlin H. Diminishing marginal value as delay discounting. J. Exp. Anal. Behav. 1992, 57:407-415.
61. Rachlin H. Notes on discounting. J. Exp. Anal. Behav. 2006, 85:425-435.
62. Reed P., Doughty A.H. Within-subject testing of the signaled-reinforcement effect on operant responding as measured by response rate and resistance to change. J. Exp. Anal. Behav. 2005, 83:31-45.
63. Reilly M.P., Posadas-Sanchez D., Kettle L.C., Killeen P.R. Making the trip worthwhile: do rats (Rattus norvegicus) and pigeons (Columba livia) forage prospectively?. Behav. Processes 2011, in review.
64. Richards R.W. A comparison of signaled and unsignaled delay of reinforcement. J. Exp. Anal. Behav. 1981, 35:145-152.
65. Roberts W.A., Feeney M.C. The comparative study of mental time travel. Trends Cogn. Sci. 2009, 13:271-277.
66. Roberts W.A., Feeney M.C., McMillan N., MacPherson K., Musolino E., Petter M. Do pigeons (Columba livia) study for a test?. J. Exp. Psychol. Anim. Behav. Process 2009, 35:129-142.
67. Sagvolden T., Johansen E.B., Wøien G., Walaas S.I., Storm-Mathisen J., Bergersen L.H., Hvalby Ø., Jensen V., Aase H., Russell V.A., Killeen P.R., DasBanerjee T., Middleton F.A., Faraone S.V. The spontaneously hypertensive rat model of ADHD-the importance of selecting the appropriate reference strain. Neuropharmacology 2009, 57:619-626.
68. Sanabria F., Killeen P.R. Temporal generalization accounts for response resurgence in the peak procedure. Behav. Processes 2007, 74:126-141.
69. Schaal D.W., Branch M.N. Responding of pigeons under variable-interval schedules of unsignaled, briefly signaled, and completely signaled delays to reinforcement. J. Exp. Anal. Behav. 1988, 50:33-54.
70. Schaal D.W., Branch M.N. Responding of pigeons under variable-interval schedules of signaled-delayed reinforcement: effects of delay-signal duration. J. Exp. Anal. Behav. 1990, 53:103-121.
71. Schwartz B. Positive and negative conditioned suppression in the pigeon: effects of the locus and modality of the CS. Learn. Motiv. 1976, 7:86-100.
72. Shettleworth S.J., Plowright C. Time horizons of pigeons on a two-armed bandit. Anim. Behav. 1989, 37:610-623.
73. Singh S.P., Sutton R.S. Reinforcement learning with replacing eligibility traces. Mach. Learn. 1996, 22:123-158.
74. Sutton R.S., Barto A.G. Time-derivative models of Pavlovian reinforcement. Learning and Computational Neuroscience: Foundations of Adaptive Networks 1990, MIT Press, Cambridge, MA. M. Gabriel, J. Moore (Eds.).
75. Thomas G.V., Lieberman D.A., McIntosh D.C., Ronaldson P. The role of marking when reward is delayed. J. Exp. Psychol. Anim. Behav. Process 1983, 9:401-411.
76. Timberlake W., Lucas G.A. Behavior systems and learning: from misbehavior to general principles. Contemporary Learning Theories: Instrumental Conditioning Theory and the Impact of Constraints on Learning 1990, Erlbaum, Hillsdale, NJ. S.B. Klein, R.R. Mowrer (Eds.).
77. Uttal W.R. The War Between Mentalism and Behaviorism: On the Accessibility of Mental Processes 2000, Lawrence Erlbaum Associates, Inc., Mahwah, NJ.
78. Uttal W.R. Time, Space, and Number in Physics and Psychology 2008, Sloan Publishing, Cornwall-on-Hudson, NY.
79. Wilkenfield J., Nickel M., Blakely E., Poling A. Acquisition of lever-press responding in rats with delayed reinforcement: a comparison of three procedures. J. Exp. Anal. Behav. 1992, 58:431-443.
80. Williams B.A. Associative competition in operant conditioning: blocking the response-reinforcer association. Psychon. Bull. Rev. 1999, 6:618-623.
81. Williams B.A., Dunn R. Preference for conditioned reinforcement. J. Exp. Anal. Behav. 1991, 55:37-46.