Simple artificial neural networks that match probability and exploit and explore when confronting a multiarmed bandit

IEEE Transactions on Neural Networks

Volumn 20, Issue 8, 2009, Pages 1368-1371

  Dawson, Michael R W ^a   Dupuis, Brian ^a   Spetch, Marcia L ^a   Kelly, Debbie M ^b  

^a UNIVERSITY OF ALBERTA   (Canada)

^b UNIVERSITY OF SASKATCHEWAN   (Canada)

Author keywords

Instrumental learning; Multiarmed bandit; Operant conditioning; Perceptron; Probability matching

Indexed keywords

INSTRUMENTAL LEARNING; MULTIARMED BANDIT; OPERANT CONDITIONING; PERCEPTRON; PROBABILITY MATCHING;

BACKPROPAGATION; EDUCATION; NEURAL NETWORKS; PATTERN RECOGNITION SYSTEMS; PROBABILITY; REINFORCEMENT;

LEARNING ALGORITHMS;

ANIMAL; ARTICLE; ARTIFICIAL INTELLIGENCE; ARTIFICIAL NEURAL NETWORK; COMPUTER SIMULATION; DECISION MAKING; HUMAN; INSTRUMENTAL CONDITIONING; PROBABILITY; REINFORCEMENT;

ANIMALS; ARTIFICIAL INTELLIGENCE; CHOICE BEHAVIOR; COMPUTER SIMULATION; CONDITIONING, OPERANT; HUMANS; NEURAL NETWORKS (COMPUTER); PROBABILITY; REINFORCEMENT (PSYCHOLOGY); REINFORCEMENT SCHEDULE;

EID: 68949216971     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2009.2025588     Document Type: Article

References (39)

1. R. J. Herrnstein, "Relative and absolute strength of response as a function of frequency of reinforcement," J. Exp. Anal. Behav., vol. 4, pp. 267-272, 1961.
2. N. Vulkan, "An economist's perspective on probability matching," J. Econom. Surv., vol. 14, pp. 101-118, Feb. 2000.
3. J. C. Gittins, Multi-Armed Bandit Allocation Indices. Chichester, U.K.: Wiley, 1989.
4. J. J. McDowell, "On the classic and modern theories of matching," J. Exp. Anal. Behav., vol. 84, pp. 111-127, Jul. 2005.
5. P. de Villiers and R. J. Herrnstein, "Toward a law of response strength," Psychol. Bull., vol. 83, pp. 1131-1153, 1976.
6. M. Davison and D. McCarthy, The Matching Law : A Research Review. Hillsdale, N.J.: L. Erlbaum, 1988.
7. P. de Villiers, "Choice in concurrent schedules and a quantitative formulation of the law of effect," in Handbook of Operant Behavior W.K. Honig and J. E. R. Staddon, Eds. Englewood Cliffs, NJ: Prentice-Hall, 1977, pp. 233-287.
8. R. J. Herrnstein, The Matching Law : Papers in Psychology and Economics. New York: Harvard Univ. Press, 1997.
9. R. J. Herrnstein and D. H. Loveland, "Maximizing and matching on concurrent ratio schedules," J. Exp. Anal. Behav., vol. 24, pp. 107-116, 1975.
10. F. Rosenblatt, "The perceptron: A probabilistic model for information storage and organization in the brain," Psychol. Rev., vol. 65, pp. 386-408, 1958.
11. F. Rosenblatt, Principles of Neurodynamics. Washington, DC: Spartan Books, 1962.
12. M. R. W. Dawson, Minds and Machines : Connectionism and Psychological Modeling. Malden, MA: Blackwell, 2004.
13. M. R. W. Dawson, "Connectionism and classical conditioning," Comparat. Cogn. Behav. Rev., vol. 3, Monograph, pp. 1-115, 2008.
14. M. R. W. Dawson, Connectionism : A Hands-on Approach, 1st ed. Malden, MA: Blackwell, 2005.
15. M. E. Fischer, P. A. Couvillon, and M. E. Bitterman, "Choice in honeybees as a function of the probability of reward," Animal Learn. Behav., vol. 21, pp. 187-195, Aug. 1993.
16. T. Keasar, E. Rashkovich, D. Cohen, and A. Shmida, "Bees in two-armed bandit situations: Foraging choices and possible decision mechanisms," Behav. Ecol., vol. 13, pp. 757-765, Nov.-Dec. 2002.
17. N. Longo, "Probability-learning and habit-reversal in the cockroach," Amer. J. Psychol., vol. 77, pp. 29-41, 1964.
18. Y. Niv, D. Joel, I. Meilijson, and E. Ruppin, "Evolution of reinforcement learning in uncertain environments: A simple explanation for complex foraging behaviors," Adapt. Behav., vol. 10, pp. 5-24, 2002.
19. E. R. Behrend and M. E. Bitterman, "Probability-matching in the fish," Amer. J. Psychol., vol. 74, pp. 542-551, 1961.
20. K. L. Kirk and M. E. Bitterman, "Probability-learning by the turtle," Science, vol. 148, pp. 1484-1485, 1965.
21. V. Graf, D. H. Bullock, and M. E. Bitterman, "Further experiments on probability-matching in the pigeon," J. Exp. Anal. Behav., vol. 7, pp. 151-157, 1964.
22. W. K. Estes and J. H. Straughan, "Analysis of a verbal conditioning situation in terms of statistical learning theory," J. Exp. Psychol. vol. 47, pp. 225-234, 1954.
23. M. A. L. Thathachar and P. S. Sastry, "A new approach to the design of reinforcement schemes for learning automata," IEEE Trans. Syst. Man Cybern., vol. SMC-15, no. 1, pp. 168-175, Feb. 1985.
24. L. P. Kaelbling, M. L. Littman, and A. W. Moore, "Reinforcement learning: A survey," J. Artif. Intell. Res., vol. 4, pp. 237-285, 1996.
25. M. R. W. Dawson and D. P. Schopflocher, "Autonomous processing in PDP networks," Philosoph. Psychol., vol. 5, pp. 199-219, 1992.
26. J. J. McDowell, "Computational model of selection by consequences," J. Exp. Anal. Behav., vol. 81, pp. 297-317, May 2004.
27. J. J. McDowell and Z. Ansari, "The quantitative law of effect is a robust emergent property of an evolutionary algorithm for reinforcement learning," in Advances in Artificial Life. Cambridge, MA:MIT Press, 2005, vol. 3630, pp. 413-422.
28. J. J. McDowell and M. L. Caron, "Undermatching is an emergent property of selection by consequences," Behav. Processes, vol. 75, pp. 97-106, Jun. 2007.
29. J. J. McDowell, P. L. Soto, J. Dallery, and S. Kulubekova, M. Keijzer, Ed., "A computational theory of adaptive behavior based on an evolutionary reinforcement mechanism," in Proc. Conf. Genetic Evol. Comput., New York, 2006, pp. 175-182.
30. M. Piattelli-Palmarini, "Evolution, selection and cognition: From "learning" to parameter setting in biology and in the study of language," Cognition, vol. 31, pp. 1-44, 1989.
31. J. W. Donahoe and D. C. Palmer, Learning and Complex Behavior. Boston, MA: Allyn and Bacon, 1994.
32. R. B. T. Lowry and M. R. W. Dawson, "Connectionist selectionism: A case study of parity," Neural Inf. Process. - Lett. Rev., vol. 9, pp. 59-67, 2005.
33. R. J. Herrnstein, "Derivatives of matching," Psychol. Rev., vol. 86, pp. 486-495, 1979.
34. R. S. Sutton and A. G. Barto, "Toward a modern theory of adaptive networks: Expectation and prediction," Psychol. Rev., vol. 88, pp. 135-170, 1981.
35. D. R. Shanks, The Psychology of Associative Learning. Cambridge, U.K.: Cambridge Univ. Press, 1995.
36. D. Danks, "Equilibria of the Rescorla-Wagner model," J. Math. Psychol., vol. 47, pp. 109-121, Apr. 2003.
37. R. S. Sutton and A. G. Barto, Reinforcement Learning. Cambridge, MA: MIT Press, 1998.
38. W. M. Baum, "On two types of deviation from the matching law: Bias and undermatching," J. Exp. Anal. Behav., vol. 22, pp. 231-242, 1974.
39. K. Cheng and N. S. Newcombe, "Is there a geometric module for spatial orientation? Squaring theory and evidence," Psychonom. Bull. Rev. vol. 12, pp. 1-23, Feb. 2005.