Multi-timescale nexting in a reinforcement learning robot

Adaptive Behavior

Volumn 22, Issue 2, 2014, Pages 146-160

  Modayil, Joseph ^a   White, Adam ^a   Sutton, Richard S ^a  

^a UNIVERSITY OF ALBERTA   (Canada)

Author keywords

predictive knowledge; Reinforcement learning; robotics; temporal difference learning

Indexed keywords

EID: 84896357393     PISSN: 10597123     EISSN: 17412633     Source Type: Journal    
DOI: 10.1177/1059712313511648     Document Type: Article

References (54)

1. Becker J. D.Computer models of thought and language. Shank R. C.Colby K. M., ed. San Francisco, CA: W. H. Freeman and Company; 1973:396-434.
2. Boots B.,Siddiqi S. M.,Gordon G. J.Closing the learning-planning loop with predictive state representations.International Journal of Robotics Research. 2011;30 (7): 954-966.
3. Box G. E.,Jenkins G. M.,Reinsel G. C.Time series analysis: Forecasting and control. New York, NY: Wiley; 2011:.
4. Brogden W.Sensory pre-conditioning.Journal of Experimental Psychology. 1939;25 (4): 323-332.
5. Butz M.,Sigaud O.,Gérard P.Anticipatory behaviour in adaptive learning systems: Foundations, theories, and systems. New York, NY: Springer; 2003:.
6. Camacho E. F.,Bordons C.Model predictive control. New York, NY: Springer; 2004:.
7. Carlsson K.,Petrovic P.,Skare S.,Petersson K.,Ingvar M.Tickling expectations: Neural processing in anticipation of a sensory stimulus.Journal of Cognitive Neuroscience. 2000;12 (4): 691-703.
8. Clark A.Whatever next? Predictive brains, situated agents, and the future of cognitive science.Behavioral and Brain Sciences. 2013;36 (3): 181-204.
9. Cunningham M.Intelligence: Its organization and development. New York, NY: Academic Press; 1972:.
10. Dayan P.,Hinton G.Feudal reinforcement learning.Advances in Neural Information Processing Systems. 1993;5:271-278.
11. Degris T.,Pilarski P. M.,Sutton R. S.Model-free reinforcement learning with continuous action in practice. Proceedings of the American Control Conference; 2012; 2012. 2177.
12. Drescher G. L.Made-up minds: A constructivist approach to artificial intelligence. Cambridge, MA: MIT Press; 1991:.
13. Gilbert D.Stumbling on happiness. New York: Knopf Press; 2006:.
14. Grush R.The emulation theory of representation: Motor control, imagery, and perception.Behavioural and Brain Sciences. 2004;27:377-442.
15. Hawkins J.,Blakeslee S.On intelligence. New York: Times Books; 2004:.
16. Huron D.Sweet anticipation: Music and the psychology of expectation. New York, NY: MIT Press; 2006:.
17. Kaelbling L.Learning to achieve goals. Proceedings of International Joint Conference on Artificial Intelligence; 1993; 1993. 1094.
18. LaValle S. M.Planning algorithms. Cambridge: Cambridge University Press; 2006:.
19. Levitin D.This is your brain on music. New York: Dutton Books; 2006:.
20. Littman M. L.,Sutton R. S.,Singh S.Predictive representations of state.Advances in Neural Information Processing Systems. 2002;14:1555-1561.
21. Ljung L.System identification: Theory for the user. Englewood Cliffs, NJ: Prentice-Hall; 1998:.
22. MaeiH. R. (2011). Gradient temporal-difference learning algorithms. PhD Thesis, University of Alberta, Canada.
23. Maei H.,Sutton R. S.GQ(λ): A general gradient algorithm for temporal-difference prediction learning with eligibility traces. Proceedings of the Third Conference on Artificial General Intelligence; 2010; 2010. 91.
24. Markoff J.Google cars drive themselves, in traffic.The New York Times. 2010;:A1.
25. Modayil J.,White A.,Sutton R. S.Multi-timescale nexting in a reinforcement learning robot. From Animals to Animats 12: 12th International Conference on Simulation of Adaptive Behavior; 2012; 2012. 299.
26. Oates T.,Schmill M. D.,Cohen P. R.A method for clustering the experiences of a mobile robot that accords with human judgments. Proceedings of the Seventeenth Conference of the Association for the Advancement of Artificial Intelligence; 2000; 2000. 846.
27. Oudeyer P.-Y.,Kaplan F.,Hafner V.Intrinsic motivation systems for autonomous mental development.IEEE Transactions on Evolutionary Computation. 2007;11 (2): 265-286.
28. Pavlov I.Conditioned reflexes: An investigation of the physiological activity of the cerebral cortex. Oxford: Oxford University Press; 1927:.
29. Peters J.,Schaal S.Natural actor-critic.Neurocomputing. 2008;71 (7): 1180-1190.
30. Pezzulo G.Coordinating with the future: The anticipatory nature of representation.Minds and Machines. 2008;18 (2): 179-225.
31. Pierce D.,Kuipers B. J.Map learning with uninterpreted sensors and effectors.Artificial Intelligence. 1997;92 (1): 169-227.
32. Rescorla R.Simultaneous and successive associations in sensory preconditioning.Journal of Experimental Psychology: Animal Behavior Processes. 1980;6 (3): 207-216.
33. Singh S.Reinforcement learning with a hierarchy of abstract models. Proceedings of the Conference of the Association for the Advancement of Artificial Intelligence; 1992; 1992. 202.
34. Singh S.,James M. R.,Rudary M. R.Predictive state representations: A new theory for modeling dynamical systems. Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence; 2004; 2004. 512.
35. Sutton R. S.Learning to predict by the method of temporal differences.Machine Learning. 1988;3:9-44.
36. Sutton R. S.Integrated architectures for learning, planning, and reacting based on approximating dynamic programming. Proceedings of the Seventh International Conference on Machine Learning; 1990; 1990. 216.
37. Sutton R. S.TD models: Modeling the world at a mixture of time scales. Proceedings of the International Conference on Machine Learning; 1995; 1995. 531.
38. Sutton R. S.The grand challenge of predictive empirical abstract knowledge. Working Notes of the IJCAI-09 Workshop on Grand Challenges for Reasoning from Experiences; 2009; 2009.
39. Sutton R. S.Beyond reward: The problem of knowledge and data. Proceedings of the 21st International Conference on Inductive Logic Programming; 2012; 2012. 2.
40. Sutton R. S.,Barto A. G.Learning and computational neuroscience: Foundations of adaptive networks. Gabriel MMoore J, ed. Cambridge, MA: MIT Press; 1990:497-537.
41. Sutton R. S.,Barto A. G.Reinforcement learning: An introduction. Cambridge, MA: MIT Press; 1998:.
42. Sutton R. S.,Modayil J.,Delp M.,Degris T.,Pilarski P. M.,White A.,Precup D.Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction. Proceedings of the 10th International Conference on Autonomous Agents and Multiagent Systems; 2011; 2011. 761.
43. Sutton R. S.,Precup D.,Singh S.Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning.Artificial Intelligence. 1999;112:181-211.
44. Sutton R. S.,Szepesvári Cs.,Maei H. R.A convergent O(n) algorithm for off-policy temporal-difference learning with linear function approximation.Advances in Neural Information Processing Systems 21. 2009;:.
45. Sutton R. S.,Maei H. R.,Precup D.,Bhatnagar S.,Silver D.,Szepesvari Cs.,. Wiewiora E.Fast gradient-descent methods for temporal-difference learning with linear function approximation. Proceedings of the 26th International Conference on Machine Learning; 2009Montreal, Canada; 2009. 993.
46. Sutton R. S.,Tanner B.Temporal-difference networks.Advances in Neural Information Processing Systems 17. 2005;:1377-1384.
47. Tedrake R.,Zhang T.,Seung H.Stochastic policy gradient reinforcement learning on a simple 3D biped. Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems; 2005; 2005. 2849.
48. Thrun S.,Montemerlo M., et al.Stanley: The robot that won the DARPA grand challenge.Journal of Field Robotics. 2006;23 (9): 661-692.
49. Tolman E. C.Purposive behavior in animals and men. Berkeley, CA: University of California Press; 1951:.
50. Wang C. C.,Thorpe C.,Thrun S.Online simultaneous localization and mapping with detection and tracking of moving objects: Theory and results from a ground vehicle in crowded urban areas. Proceedings of the IEEE International Conference on Robotics and Automation; 2003; 2003. 842.
51. Welch G.,Bishop G.An Introduction to the Kalman filter. Chapel Hill, NC: Computer Science Department, University of North Carolina; 1995:.
52. White A.,Modayil J.,Sutton R. S.Scaling life-long off-policy learning. Proceedings of the Second Joint IEEE International Conference on Development and Learning and on Epigenetic Robotics; 2012; 2012.
53. Wolpert D.,Ghahramani Z.,Jordan M.An internal model for sensori-motor integration.Science. 1995;269 (5232): 1880-1882.
54. Yamashita Y.,Tani J.Emergence of functional hierarchy in a multiple timescale neural network model: A humanoid robot experiment.PLOS: Computational Biology. 2008;4 (11): 1-18.