An information-theoretic approach to curiosity-driven reinforcement learning

Theory in Biosciences

Volumn 131, Issue 3, 2012, Pages 139-148

  Still, Susanne ^a   Precup, Doina ^b  

^a University of Hawaii at Manoa   (United States)

^b MCGILL UNIVERSITY   (Canada)

Author keywords

Adaptive behavior; Curiosity; Exploration exploitation trade off; Information theory; Rate distortion theory; Reinforcement learning

Indexed keywords

ALGORITHM; ANIMAL; ARTICLE; EXPLORATORY BEHAVIOR; HUMAN; INFORMATION SCIENCE; LEARNING;

ALGORITHMS; ANIMALS; EXPLORATORY BEHAVIOR; HUMANS; INFORMATION THEORY; LEARNING;

EID: 84865114997     PISSN: 14317613     EISSN: 16117530     Source Type: Journal    
DOI: 10.1007/s12064-011-0142-z     Document Type: Article

References (36)

1. N Ay N Bertschinger R Der F Guttler E Olbrich 2008 Predictive information and explorative behavior of autonomous robots European Physical Journal B 63 329 339 10.1140/epjb/e2008-00175-0 1:CAS:528:DC%2BD1cXpt1Whsbk%3D
2. MG Azar HJ Kappen 2010 Dynamic policy programming Journal for Machine Learning Research. arXiv:1004 2027 1 26
3. Bagnell JA, Schneider J (2003) Covariant policy search. In: International Joint Conference on Artificial Intelligence (IJCAI), Acapulco, Mexico
4. W Bialek I Nemenman N Tishby 2001 Predictability, complexity and learning Neural Comput 13 2409 2463 11674845 10.1162/089976601753195969 1:STN:280:DC%2BD3Mrmslyiuw%3D%3D (Pubitemid 33594578)
5. Brafman RI, Tennenholtz M (2002) R-max-a general polynomial time algorithm for near-optimal reinforcement learning. J Mach Learn Res 3:213-231
6. G Chechnik A Globerson N Tishby Y Weiss 2005 Information bottleneck for gaussian variables Journal of Machine Learning Research 6 165 188
7. Chigirev DV, Bialek W (2004) Optimal manifold representation of data: an information theoretic perspective. In: Thrun S, Saul L, Schölkopf B (eds) Advances in neural information processing systems 16. MIT Press, Cambridge, MA
8. JP Crutchfield DP Feldman 2001 Synchronizing to the environment: Information theoretic limits on agent learning Adv in Complex Systems 4 2 251 264
9. JP Crutchfield DP Feldman 2003 Regularities unseen, randomness observed: Levels of entropy convergence Chaos 13 1 25 54 12675408 10.1063/1.1530990 (Pubitemid 36419900)
10. ET Jaynes 1957 Information Theory and Statistical Mechanics Phys Rev 106 4 620 630 10.1103/PhysRev.106.620
11. Kearns M, Singh S (Eds) (1998) Near-optimal reinforcement learning in polynomial time. In: Proceedings of the 15th International Conference on Machine Learning, pp 260-268
12. Little DY, Sommer FT (2011) Learning in embodied action-perception loops through exploration. arXiv:1112.1125v2
13. P-Y Oudeyer F Kaplan V Hafner 2007 Intrinsic motivation systems for autonomous mental development IEEE Transactions on Evolutionary Computation 11 2 265 286 10.1109/TEVC.2006.890271 (Pubitemid 46547111)
14. Pereira F, Tishby N, Lee L (1993) Distributional clustering of english words. In 30th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, pp 183-190. http://xxx.lanl.gov/pdf/ cmp-lg/9408011
15. Peters J, Muelling K, Altun Y (2010) Relative entropy policy search. In: Proceedings of the Twenty-Fourth National Conference on Artificial Intelligence (AAAI). AAAI Press, Menlo Park
16. J Peters S Schaal 2008 Reinforcement learning of motor skills with policy gradients Neural Networks 21 4 682 697 18482830 10.1016/j.neunet.2008.02.003
17. Ratitch B, Precup D (2003) Using MDP characteristics to guide exploration in reinforcement learning. In: Proceedings of ECML, pp 313-324 (Pubitemid 37230987)
18. K Rose 1998 Deterministic annealing for clustering, compression, classification, regression, and related optimization problems Proc. IEEE 86 11 2210 2239 10.1109/5.726788 (Pubitemid 128720301)
19. K Rose E Gurewitz GC Fox 1990 Statistical Mechanics and Phase Transitions in CLustering Phys. Rev. Lett 65 8 945 948 10043066 10.1103/PhysRevLett.65.945
20. Schmidhuber J (1991) Curious model-building control systems. In Proceedings of IJCNN, pp 1458-1463
21. Schmidhuber J (2009) Art and science as by-products of the search for novel patterns, or data compressible in unknown yet learnable ways. In: Multiple ways to design research. Research cases that reshape the design discipline. Swiss Design Network-et al. Edizioni, 2009, pp 98-112
22. Shannon CE (1948) A mathematical theory of communication. Bell Syst Tech J 27:379-423, 623-656
23. Shaw R (1984) The dripping faucet as a model chaotic system. Aerial Press, Santa Cruz, California
24. Singh S, Barto AG, Chentanez N (2005) Intrinsically motivated reinforcement learning. In Proceedings of NIPS, pp 1281-1288
25. Still S (2009) Information-theoretic approach to interactive learning. EPL 85 28005. doi: 10.1209/0295-5075/85/28005
26. S Still W Bialek 2004 How many clusters? An information theoretic perspective Neural Computation 16 12 2483 2506 15516271 10.1162/0899766042321751 (Pubitemid 39604007)
27. Still S, Bialek W, Bottou L (2004) Geometric clustering using the information bottleneck method. In: Thrun S, Saul LK, Schölkopf B (eds) Advances in neural information processing systems 16. MIT Press, Cambridge, MA
28. Strehl AL, Li L, Littman ML (2006) Incremental model-based learners with formal learning-time guarantees. In: Proceedings of the 22nd Conference on Uncertainty in Artificial Intelligence, Cambridge, MA
29. Sutton RS, Barto AG (1998) Reinforcement learning: an introduction. MIT Press, Cambridge
30. ME Taylor P Stone 2009 Transfer learning for reinforcement learning domains: A survey Journal of Machine Learning Research 10 1 1633 1685
31. Thrun S, Moeller K (1992) Active exploration in dynamic environments. In: Advances in Neural Information Processing Systems (NIPS) 4, San Mateo, CA, pp 531-538
32. Tishby N, Pereira F, Bialek W (1999) The information bottleneck method. In: Proceedings of the 37th Annual Allerton Conference, pp 363-377
33. Tishby N, Polani D (2010) Information theory of decisions and actions. In: Perception-reason-action cycle: models, algorithms and systems. Springer, New York
34. E Todorov 2009 Efficient computation of optimal actions PNAS 106 28 11478 11483 19574462 10.1073/pnas.0710743106 1:CAS:528:DC%2BD1MXptVKjsL8%3D
35. Watkins CJCH (1989) Learning from delayed rewards. PhD thesis, Cambridge University
36. Wingate D, Singh S (2007) On discovery and learning of models with predictive representations of state for agents with continuous actions and observations. In Proceedings of International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pp 1128-1135