Black-box policy search with probabilistic programs

Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, AISTATS 2016

Volumn , Issue , 2016, Pages 1195-1204

  van de Meent, Jan Willem ^a   Paige, Brooks ^a   Tolpin, David ^a   Wood, Frank ^a  

^a UNIVERSITY OF OXFORD   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; STOCHASTIC SYSTEMS;

BLACK BOXES; PARAMETER LEARNING; POLICY SEARCH; PROBABILISTIC PROGRAMMING; PROBABILISTIC PROGRAMS; STOCHASTIC SIMULATORS; TUNABLE PARAMETER; VARIATIONAL INFERENCE;

REINFORCEMENT LEARNING;

EID: 85002089697     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (44)

1. D. Andre and S. J. Russell. State Abstraction for Programmable Reinforcement Learning Agents. In AAAI, 2002.
2. J. Ba, V. Mnih, and K. Kavukcuoglu. Multiple object recognition with visual attention. In Proceedings of the International Conference on Learning Representations, 2014. arXiv:1412.7755.
3. J. Baxter and P. Bartlett. Direct gradient-based reinforcement learning: I. Gradient estimation algorithms. Technical report, Computer Sciences Laboratory, Australian National University, 1999.
4. J. Baxter, L. Weaver, and P. Bartlett. Direct gradient-based reinforcement learning: II. Gradient ascent algorithms and experiments. Technical report, Computer Sciences Laboratory, Australian National University, 1999.
5. T. Coster and O. Coster. Guess Who? http://theoradesign.com, 1979.
6. P. Dayan, G. E. Hinton, R. M. Neal, and R. S. Zemel. The Helmholtz machine. Neural Computation, 7(5):889–904, 1995.
7. M. P. Deisenroth, G. Neumann, and J. Peters. A Survey on Policy Search for Robotics. Foundations and Trends in Robotics, 2(2011):1–142, 2011.
8. P. Eyerich, T. Keller, and M. Helmert. High-quality policies for the Canadian traveler’s problem. In AAAI, 2010.
9. D. Fried, S. E. Shimony, A. Benbassat, and C. Wenner. Complexity of Canadian traveler problem variants. Theor. Comput. Sci., 487:1–16, 2013.
10. N. Goodman, V. Mansinghka, D. M. Roy, K. Bonawitz, and J. B. Tenenbaum. Church: a language for generative models. In Uncertainty in Artificial Intelligence, pages 220–229, 2008.
11. A. D. Gordon, T. A. Henzinger, A. V. Nori, and S. K. Rajamani. Probabilistic programming. In International Conference on Software Engineering (ICSE, FOSE track), 2014.
12. N. Hay, S. Russell, D. Tolpin, and S. Shimony. Selecting Computations: Theory and Applications. In Uncertainty in Artificial Intelligence, 2012.
13. G. Hinton, N. Srivastava, and K. Swersky. http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf.
14. M. D. Hoffman, D. M. Blei, C. Wang, and J. Paisley. Stochastic variational inference. Journal of Machine Learning Research, 14:1303–1347, 2013.
15. M. W. Hoffman, N. d. Freitas, A. Doucet, and J. R. Peters. An expectation maximization algorithm for continuous Markov Decision Processes with arbitrary reward. In International Conference on Artificial Intelligence and Statistics, pages 232–239, 2009a.
16. M. W. Hoffman, H. Kueck, N. de Freitas, and A. Doucet. New inference strategies for solving Markov decision processes using reversible jump MCMC. In Uncertainty in Artificial Intelligence, pages 223–231. AUAI Press, 2009b.
17. H. J. Kappen. Path integrals and symmetry breaking for optimal control theory. Journal of Statistical Mechanics: Theory and Experiment, 2005(11), 2005. P11011.
18. L. Kocsis and C. Szepesvári. Bandit based Monte-Carlo planning. In European Conference on Machine Learning, pages 282–293, 2006.
19. A. Kucukelbir, R. Ranganath, A. Gelman, and D. M. Blei. Automatic Variational Inference in Stan. Neural Information Processing Systems, 2015.
20. S. Levine and V. Koltun. Guided Policy Search. In International Conference on Machine Learning, volume 28, pages 1–9, 2013.
21. V. Mansinghka, D. Selsam, and Y. Perov. Venture: a higher-order probabilistic programming platform with programmable inference. arXiv preprint arXiv:1404.0099, 2014.
22. J. S. Maritz and T. Lwin. Empirical Bayes methods, volume 35. Chapman and Hall, London, 1989. ISBN 0412277603.
23. B. Milch, B. Marthi, S. Russell, D. Sontag, D. L. Ong, and A. Kolobov. Blog: Probabilistic models with unknown objects. Statistical relational learning, page 373, 2007.
24. T. Minka, J. Winn, J. Guiver, S. Webster, Y. Zaykov, B. Yangel, A. Spengler, and J. Bronskill. Infer.NET 2.6, 2014. Microsoft Research Cambridge. http://research.microsoft.com/infernet.
25. A. Mnih and K. Gregor. Neural variational inference and learning in belief networks. In Proceedings of The 31st International Conference on Machine Learning, pages 1791–1799, 2014.
26. G. Neumann. Variational Inference for Policy Search in Changing Situations. In International Conference on Machine Learning, 2011.
27. D. Nitti, V. Belle, and L. De Raedt. Planning in Discrete and Continuous Markov Decision Processes by Probabilistic Programming. In ECML PKDD, Lecture Notes in Computer Science, pages 327–342, Cham, 2015. Springer International Publishing.
28. C. H. Papadimitriou and M. Yannakakis. Shortest paths without a map. Theor. Comput. Sci., 84(1):127–150, July 1991.
29. B. A. Pearlmutter and J. M. Siskind. Using programming language theory to make automatic differentiation sound and efficient. Advances in Automatic Differentiation, pages 79–90, 2008.
30. A. Pfeffer. Figaro: An object-oriented probabilistic programming language. Technical report, 2009.
31. J. Pineau, G. Gordon, and S. Thrun. Point-based value iteration: An anytime algorithm for POMDPs. In International Joint Conference on Artificial Intelligence, pages 1025–1030, 2003.
32. R. Ranganath, S. Gerrish, and D. M. Blei. Black Box Variational Inference. In Artificial Intelligence and Statistics, 2014.
33. K. Rawlik, M. Toussaint, and S. Vijayakumar. On Stochastic Optimal Control and Reinforcement Learning by Approximate Inference. On Stochastic Optimal Control and Reinforcement Learning by Approximate Inference, (2), 2012.
34. T. Smith and R. Simmons. Heuristic Search Value Iteration for POMDPs. In Uncertainty in Artificial Intelligence, pages 520–527, Arlington, Virginia, United States, 2004. AUAI Press.
35. S. Srivastava, S. Russell, P. Ruan, and X. Cheng. First-Order Open-Universe POMDPs. In Uncertainty in Artificial Intelligence, 2014.
36. R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour. Policy Gradient Methods for Reinforcement Learning with Function Approximation. Neural Information Processing Systems, pages 1057–1063, 1999.
37. E. Todorov. Efficient computation of optimal actions. Proc. Nat. Acad. Sci. of America, 106(28):11478–11483, 2009.
38. M. Toussaint, S. Harmeling, and A. Storkey. Probabilistic inference for solving (PO)MDPs. Neural Computation, 31(December):357–373, 2006.
39. M. J. Wainwright and M. I. Jordan. Graphical Models, Exponential Families, and Variational Inference. Foundations and Trends in Machine Learning, 1(1âĂŞ2):1–305, 2008.
40. R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229–256, 1992.
41. D. Wingate and T. Weber. Automated variational inference in probabilistic programming. arXiv preprint arXiv:1301.1299, 2013.
42. D. Wingate, N. D. Goodman, D. M. Roy, L. P. Kaelbling, and J. B. Tenenbaum. Bayesian policy search with policy priors. In International Joint Conference on Artificial Intelligence, pages 1565–1570, 2011.
43. D. Wingate, C. Diuk, T. O. Donnell, J. Tenenbaum, S. Gershman, L. Labs, and J. B. Tenenbaum. Compositional Policy Priors. Technical report, Computer Science and Artificial Intelligence Laboratory, Cambridge, MA, 2013.
44. F. Wood, J. van de Meent, and V. Mansinghka. A new approach to probabilistic programming inference. In Artificial Intelligence and Statistics, pages 1024–1032, 2014.