Simulated stochastic approximation annealing for global optimization with a square-root cooling schedule

Journal of the American Statistical Association

Volumn 109, Issue 506, 2014, Pages 847-863

  Liang, Faming ^a   Cheng, Yichen ^a   Lin, Guang ^b  

^a TEXAS A AND M UNIVERSITY   (United States)

^b PACIFIC NORTHWEST NATIONAL LABORATORY   (United States)

Author keywords

Local trap; Markov chain monte carlo; Simulated annealing; Stochastic approximation monte carlo; Stochastic optimization

Indexed keywords

EID: 84987861807     PISSN: 01621459     EISSN: 1537274X     Source Type: Journal    
DOI: 10.1080/01621459.2013.872993     Document Type: Article

References (61)

1. Ali, M. M., Khompatraporn, C., and Zabinsky, Z. B. (2005), "A Numerical Evaluation of Several Stochastic Algorithms on Selected Continuous Global Optimization Test Problems," Journal of Global Optimization, 31, 635-672. [858]
2. Ali, M. M., and Storey, C. (1994), "Modified Controlled Random Search Algorithms," International Journal of Computer Mathematics, 54, 229-235. [858]
3. Amato, S., Apolloni, B., Caporali, G., Madesani, U., and Zanaboni, A. (1991), "Simulated Annealing Approach in Backpropagation," Neurocomputing, 3, 207-220. [859]
4. Andrieu, C., Moulines, É., and Priouret, P. (2005), "Stability of Stochastic Approximation Under Verifiable Conditions," SIAM Journal of Control and Optimization, 44, 283-312. [850,851,852]
5. Andrieu, C., and Robert, C. P. (2001), "Controlled MCMC for Optimal Sampling," Technical Report 0125, Cahiers de Mathématiques du Ceremade, Université Paris-Dauphine. [849]
6. Andrieu, C., and Thoms, J. (2008), "A Tutorial on Adaptive MCMC," Statistics and Computing, 18, 343-373. [849]
7. Atchadé, Y. F., and Liu, J. S. (2010), "The Wang-Landau Algorithm for Monte Carlo Computation in General State Spaces," Statistica Sinica, 20, 209-233. [849]
8. Baum, E. B., and Lang, K. J. (1991), "Constructing Hidden Units Using Examples and Queries," in Advances in Neural Information Processing Systems (3) eds. R. P. Lippmann, J. E. Moody, and D. S. Touretzky, San Mateo: Morgan Kaufmann, pp. 904-910. [859]
9. Benveniste, A., Métivier, M., and Priouret, P. (1990), Adaptive Algorithms and Stochastic Approximations, New York: Springer-Verlag. [848,850]
10. Berger, J. O. (1993), "The Present and Future of Bayesian Multivariate Analysis," in Multivariate Analysis: Future Directions, ed. C. R. Rao, Amsterdam: North-Holland, pp. 25-53. [857]
11. Broyden, C. G. (1970), "The Convergence of a Class of Double Rank Minimization Algorithms, Parts I and II," IMA Journal of Applied Mathematics, 6, 76-90 and 222-231. [859]
12. Chen, H. F. (2002), Stochastic Approximation and Its Applications, Dordrecht: Kluwer Academic Publishers. [848,850,851]
13. Chen, H. F., and Zhu, Y. M. (1986), "Stochastic Approximation Procedures With Randomly Varying Truncations," Scientia Sinica, Series A, 29, 914-926. [850]
14. Chen, M.-H., and Schmeiser, B. W. (1996), "General Hit-and-Run Monte Carlo Sampling for Evaluating Multidimensional Integrals," Operations Research Letters, 19, 161-169. [857]
15. Cybenko, G. (1989), "Approximations by Superpositions of Sigmoidal Functions," Mathematics of Control, Signals, and Systems, 2, 303-314. [858]
16. Dekkers, A., and Aarts, E. (1991), "Global Optimization and Simulated Annealing," Mathematical Programming, 50, 367-393. [852]
17. Dill, K. A. (1985), "Theory for the Folding and Stability of Globular Proteins," Biochemistry, 24, 1501-1509. [860]
18. Dorsey, R. E., and Mayer, W. J. (1995), "Genetic Algorithms for Estimation Problems With Multiple Optima, Non-differentiability, and Other Irregular Features," Journal of Business and Economic Statistics, 13, 53-66. [857]
19. Fahlman, S. E., and Lebiere, C. (1990), "The Cascade-Correlation Learning Architecture," in Advances in Neural Information Processing Systems 2, ed. D. S. Touretzky., San Mateco, CA: Morgan Kaufmann, pp. 524-532. [859]
20. Fletcher, R. (1970), "A New Approach to Variable Metric Algorithms," Computer Journal, 13, 317-322. [859]
21. Geman, S., and Geman, D. (1984), "Stochastic Relaxation, Gibbs Distributions and the Bayesian Restoration of Images," IEEE Transactions on Pattern Analysis and Machine Intelligence, 6, 721-741. [847]
22. Goldfarb, D. (1970), "A Family of Variable Metric Methods Derived by Variational Means," Mathematics Computer, 24, 23-26. [859]
23. Gu, M. G., and Kong, F. H. (1998), "A Stochastic Approximation Algorithm With Markov Chain Monte Carlo Method for Incomplete Data Estimation Problems," Proceedings of the National Academy of Sciences USA, 95, 7270-7274. [849]
24. Haario, H., and Saksman, E. (1991), "Simulated Annealing Process in General State Space," Advance in Applied Probability, 23, 866-893. [847,853]
25. Haario, H., Saksman, E., and Tamminen, J. (2001), "An Adaptive Metropolis Algorithm," Bernoulli, 7, 223-242. [847,849]
26. Hoffman, D. L., and Schmidt, P. (1981), "Testing the Restrictions Implied by the Rational Expectations Hypothesis," Journal of Econometrics, 15, 265-287. [855,856]
27. Holley, R. A., Kusuoka, S., and Stroock, D. W. (1989), "Asymptotics of the Spectral Gap With Applications to the Theory of Simulated Annealing," Journal of Functional Analysis, 83, 333-347. [847]
28. Hornik, K. (1991), "Approximation Capabilities of Multilayer Feedforward Networks," Neural Networks, 4, 251-257. [858]
29. Hsu, H.-P., Mehra, V., and Grassberger, P. (2003), "Structure Optimization in an Off-Lattice Protein Model," Physical Review E, 68, 037703. [860]
30. Kirkpatrick, S., Gelatt, C. D., and Vecchi, M. P. (1983), "Optimization by Simulated Annealing," Science, 220, 671-680. [847]
31. Kulkarni, S. R., and Horn, C. (1995), "Necessary and Sufficient Conditions for Convergence of Stochastic Approximation Algorithms Under Arbitrary Disturbances," in Proceedings of the 34th Conference on Decision & Control, New Orleans, LA, pp. 3843-3848. [851]
32. Kushner, H. K., and Clark, D. S. (1978), Stochastic Approximation Methods for Constrained and Unconstrained Systems, New York: Springer. [851]
33. Lang, K. J., and Witbrock, M. J. (1989), "Learning to Tell Two Spirals Apart," in Proceedings of the 1988 Connectionist Models, eds. D. Touretzky, G. Hinton, and T. Sejnowski, San Mateo, CA: Morgan Kaufmann, pp. 52-59. [859]
34. Lee, J., Joo, K., Kim, S.-Y., and Lee, J. (2008), "Reexamination of Structure Optimization of Off-lattice Protein AB Models by Conformational Space Annealing," Journal of Computational Chemistry, 29, 2479-2484. [860,861]
35. Liang, F. (2004), "Annealing Contour Monte Carlo for Structure Optimization in an Off-lattice Protein Model," Journal of Chemical Physics, 120, 6756-6763. [860]
36. Liang, F. (2007a), "Annealing Stochastic Approximation Monte Carlo for Neural Network Training," Machine Learning, 68, 201-233. [849,859]
37. Liang, F. (2007b), "Continuous Contour Monte Carlo for Marginal Density Estimation With an Application to a Spatial Statistical Model," Journal of Computational and Graphical Statistics, 16, 608-632. [849]
38. Liang, F. (2009a), "On the Use of Stochastic Approximation Monte Carlo for Monte Carlo Integration," Statistics & Probability Letters, 79, 581-587. [848]
39. Liang, F. (2009b), "Improving SAMC Using Smoothing Methods: Theory and Applications to Bayesian Model Selection Problems," The Annals of Statistics, 37, 2626-2654. [851]
40. Liang, F. (2011), "Annealing Evolutionary Stochastic Approximation Monte Carlo for Global Optimization," Statistics and Computing, 21, 375-393. [857,858]
41. Liang, F., Liu, C., and Carroll, R. J. (2007), "Stochastic Approximation in Monte Carlo Computation," Journal of the American Statistical Association, 102, 305-320. [847,849,850]
42. Meyn, S., and Tweedie, R. L. (2009), Markov Chain and Stochastic Stability (2nd ed.), Cambridge: Cambridge University Press. [851]
43. Nummelin, E. (1984), General Irreducible Markov Chains and Nonnegative Operators, Cambridge: Cambridge University Press. [851]
44. Owen, C. B., and Abunawass, A. M. (1993), "Applications of Simulated Annealing to the Backpropagation Model Improves Convergence," SPIE Proceedings, 1966, pp. 269-276. [859]
45. Price, W. L. (1983), "Global Optimization by Controlled Random Search," Journal of Optimization Theory and Applications, 40, 333-348. [858]
46. Robbins, H., and Monro, S. (1951), "A Stochastic Approximation Method," Annals of Mathematical Statistics, 22, 400-407. [848,849]
47. Robert, C. P., and Casella, G. (2004), Monte Carlo Statistical Methods (2nd ed.), New York: Springer. [854]
48. Roberts, G. O., and Tweedie, R. L. (1996), "Geometric Convergence and Central Limit Theorems for Multidimensional Hastings and Metropolis Algorithms," Biometrika, 83, 95-110. [851]
49. Rosenthal, J. S. (1995), "Minorization Conditions and Convergence Rate for Markov chain Monte Carlo," Journal of the American Statistical Association, 90, 558-566. [851]
50. Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986), "Learning Internal Representations by Backpropagating Errors," in Parallel Distributed Processing: Explorations in the Microstructure of Cognition (Vol. 1), eds. D. E. Rumelhart and J. L. McClelland, Cambridge, MA: MIT Press, pp. 318-362. [859]
51. Shanno, D. F. (1970), "Conditioning of Quasi-Newton Methods for Function Minimization," Mathematics Computer, 24, 647-656. [859]
52. Smith, R. L. (1984), "Efficient Monte-Carlo Procedures for Generating Points Uniformly Distributed Over Bounded Regions," Operations Research, 32, 1296-1308. [857]
53. Stillinger, F. H., and Head-Gordon, T. (1995), "Collective Aspects of Protein Folding Illustrated by a Toy Model," Physical Review E, 52, 2872-2877. [860]
54. Stillinger, F. H., Head-Gordon, T., and Hirschfeld, C. L. (1993), "Toy Model for Protein Folding," Physical Review E, 48, 1469-1477. [860]
55. Storn, R., and Price, K. (1997), "Differential Evolution: A Simple and Efficient Heuristic for Global Optimization over Continuous Spaces," Journal of Global Optimization, 11, 341-359. [858]
56. Tan, C. M. (2008), Simulated Annealing, Vienna, Austria: In-Tech. [847]
57. Tröster, A., and Dellago, C. (2005), "Wang-Landau Sampling With Self-Adaptive Range," Physical Review E, 71, 066705. [853]
58. Villalobos-Arias, M., Coello Coello, C. A., and Hernández-Lerma, O. (2006), "Asymptotic Convergence of a Simulated Annealing Algorithm for Multiobjective Optimization Problems," Mathematical Methods of Operations Research, 64, 353-362. [852]
59. Wang, F., and Landau, D. P. (2001), "Efficient, Multiple-Range Random Walk Algorithm to Calculate Density of States," Physical Review Letters, 86, 2050-2053. [848,849]
60. Wong, W. H., and Liang, F. (1997), "Dynamic Weighting in Monte Carlo and Optimization," Proceedings of the National Academy of Sciences USA, 94, 14220-14224. [859]
61. Younes, L. (1989), "Parametric Inference for Imperfectly Observed Gibb-sian Fields," Probabability Theory and Related Field, 82, 625-645. [849]