Second-Order Neural Nets for Constrained Optimization

IEEE Transactions on Neural Networks

Volumn 3, Issue 6, 1992, Pages 1021-1024

  Zhang, Shengwei ^a   Zhu, Xianing ^a   Zou, Li He ^b  

^a Exper Vision Inc ^*  (United States)

^b LOUISIANA TECH UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ALGORITHMS; CONSTRAINT THEORY; OPTIMIZATION;

COMBINATORIAL OPTIMIZATION; CONSTRAINED OPTIMIZATION;

NEURAL NETWORKS;

EID: 0026953235     PISSN: 10459227     EISSN: 19410093     Source Type: Journal    
DOI: 10.1109/72.165605     Document Type: Article

References (21)

1. R. Hecht-Nielsen, “Theory of backpropagation neural network,” in Proc. IJCNN′89, Washington, DC, vol. I, June 1989, pp. 593–605.
2. K. Hornik, M. Stinchcombe, and H. White, “Multilayer feedforward networks are universal approximators,” Neural Networks, vol. 2, pp. 359–366, 1989.
3. G. Cybenko, “Continuous valued neural networks with two hidden layers are sufficient,” Mathematics of Controls, Signals and Systems, vol. 2, pp. 303–314, 1989.
4. E.K. Blum and L.K. Li, “Approximation theory and feedforward networks,” Neural Networks, vol. 4, pp. 511–515, 1991.
5. E.B. Baum, “On the capabilities of multilayer perceptrons,” J. Complexity, vol. 4, pp. 193–215, 1988.
6. S.C. Huang and Y.F. Huang, “Bounds on number of hidden neurons in multilayer perceptrons,” IEEE Trans. Neural Networks, vol. 2, pp. 47–55, Jan. 1991.
7. M.A. Sartori and P.J. Antsaklis, “A simple method to derive bounds on the size and to train multilayer neural networks,” IEEE Trans. Neural Networks, vol. 2, pp. 467–471, July 1991.
8. L.G. Allred and G.E. Kelly, “Supervised learning technique for backpropagation networks,” in Proc. IJCNN′90, San Diego, CA, June 1990, pp. 702–709.
9. X. -H. Yu “Training algorithms for backpropagation neural network with optimal descent factor,” IEE Electron. Lett., vol. 27, pp. 1698–1700, Sept. 1990.
10. S. Chen, C.F. Cowan, S.A. Billings, and P.M. Grant, “Parallel recursive prediction error algorithm for training layered neural networks,” Int. J. Control, vol. 51, pp. 1215–1228, 1990.
11. E.D. Sontag and H.J. Sussman, “Backpropagation separates when perceptrons do,” in Proc. IJCNN′89, Washington, DC, vol. II, June 1989, pp. 639–642.
12. T. Poston, C. -N. Lee, Y. Choie, and Y. Kwon, “Local minima and back propagation,” in Proc. IJCNN′91, Seattle, WA, vol. II, July 1991, pp. 173–176.
13. J. Jordan and G. Clement, “Using the symmetries of a multilayered network to reduce the weight space,” in Proc. IJCNN′91, Seattle, WA, vol. II, July 1991, pp. 391–396. Second-Order Neural Nets for Constrained Optimization Shengwei Zhang, Xianing Zhu, and Li-He Zou Abstract-Finding a global minimum of an arbitrary function is never easy, but certain statistical systems, such as simulated annealing, can help us to achieve this goal. For deterministic systems a useful strategy in dealing with optimization is to find local minima, or points satisfying the necessary conditions for optimality. In this letter analog neural nets for constrained optimization are proposed as an analogue of Newton's algorithm in numerical analysis. The neural model is globally stable and can converge to the constrained stationary points. Nonlinear neurons are introduced into the net, which makes it possible to solve optimization problems where the variables take discrete values, i.e., combinatorial optimization.
14. X. -H. Yu and S.X. Cheng, “Adaptive implementation of minimum error rate channel equalization via backpropagation,” in Proc. ICASSP′92, San Francisco, CA paper 57.7, Mar. 1992.
15. X. -H. Yu, “On the nonexistence of local minima of backpropagation error surfaces,” in Proc. IJCNN′91, Singapore, P.R. China, Nov. 1991.
16. S. Grossberg, Studies of Mind and Brain. Boston, MA: D. Reidel, 1982.
17. D.P. Bertsekas, Constrained Optimization and Lagrange Multiplier Methods. New York: Academic, 1982.
18. E.L. Lawler, J.K. Lenstra, A.H.G. Rinnooy Kan, and D.B. Shmoys, The Traveling Salesman Problem. New York: Wiley, 1985.
19. J.C. Piatt and A.H. Barr, “Constrained differential optimization,” in Neural Information Processing Systems, D.Z. Anderson, Ed. New York: American Institute of Physics, 1988.
20. S. Zhang and X. Zhu, “Nonlinear neural nets for constrained optimization,” submitted to IEEE Trans. Neural Networks.
21. J.J. Hopfield, “Neurons with graded response have collective computational properties like those of two-state neurons,” Proc. Nat. Acad. Sci. U.S., vol. 81, pp. 3088–3092, 1984.