OptNet: Differentiable optimization as a layer in neural networks

34th International Conference on Machine Learning, ICML 2017

Volumn 1, Issue , 2017, Pages 179-191

  Amos, Brandon ^a   Kolter, J Zico ^a  

^a Carnegie Mellon University ^*  (United States)

Author keywords

[No Author keywords available]

Indexed keywords

LEARNING SYSTEMS; MOBILE TELECOMMUNICATION SYSTEMS; NETWORK ARCHITECTURE; NEURAL NETWORKS; QUADRATIC PROGRAMMING; SENSITIVITY ANALYSIS;

BI-LEVEL OPTIMIZATION; FULLY-CONNECTED LAYERS; INPUT AND OUTPUTS; NEURAL ARCHITECTURES; OPTIMIZATION PROBLEMS; PRIMAL-DUAL INTERIOR POINT METHODS; PRIORI INFORMATION; QUADRATIC PROGRAMS;

NETWORK LAYERS;

EID: 85046994372     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (36)

1. Amos, Brandon, Xu, Lei, and Kolter, J Zico. Input convex neural networks. In Proceedings of the International Conference on Machine Learning, 2017.
2. Belanger, David and McCallum, Andrew. Structured prediction energy networks. In Proceedings of the International Conference on Machine Learning, 2016.
3. Belanger, David, Yang, Bishan, and McCallum, Andrew. End-to-end learning for structured prediction energy networks. In Proceedings of the International Conference on Machine Learning, 2017.
4. Bertsekas, Dimitri P. Nonlinear programming. Athena scientific Belmont, 1999.
5. Bonnans, J Frédéric and Shapiro, Alexander. Perturbation analysis of optimization problems. Springer Science & Business Media, 2013.
6. Boyd, Stephen and Vandenberghe, Lieven. Convex optimization. Cambridge university press, 2004.
7. Brakel, Philemon, Stroobandt, Dirk, and Schrauwen, Benjamin. Training energy-based models for time-series imputation. Journal of Machine Learning Research, 14(1):2771-2797, 2013.
8. Chen, Liang-Chieh, Schwing, Alexander G, Yuille, Alan L, and Urtasun, Raquel. Learning deep structured models. In Proceedings of the International Conference on Machine learning, 2015.
9. Clarke, Frank H. Generalized gradients and applications. Transactions of the American Mathematical Society, 205:247-262, 1975.
10. Domke, Justin. Generic methods for optimization-based modeling. In AISTATS, volume 22, pp. 318-326, 2012.
11. Dontchev, Asen L and Rockafellar, R Tyrrell. Implicit functions and solution mappings. Springer Monogr. Math., 2009.
12. Duchi, John, Shalev-Shwartz, Shai, Singer, Yoram, and Chandra, Tushar. Efficient projections onto the 1 1-ball for learning in high dimensions. In Proceedings of the 25th international conference on Machine learning, pp. 272-279, 2008.
13. Fiacco, Anthony V and Ishizuka, Yo. Sensitivity and stability analysis for nonlinear programming. Annals of Operations Research, 27(1):215-235, 1990.
14. Goodfellow, Ian, Mirza, Mehdi, Courville, Aaron, and Bengio, Yoshua. Multi-prediction deep boltzmann machines. In Advances in Neural Information Processing Systems, pp. 548-556, 2013.
15. Goodfellow, Ian, Pouget-Abadie, Jean, Mirza, Mehdi, Xu, Bing, Warde-Farley, David, Ozair, Sherjil, Courville, Aaron, and Bengio, Yoshua. Generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 2672-2680, 2014.
16. Gould, Stephen, Fernando, Basura, Cherian, Anoop, Anderson, Peter, Santa Cruz, Rodrigo, and Guo, Edison. On differentiating parameterized argmin and argmax problems with application to bi-level optimization. arXiv preprint arXiv:1607.05447, 2016.
17. Griewank, Andreas and Walther, Andrea. Evaluating derivatives: principles and techniques of algorithmic differentiation. SIAM, 2008.
18. Johnson, Matthew, Duvenaud, David K, Wiltschko, Alex, Adams, Ryan P, and Datta, Sandeep R. Composing graphical models with neural networks for structured representations and fast inference. In Advances in Neural Information Processing Systems, pp. 2946-2954, 2016.
19. Kennedy, Michael Peter and Chua, Leon O. Neural networks for nonlinear programming. IEEE Transactions on Circuits and Systems, 35(5):554-562, 1988.
20. Kingma, Diederik and Ba, Jimmy. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
21. Kunisch, Karl and Pock, Thomas. A bilevel optimization approach for parameter learning in variational models. SIAM Journal on Imaging Sciences, 6(2):938-983, 2013.
22. LeCun, Yann, Chopra, Sumit, Hadsell, Raia, Ranzato, M, and Huang, F. A tutorial on energy-based learning. Predicting structured data, 1:0, 2006.
23. Lillo, Walter E, Loh, Mei Heng, Hui, Stefen, and Zak, Stanislaw H. On solving constrained optimization problems with neural networks: A penalty method approach. IEEE Transactions on neural networks, 4(6):931-940, 1993.
24. Lötstedt, Per. Numerical simulation of time-dependent contact and friction problems in rigid body mechanics. SIAM journal on scientific and statistical computing, 5(2):370-393, 1984.
25. Magnus, X and Neudecker, Heinz. Matrix differential calculus. New York, 1988.
26. Mairai, Julien, Bach, Francis, and Ponce, Jean. Task-driven dictionary learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(4):791-804, 2012.
27. Mattingley, Jacob and Boyd, Stephen. Cvxgen: A code generator for embedded convex optimization. Optimization and Engineering, 13(1):1-27, 2012.
28. Metz, Luke, Poole, Ben, Pfau, David, and Sohl-Dickstein, Jascha. Unrolled generative adversarial networks. arXiv preprint arXiv:1611.02163, 2016.
29. Morari, Manfred and Lee, Jay H. Model predictive control: past, present and future. Computers & Chemical Engineering, 23(4):667-682, 1999.
30. Peng, Jian, Bo, Liefeng, and Xu, Jinbo. Conditional neural fields. In Advances in neural information processing systems, pp. 1419-1427, 2009.
31. Sastry, Shankar and Bodson, Marc. Adaptive control: stability, convergence and robustness. Courier Corporation, 2011.
32. Schmidt, Uwe and Roth, Stefan. Shrinkage fields for effective image restoration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2774-2781, 2014.
33. Sra, Suvrit, Nowozin, Sebastian, and Wright, Stephen J. Optimization for machine learning. Mit Press, 2012.
34. Stoyanov, Veselin, Ropson, Alexander, and Eisner, Jason. Empirical risk minimization of graphical model parameters given approximate inference, decoding, and model structure. In AlSTATS, pp. 725-733, 2011.
35. Tappen, Marshall F, Liu, Ce, Adelson, Edward H, and Freeman, William T. Learning gaussian conditional random fields for low-level vision. In Computer Vision and Pattern Recognition, 2007. CVPR'07. IEEE Conference on, pp. 1-8. IEEE, 2007.
36. Zheng, Shuai, Jayasumana, Sadeep, Romera-Paredes, Bernardino, Vineet, Vibhav, Su, Zhizhong, Du, Dalong, Huang, Chang, and Torr, Philip HS. Conditional random fields as recurrent neural networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1529-1537, 2015.