End-to-end learning for structured prediction energy networks

34th International Conference on Machine Learning, ICML 2017

Volumn 1, Issue , 2017, Pages 670-681

  Belanger, David ^a   Yang, Bishan ^b   McCallum, Andrew ^a  

^a UNIVERSITY OF MASSACHUSETTS   (United States)

^b CARNEGIE MELLON UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; IMAGE DENOISING; IMAGE ENHANCEMENT; LEARNING SYSTEMS; SEMANTICS;

BASELINE METHODS; ENERGY FUNCTIONS; ENERGY NETWORKS; GRADIENT-BASED OPTIMIZATION; MEMORY REQUIREMENTS; NON-CONVEX ENERGY; SEMANTIC ROLE LABELING; STRUCTURED PREDICTION;

FORECASTING;

EID: 85048667360     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (66)

1. Amos, Brandon, Xu, Lei, and Kolter, J Zico. Input-convex deep networks. ICML, 2017.
2. Andrychowicz, Marcin, Denil, Misha, Gomez, Sergio, Hoffman, Matthew W, Pfau, David, Schaul, Tom, and de Freitas, Nando. Learning to learn by gradient descent by gradient descent. NIPS, 2016.
3. Barbu, Adrian. Training an active random field for realtime image denoising. IEEE Transactions on Image Processing, 18(11):2451-2462, 2009.
4. Beck, Amir and Teboulle, Marc. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters, 31(3), 2003.
5. Bélanger, David and McCallum, Andrew. Structured prediction energy networks. In ICML, 2016.
6. Bengio, Yoshua, Simard, Patrice, and Frasconi, Paolo. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.
7. Brakel, Philemon, Stroobandt, Dirk, and Schrauwen, Benjamin. Training energy-based models for time-series imputation. JMLR, 14, 2013.
8. Carreras, Xavier and Márquez, Lluís. Introduction to the conll-2005 shared task: Semantic role labeling. In CoNLL, 2005.
9. Chen, Liang-Chieh, Papandreou, George, Kokkinos, Iasonas, Murphy, Kevin, and Yuille, Alan L. Semantic image segmentation with deep convolutional nets and fully connected crfs. ICLR, 2015.
10. Dabov, Kostadin, Foi, Alcssandro, Katkovnik, Vladimir, and Egiazarian, Karen. Image denoising by sparse 3-d transform-domain collaborative filtering. IEEE Transactions on image processing, 16(8):2080-2095, 2007.
11. Das, Dipanjan, Martins, André FT, and Smith, Noah A. An exact dual decomposition algorithm for shallow semantic parsing with constraints. In Conference on Lexical and Computational Semantics, 2012.
12. Domke, Justin. Generic methods for optimization-based modeling. In AISTATS, 2012.
13. Domke, Justin. Learning graphical model parameters with approximate marginal inference. Pattern Analysis and Machine Intelligence, 2013.
14. Duchi, John, Shalev-Shwartz, Shai, Singer, Yoram, and Chandra, Tushar. Efficient projections onto the 1 1-ball for learning in high dimensions. In ICML, 2008.
15. Duvenaud, David, Maclaurin, Dougal, and Adams, Ryan P. Early stopping as nonparametric variational inference. In AISTATS, 2016.
16. Fanello, Sean Ryan, Keskin, Cem, Kohli, Pushmeet, Izadi, Shahram, Shotton, Jamie, Criminisi, Antonio, Pattacini, Ugo, and Paek, Tim. Filter forests for learning datadependent convolutional kernels. In CVPR, 2014.
17. Finley, Thomas and Joachims, Thorsten. Training structural svms when exact inference is intractable. In ICML, 2008.
18. FitzGerald, Nicholas, Täckström, Oscar, Ganchev, Kuzman, and Das, Dipanjan. Semantic role labeling with neural network factors. In EMNLP, pp. 960-970, 2015.
19. Foo, Chuan-sheng, Do, Chuong B, and Ng, Andrew Y. Efficient multiple hyperparameter learning for log-linear models. In NIPS, 2008.
20. Geoffrey, Zweig and Padmanabhan, Mukund. Exact alphabeta computation in logarithmic space with application to map word graph construction. 2000.
21. Gildea, Daniel and Jurafsky, Daniel. Automatic labeling of semantic roles. Computational linguistics, 28(3):245-288, 2002.
22. Greff, Klaus, Srivastava, Rupesh K, and Schmidhuber, Jürgen. Highway and residual networks learn unrolled iterative estimation. ICLR, 2017.
23. Gregor, Karol and LeCun, Yann. Learning fast approximations of sparse coding. In ICML, 2010.
24. Gygli, M., Norouzi, M., and Angelova, A. Deep Value Networks Learn to Evaluate and Iterativcly Refine Structured Outputs. In ICML, 2017.
25. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Deep residual learning for image recognition. In CVPR, 2016.
26. Hershey, John R, Roux, Jonathan Le, and Weninger, Felix. Deep unfolding: Model-based inspiration of novel deep architectures. arXiv preprint arXiv:1409.2574, 2014.
27. Hinton, Geoffrey, Osindero, Simon, Welling, Max, and Teh, Yee-Whye. Unsupervised discovery of nonlinear structure using contrastive backpropagation. Cognitive science, 30(4):725-731, 2006.
28. Hoang, Cong Duy Vu, Haffari, Gholamreza, and Cohn, Trevor. Decoding as continuous optimization in neural machine translation. arXiv preprint:1701.02854, 2017.
29. Hochreiter, Sepp and Schmidhuber, Jürgen. Long shortterm memory. Neural computation, 1997.
30. Kingma, Diederik and Ba, Jimmy. Adam: A method for stochastic optimization. ICLR, 2015.
31. Koller, Daphne and Friedman, Nir. Probabilistic graphical models: principles and techniques. MIT press, 2009.
32. Kschischang, Frank R, Frey, Brendan J, and Loeliger, H-A. Factor graphs and the sum-product algorithm. IEEE Transactions on information theory, 47(2):498-519, 2001.
33. Lacoste-Julien, Simon, Jaggi, Martin, Schmidt, Mark, and Pletscher, Patrick. Block-coordinate frank-wolfe optimization for structural svms. arXiv preprint arXiv:1207.4747, 2012.
34. Lafferty, John, McCallum, Andrew, and Pereira, Fernando. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML, 2001.
35. LeCun, Yann, Chopra, Sumit, Hadsell, Raia, Ranzato, M, and Huang, F. A tutorial on energy-based learning. Predicting Structured Data, 1, 2006.
36. Lewis, Bil. Debugging backwards in time. arXiv preprint cs/0310016, 2003.
37. Li, Yujia and Zemel, Richard S. Mean-field networks. ICML Workshop on Learning Tractable Probabilistic Models, 2014.
38. Liu, Dong C and Nocedal, Jorge. On the limited memory bfgs method for large scale optimization. Mathematical programming, 45(1):503-528, 1989.
39. Maclaurin, Dougal, Duvenaud, David, and Adams, Ryan P. Gradient-based hyperparameter optimization through reversible learning. In ICML, 2015.
40. Martins, André FT, Figeuiredo, Mario AT, Aguiar, Pedro MQ, Smith, Noah A, and Xing, Eric P. An augmented lagrangian approach to constrained map inference. In ICML, 2011.
41. Metz, Luke, Poole, Ben, Pfau, David, and Sohl-Dickstein, Jascha. Unrolled generative adversarial networks. ICLR, 2017.
42. Mikolov, Tomas, Sutskever, Ilya, Chen, Kai, Corrado, Greg S, and Dean, Jeff. Distributed representations of words and phrases and their compositionality. In NIPS, 2013.
43. Mnih, Andriy and Hinton, Geoffrey. Learning nonlinear constraints with contrastive backpropagation. In IJCNN, 2005.
44. Newcombe, Richard A, Izadi, Shahram, Hilliges, Otmar, Molyneaux, David, Kim, David, Davison, Andrew J, Kohi, Pushmeet, Shotton, Jamie, Hodges, Steve, and Fitzgibbon, Andrew. Kinectfusion: Real-time dense surface mapping and tracking. In IEEE international symposium on Mixed and augmented reality, 2011.
45. Ngiam, Jiquan, Chen, Zhenghao, Koh, Pang W, and Ng, Andrew Y. Learning deep energy models. In ICML, 2011.
46. Pearlmutter, Barak A. Fast exact multiplication by the hessian. Neural computation, 6(1):147-160, 1994.
47. Punyakanok, Vasin, Roth, Dan, and Yih, Wen-tau. The importance of syntactic parsing and inference in semantic role labeling. Computational Linguistics, 34, 2008.
48. Roth, Stefan and Black, Michael J. Fields of experts: A framework for learning image priors. In CVPR, 2005.
49. Samuel, Kegan GG and Tappen, Marshall F. Learning optimized map estimates in continuously-valued mrf models. In CVPR, 2009.
50. Schmidt, Uwe, Gao, Qi, and Roth, Stefan. A generative perspective on mrfs in low-level vision. In CVPR, 2010.
51. Srivastava, Rupesh K, Greff, Klaus, and Schmidhuber, Jürgen. Training very deep networks. In NIPS, 2015.
52. Stoyanov, Veselin, Ropson, Alexander, and Eisner, Jason. Empirical risk minimization of graphical model parameters given approximate inference, decoding, and model structure. In AISTATS, 2011.
53. Sun, Jian and Tappen, Marshall F. Learning non-local range markov random field for image restoration. In CVPR, 2011.
54. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc V. Sequence to sequence learning with neural networks. In NIPS, 2014.
55. Täckström, Oscar, Ganchev, Kuzman, and Das, Dipanjan. Efficient inference and structured learning for semantic role labeling. TACL, 2015.
56. Tappen, Marshall F, Liu, Ce, Adelson, Edward H, and Freeman, William T. Learning gaussian conditional random fields for low-level vision. In CVPR, 2007.
57. Taskar, B., Guestrin, C, and Koller, D. Max-margin Markov networks. NIPS, 2004.
58. Tompson, Jonathan J, Jain, Arjun, LeCun, Yann, and Bregler, Christoph. Joint training of a convolutional network and a graphical model for human pose estimation. In Advances in neural information processing systems, pp. 1799-1807, 2014.
59. Tsochantaridis, Ioannis, Hofmann, Thomas, Joachims, Thorsten, and Altun, Yasemin. Support vector machine learning for interdependent and structured output spaces. In ICML, 2004.
60. Vilnis, Luke, Belanger, David, Sheldon, Daniel, and McCallum, Andrew. Bethe projections for non-local inference. UAI, 2015.
61. Vincent, Pascal. A connection between score matching and denoising autoencoders. Neural Computation, 2011.
62. Wang, Shenlong, Schwing, Alex, and Urtasun, Raquel. Efficient inference of continuous markov random fields with polynomial potentials. In NIPS, 2014.
63. Wang, Shenlong, Fidler, Sanja, and Urtasun, Raquel. Proximal deep structured models. In NIPS, 2016.
64. Zhai, Shuangfci, Cheng, Yu, Lu, Wcining, and Zhang, Zhongfci. Deep structured energy based models for anomaly detection. In ICML, 2016.
65. Zheng, Shuai, Jayasumana, Sadeep, Romera-Paredes, Bernardino, Vineet, Vibhav, Su, Zhizhong, Du, Dalong, Huang, Chang, and Torr, Philip HS. Conditional random fields as recurrent neural networks. In ICCV, 2015.
66. Zhou, Jie and Xu, Wei. End-to-end learning of semantic role labeling using recurrent neural networks. In ACL, 2015.