Multi-loss Regularized Deep Neural Network

IEEE Transactions on Circuits and Systems for Video Technology

Volumn 26, Issue 12, 2016, Pages 2273-2283

  Xu, Chunyan ^a,b   Lu, Canyi ^a   Liang, Xiaodan ^a   Gao, Junbin ^c   Zheng, Wei ^d   Wang, Tianjiang ^e   Yan, Shuicheng ^a  

^a NATIONAL UNIVERSITY OF SINGAPORE   (Singapore)

^b NANJING UNIVERSITY OF SCIENCE AND TECHNOLOGY   (China)

^c CHARLES STURT UNIVERSITY   (Australia)

^d Beijing Samsung Telecom R and D Center   (China)

^e HUAZHONG UNIVERSITY OF SCIENCE AND TECHNOLOGY   (China)

Author keywords

Deep neural network (DNN); multi loss; overfitting; visual classification

Indexed keywords

NETWORKS (CIRCUITS); VIDEO SIGNAL PROCESSING;

BENCHMARK DATASETS; GENERALIZATION CAPABILITY; IN NETWORKS; LEARNING TASKS; LOSS FUNCTIONS; OVERFITTING; STATE-OF-THE-ART METHODS; VISUAL CLASSIFICATION;

DEEP NEURAL NETWORKS;

EID: 85027574774     PISSN: 10518215     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCSVT.2015.2477937     Document Type: Article

References (35)

1. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks," in Proc. Adv. Neural Inf. Process. Syst., 2012, pp. 1097-1105.
2. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proc. IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998.
3. H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng, "Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations," in Proc. 26th Annu. Int. Conf. Mach. Learn., 2009, pp. 609-616.
4. M. Lin, Q. Chen, and S. Yan, "Network in network," in Proc. Int. Conf. Learn. Represent., 2014.
5. I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio. (2013). "Maxout networks." [Online]. Available: http://arxiv.org/abs/1302.4389
6. Y. Wei et al. (2014). "CNN: Single-label to multi-label." [Online]. Available: http://arxiv.org/abs/1406.5726
7. X. Zeng, W. Ouyang, M. Wang, and X. Wang, "Deep learning of scenespecific classifier for pedestrian detection," in Proc. 13th Eur. Conf. Comput. Vis., 2014, pp. 472-487.
8. C. Farabet, C. Couprie, L. Najman, and Y. LeCun, "Learning hierarchical features for scene labeling," IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 8, pp. 1915-1929, Aug. 2013.
9. C. Szegedy et al. (2014). "Going deeper with convolutions." [Online]. Available: http://arxiv.org/abs/1409.4842
10. D. Ciresan, U. Meier, and J. Schmidhuber, "Multi-column deep neural networks for image classification," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2012, pp. 3642-3649.
11. P. Y. Simard, D. Steinkraus, and J. C. Platt, "Best practices for convolutional neural networks applied to visual document analysis," in Proc. 7th Int. Conf. Document Anal. Recognit., vol. 2. 2003, pp. 958-963.
12. C. M. Bishop, Neural Networks for Pattern Recognition. Oxford, U.K.: Clarendon, 1995.
13. G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. (2012). "Improving neural networks by preventing co-adaptation of feature detectors." [Online]. Available: http://arxiv.org/abs/1207.0580
14. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: A simple way to prevent neural networks from overfitting," J. Mach. Learn. Res., vol. 15, no. 1, pp. 1929-1958, 2014.
15. L. Wan, M. Zeiler, S. Zhang, Y. LeCun, and R. Fergus, "Regularization of neural networks using DropConnect," in Proc. 30th Int. Conf. Mach. Learn., 2013, pp. 1058-1066.
16. M. D. Zeiler and R. Fergus. (2013). "Stochastic pooling for regularization of deep convolutional neural networks." [Online]. Available: http://arxiv.org/abs/1301.3557
17. A. Krizhevsky and G. Hinton, "Learning multiple layers of features from tiny images," M.S. thesis, Dept. Comput. Sci., Univ. Toronto, Toronto, ON, Canada, 2009.
18. T. Joachims, "Optimizing search engines using clickthrough data," in Proc. 8th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, 2002, pp. 133-142.
19. C. Burges et al., "Learning to rank using gradient descent," in Proc. 22nd Int. Conf. Mach. Learn., 2005, pp. 89-96.
20. P. Donmez, K. M. Svore, and C. J. C. Burges, "On the local optimality of LambdaRank," in Proc. 32nd Int. ACM SIGIR Conf. Res. Develop. Inf. Retr., 2009, pp. 460-467.
21. S. Mohamed and G. Rubino, "A study of real-time packet video quality using random neural networks," IEEE Trans. Circuits Syst. Video Technol., vol. 12, no. 12, pp. 1071-1083, Dec. 2002.
22. H. Choi and C. Lee, "Motion adaptive deinterlacing with modular neural networks," IEEE Trans. Circuits Syst. Video Technol., vol. 21, no. 6, pp. 844-849, Jun. 2011.
23. K. Kim, S. Lee, J.-Y. Kim, M. Kim, and H.-J. Yoo, "A configurable heterogeneous multicore architecture with cellular neural network for real-time object recognition," IEEE Trans. Circuits Syst. Video Technol., vol. 19, no. 11, pp. 1612-1622, Nov. 2009.
24. N. Sudha, A. R. Mohan, and P. K. Meher, "A self-configurable systolic architecture for face recognition system based on principal component neural network," IEEE Trans. Circuits Syst. Video Technol., vol. 21, no. 8, pp. 1071-1084, Aug. 2011.
25. G. Gkioxari, B. Hariharan, R. Girshick, and J. Malik. (2014). "R-CNNs for pose estimation and action detection." [Online]. Available: http://arxiv.org/abs/1406.5212
26. S. Li, Z.-Q. Liu, and A. B. Chan, "Heterogeneous multi-task learning for human pose estimation with deep convolutional neural network," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, Jun. 2014, pp. 488-495.
27. J. Li, Y. Tian, T. Huang, and W. Gao, "Multi-task rank learning for visual saliency estimation," IEEE Trans. Circuits Syst. Video Technol., vol. 21, no. 5, pp. 623-636, May 2011.
28. C. J. C. Burges, "From RankNet to LambdaRank to LambdaMart: An overview," Learning, vol. 11, pp. 523-581, 2010.
29. J. T. Springenberg and M. Riedmiller. (2013). "Improving deep neural networks with probabilistic maxout units." [Online]. Available: http://arxiv.org/abs/1312.6116
30. I. J. Goodfellow, Y. Bulatov, J. Ibarz, S. Arnoud, and V. Shet. (2013). "Multi-digit number recognition from street view imagery using deep convolutional neural networks." [Online]. Available: http://arxiv.org/abs/1312.6082
31. Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, "Reading digits in natural images with unsupervised feature learning," in Proc. NIPS Workshop Deep Learn. Unsupervised Feature Learn., 2011, pp. 4-12.
32. J. Deng, W. Dong, R. Socher, L. J. Li, K. Li, and F. F. Li, "ImageNet: A large-scale hierarchical image database," in Proc. IEEE Conf. CVPR, Jun. 2009, pp. 248-255.
33. K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," in Proc. Int. Conf. Learn. Represent., 2014.
34. A. Berg, J. Deng, and F. F. Li. (2010). ImagNet Large-Scale Visual Recognition Challenge. [Online]. Available: http://imagenet. org/challenges/LSVRC/2010/
35. J. Sanchez and F. Perronnin, "High-dimensional signature compression for large-scale image classification," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2011, pp. 1665-1672.