Do Convolutional Neural Networks Learn Class Hierarchy?

IEEE Transactions on Visualization and Computer Graphics

Volumn 24, Issue 1, 2018, Pages 152-165

  Bilal, Alsallakh ^a   Jourabloo, Amin ^b   Ye, Mao ^a   Liu, Xiaoming ^b   Ren, Liu ^a  

^a Bosch Research   (United States)

^b MICHIGAN STATE UNIVERSITY   (United States)

Author keywords

confusion matrix; Convolutional Neural Networks; deep learning; image classification; large scale classification

Indexed keywords

CLASSIFICATION (OF INFORMATION); CONVOLUTION; DATA VISUALIZATION; DEEP LEARNING; IMAGE CLASSIFICATION; IMAGE RECOGNITION; NETWORK LAYERS; NEURAL NETWORKS; NEURONS; PERSONNEL TRAINING; VISUALIZATION;

CLASS HIERARCHIES; CONFUSION MATRICES; CONVOLUTIONAL NEURAL NETWORK; HIERARCHICAL STRUCTURES; LARGE SCALE CLASSIFICATIONS; LEARNING BEHAVIOR; MODEL CONVERGENCE; TRAINING DATA;

FEATURE EXTRACTION;

EID: 85028725738     PISSN: 10772626     EISSN: None     Source Type: Journal    
DOI: 10.1109/TVCG.2017.2744683     Document Type: Article

References (77)

1. B. Alsallakh, A. Hanbury, H. Hauser, S. Miksch, and A. Rauber. Visualmethods for analyzing probabilistic classification data. IEEE Transactions on Visualization and Computer Graphics, 20(12): 1703-1712, 2014.
2. S. Amershi, M. Chickering, S. M. Drucker, B. Lee, P. Simard, andJ. Suh. ModelTracker: Redesigning performance analysis tools for machine learning. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pages 337-346. ACM, 2015.
3. M. Aubry and B. C. Russell. Understanding deep features with computer-generated imagery. In Proceedings ofthe IEEE International Conferenceon Computer Vision (ICCV), pages 2875-2883, 2015.
4. S. Bach, A. Binder, G. Montavon, F. Klauschen, K.-R. Muller, and W. Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015.
5. E. Beauxis-Aussalet and L. Hardman. Visualization of confusion matrix for non-expert users. In IEEE Conference on Visual Analytics Science and Technology (VAST)-Poster Proceedings, 2014.
6. M. Behrisch, B. Bach, N. Henry Riche, T. Schreck, and J.-D. Fekete. Matrix reordering methods for table and network visualization. Computer Graphics Forum, 35(3):693-716, 2016.
7. A. Bendale and T. E. Boult. Towards open set deep networks. In Proceedings ofthe IEEE Conference on Computer Vision and Pattern Recognition(CVPR), pages 1563-1572, 2016.
8. J. Bertin. Semiology of graphics: diagrams, networks, maps. 1983.
9. R. Brasselet, R. Johansson, and A. Arleo. Optimal context separation of spiking haptic signals by second-order somatosensory neurons.In Advances in Neural Information Processing Systems (NIPS), pages 180188, 2009.
10. M. Brooks, S. Amershi, B. Lee, S. M. Drucker, A. Kapoor, and P. Simard. FeatureInsight: Visual support for error-driven feature ideation in text classification. In Visual Analytics Science and Technology (VAST), 2015 IEEE Conference on, pages 105-112. IEEE, 2015.
11. D. Bruckner. ML-o-scope: A diagnostic visualization system for deep machine learning pipelines. Technical report, University of California at Berkeley, 2014. UCB/EECS-2014-99.
12. N. Cao, Y.-R. Lin, and D. Gotz. Untangle map: Visual analysis of prob-abilistic multi-label data. IEEE Transactions on Visualization and Computer Graphics, 22(2):1149-1163, 2016.
13. S. Chung, C. Park, S. Suh, K. Kang, J. Choo, and B. C. Kwon. Re-VACNN: Steering convolutional neural network via real-time visual analytics. In NIPS Workshop-The Future ofInteractive Machine Learning,pages 577-585, 2016.
14. N. Dalal and B. Triggs. Histograms of oriented gradients for human de-tection. In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), volume 1, pages 886-893. IEEE, 2005.
15. J. Deng, A. C. Berg, K. Li, and L. Fei-Fei. What does classifying more than 10,000 image categories tell us In European Conference on Computer Vision (ECCV), pages 71-84. Springer, 2010.
16. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Ima-geNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248-255. IEEE, 2009.
17. J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. DeCAF: A deep convolutional activation feature for genericvisual recognition.In International Conference on Machine Learning (ICML), volume 32, pages 647-655, 2014.
18. A. Dosovitskiy and T. Brox. Inverting visual representations with convo-lutional networks. In Proceedings ofthe IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4829-4837, 2016.
19. N. Elmqvist, T.-N. Do, H. Goodell, N. Henry, and J.-D. Fekete. Zame: Interactive large-scale graph visualization. In Visualization Symposium, 2008. PacificVIS'08. IEEE Pacific, pages 215-222. IEEE, 2008.
20. D. Erhan, Y. Bengio, A. Courville, P.-A. Manzagol, P. Vincent, and S. Bengio. Why does unsupervised pre-training help deep learning Journal of Machine Learning Research, 11(Feb):625-660,2010
21. D. Erhan, P.-A. Manzagol, Y. Bengio, S. Bengio, and P. Vincent. The difficulty of training deep architectures and the effect of unsupervised pre-training. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), volume 5, pages 153-160, 2009.
22. S. Escalera, X. Baro, J. Gonzalez, M. A. Bautista, M. Madadi, M. Reyes, V. Ponce-Lopez, H. J. Escalante, J. Shotton, and I. Guyon. Chalearn looking at people challenge 2014: Dataset and results. In Workshop at the European Conference on Computer Vision (ICCV), pages 459-473. Springer, 2014.
23. G. Griffin and P. Perona. Learning and using taxonomies for fast visual categorization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-8. IEEE, 2008.
24. F. Griin, C. Rupprecht, N. Navab, and F. Tombari. A taxonomy and library for visualizing learned features in convolutional neural networks. In ICML Workshop on Visualization for Deep Learning, page 8, 2016.
25. S. Guattery and G. L. Miller. On the quality of spectral separators. SIAM Journal on Matrix Analysis and Applications, 19(3):701-719, 1998.
26. A. W. Harley. An Interactive Node-Link Visualization of Convolutional Neural Networks, pages 867-877. Springer International Publishing, 2015.
27. G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. ArXiv preprint arXiv:1503.02531, 2015.
28. L. Hubert. Some applications of graph theory and related non-metric techniques to problems of approximate seriation: The case of symmetric proximity measures. British Journal of Mathematical and Statistical Psychology, 27(2):133-153, 1974.
29. Y. Ioannou, D. Robertson, D. Zikic, P. Kontschieder, J. Shotton, M. Brown, and A. Criminisi. Decision forests, convolutional networks and the models in-between. ArXiv preprint arXiv:1603.01250, 2016.
30. M. Jaderberg, A. Vedaldi, and A. Zisserman. Deep features for text spotting. In European Conference on Computer Vision (ECCV), pages 512528. Springer, 2014.
31. Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proceedings ofthe 22nd ACM international conference on Multimedia, pages 675-678. ACM, 2014.
32. A. J. Joshi, F. Porikli, and N. P. Papanikolopoulos. Scalable active learning for multiclass image classification. IEEE transactions on Pattern Analysis and Machine Intelligence, 34(11):2259-2273, 2012.
33. A. Kapoor, B. Lee, D. Tan, and E. Horvitz. Interactive optimization for steering machine classification. In Proceedings of the SIGCHI Conference on Human Factors in ComputingSystems, pages 1343-1352. ACM, 2010.
34. A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS), pages 1097-1105, 2012.
35. J. B. Kruskal and J. M. Landwehr. Icicle plots: Better displays for hierarchical clustering. The American Statistician, 37(2):162-168, 1983.
36. Y. LeCun, Y. Bengio, and G. Hinton.Deep learning.Nature, 521(7553):436-444, 2015.
37. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to documentrecognition. Proceedings ofthe IEEE, 86(11):2278-2324, 1998.
38. Y. A. LeCun, L. Bottou, G. B. Orr, and K.-R. Muller. Efficient backprop. In Neural networks: Tricks ofthe trade, pages 9-48. Springer, 2012.
39. M. Liu, J. Shi, Z. Li, C. Li, J. Zhu, and S. Liu. Towards better analysis of deep convolutional neural networks. IEEE Transactions on Visualization and Computer Graphics, 23(1):91-100, 2017.
40. S. Liu, X. Wang, M. Liu, and J. Zhu. Towards better analysis of machine learning models: A visual analytics perspective. Visual Informatics, 2017.
41. D. G. Lowe. Object recognition from local scale-invariant features. In Proceedings ofthe IEEE international conference on Computer vision (ICCV), volume 2, pages 1150-1157. IEEE, 1999.
42. A. Mahendran and A. Vedaldi. Understanding deep image representations by inverting them. In Proceedings ofthe IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5188-5196, 2015.
43. A. Mahendran and A. Vedaldi. Visualizing deep convolutional neural networks using natural pre-images. International Journal ofComputer Vision, 120(3):233-255, 2016.
44. E. Makinen and H. Siirtola. The barycenter heuristic and the reorderable matrix. Informatica (Slovenia), 29(3):357-364, 2005.
45. G. A. Miller. WordNet: A lexical database for english. Communications of the ACM, 38(11):39-41, 1995.
46. V. N. Murthy, V. Singh, T. Chen, R. Manmatha, and D. Comaniciu. Deep decision network for multi-class image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2240-2248, 2016.
47. A. Nguyen, A. Dosovitskiy, J. Yosinski, T. Brox, and J. Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In Advances in Neurallnformation Processing Systems (NIPS), pages 3387-3395, 2016.
48. A. Nguyen, J. Yosinski, and J. Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In Proceedings ofthe IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 427-436, 2015.
49. A. Nguyen, J. Yosinski, and J. Clune. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. In ICML Workshop on Visualization for Deep Learning, 2016.
50. D. Oelke, H. Janetzko, S. Simon, K. Neuhaus, and D. A. Keim. Visual boosting in pixel-based visualizations. In Computer Graphics Forum volume 30, pages 871-880. Wiley Online Library, 2011.
51. H. Oh Song, Y. Xiang, S. Jegelka, and S. Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings ofthe IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4004-4012, 2016.
52. K. Patel, N. Bancroft, S. M. Drucker, J. Fogarty, A. J. Ko, and J. Landay. Gestalt: integrated support for implementation and analysis in machine learning. In Proceedings of the 23nd annual ACM symposium on User Interface Software and Technology (UIST), pages 37-46. ACM, 2010.
53. P. E. Rauber, S. G. Fadel, A. X. Falcao, and A. C. Telea. Visualizing the hidden activity of artificial neural networks. IEEE Transactions on Visualization and Computer Graphics, 23(1):101-110, 2017.
54. D. Ren, S. Amershi, B. Lee, J. Suh, and J. D. Williams. Squares: Supporting interactive performance analysis for multiclass classifiers. IEEE Transactions on Visualization and Computer Graphics, 23(1):61-70, 2017.
55. M. T. Ribeiro, S. Singh, and C. Guestrin. Why should i trust you: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135-1144. ACM, 2016.
56. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. ImageNet large scale visual recognition challenge. International Journal ofComputer Vision, 115(3):211-252, 2015.
57. A. Saxena. Convolutional neural networks: An illustration in TensorFlow. XRDS: Crossroads, The ACM Magazine for Students, 22(4):56-58, 2016.
58. K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In International Conference on Learning Representations (ICLR) Workshop, 2014.
59. D. Smilkov, S. Carter, D. Sculley, F. B. Viegas, and M. Wattenberg. Direct manipulation visualization of deep networks. In ICML Workshop on Visualization for Deep Learning, 2016.
60. J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller. Striving for simplicity: The all convolutional net. ArXiv preprint arXiv:1412.6806, 2014.
61. N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhut-dinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929-1958, 2014.
62. J. Stasko. Value-driven evaluation of visualizations. In Proceedings of the Fifth Workshop on Beyond Time and Errors: Novel Evaluation Methods for Visualization (BELIV), pages 46-53. ACM, 2014.
63. H. Strobelt, S. Gehrmann, B. Huber, H. Pfister, and A. M. Rush. Visual analysis of hidden state dynamics in recurrent neural networks. In Visualization in Data Science Symposium (VDS), 2016.
64. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings ofthe IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1-9, 2015.
65. J. Talbot, B. Lee, A. Kapoor, and D. S. Tan. EnsembleMatrix: interactive visualization to support machine learning with multiple classifiers. In Proceedings ofthe SIGCHI Conference on Human Factors in Computing Systems, pages 1283-1292. ACM, 2009.
66. S. Tan and K. C. Sim. Towards implicit complexity control using variable-depth deep neural networks for automatic speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5965-5969. IEEE, 2016.
67. F.-Y. Tzeng and K.-L. Ma. Opening the black box-data driven visualization of neural networks. In IEEE Visualization, pages 383-390, 2005.
68. S. Van Den Elzen and J. J. van Wijk. BaobabView: Interactive construction and analysis of decision trees. In 2011 IEEE Conference on Visual Analytics Science and Technology (VAST), pages 151-160. IEEE, 2011.
69. A. Vedaldi and K. Lenc. MatConvNet: Convolutional neural networks for MATLAB. In Proceedings ofthe 23rd ACMinternational conference on Multimedia, pages 689-692. ACM, 2015.
70. J. Wang, B. Yu, and L. Gasser. Classification visualization with shaded similarity matrix. In IEEE Visualization, 2002.
71. D. Wei, B. Zhou, A. Torrabla, and W. Freeman. Understanding intra-class knowledge inside CNN. ArXiv preprint arXiv.1507.02379, 2015.
72. Z. Yan, H. Zhang, R. Piramuthu, V. Jagadeesh, D. DeCoste, W. Di, and Y. Yu. HD-CNN: hierarchical deep convolutional neural networks for large scale visual recognition. In Proceedings ofthe IEEE International Conference on Computer Vision (ICCV), pages 2740-2748, 2015.
73. L. Yeager, G. Heinrich, J. Mancewicz, and M. Houston. Effective visualizations for training and evaluating deep models. In ICML Workshop on Visualization for Deep Learning, 2016.
74. J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson. Understanding neural networks through deep visualization. In ICML Workshop on Deep Learning, 2015.
75. M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In European Conference on Computer Vision (ECCV), pages 818-833. Springer, 2014.
76. Z. Zhang, Y. Chen, and V. Saligrama. Efficient training of very deep neural networks for supervised hashing. In Proceedings ofthe IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 14871495, 2016.
77. B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Object detectors emerge in deep scene CNNs. In International Conference on Learning Representations (ICLR), 2015.