Methods for interpreting and understanding deep neural networks

Digital Signal Processing: A Review Journal

Volumn 73, Issue , 2018, Pages 1-15

  Montavon, Grégoire ^a   Samek, Wojciech ^b   Müller, Klaus Robert ^a,c,d  

^a TECHNISCHE UNIVERSITÄT BERLIN   (Germany)

^b FRAUNHOFER HEINRICH HERTZ INSTITUTE   (Germany)

^c Korea University   (South Korea)

^d MAX PLANCK INSTITUTE FOR INFORMATICS   (Germany)

Author keywords

Activation maximization; Deep neural networks; Layer wise relevance propagation; Sensitivity analysis; Taylor decomposition

Indexed keywords

ACTIVATION ANALYSIS; NETWORK LAYERS; SENSITIVITY ANALYSIS;

ENTRY POINT; INTERPRETABILITY; LAYER-WISE; NEURAL NETWORK MODEL; TECHNICAL CHALLENGES;

DEEP NEURAL NETWORKS;

EID: 85033371689     PISSN: 10512004     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.dsp.2017.10.011     Document Type: Review

References (77)

1. Alipanahi, B., Delong, A., Weirauch, M.T., Frey, B.J., Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning. Nat. Biotechnol. 33:8 (Jul 2015), 831–838.
2. Arbabzadah, F., Montavon, G., Müller, K.-R., Samek, W., Identifying individual facial expressions by deconstructing a neural network. Pattern Recognition - 38th German Conference, GCPR 2016, Hannover, Germany, 12–15 September, 2016, Proceedings, 2016, 344–354.
3. Arras, L., Horn, F., Montavon, G., Müller, K.-R., Samek, W., “What is relevant in a text document?”: an interpretable machine learning approach. PLoS ONE, 12(8), 2017, e0181142.
4. Arras, L., Montavon, G., Müller, K., Samek, W., Explaining recurrent neural network predictions in sentiment analysis. Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, WASSA@EMNLP 2017, Copenhagen, Denmark, 8 September, 2017, 2017, 159–168.
5. Bach, S., Binder, A., Montavon, G., Klauschen, F., Müller, K.-R., Samek, W., On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, 10(7), 2015, e0130140.
6. Baehrens, D., Schroeter, T., Harmeling, S., Kawanabe, M., Hansen, K., Müller, K.-R., How to explain individual classification decisions. J. Mach. Learn. Res. 11 (2010), 1803–1831.
7. Balduzzi, D., Frean, M., Leary, L., Lewis, J.P., Ma, K.W.-D., McWilliams, B., The shattered gradients problem: if resnets are the answer, then what is the question?. Precup, D., Teh, Y.W., (eds.) Proceedings of the 34th International Conference on Machine Learning Proceedings of Machine Learning Research, vol. 70, Aug 2017, PMLR, International Convention Centre, Sydney, Australia, 342–350.
8. Bau, D., Zhou, B., Khosla, A., Oliva, A., Torralba, A., Network dissection: quantifying interpretability of deep visual representations. CoRR arXiv:1704.05796, 2017.
9. Bazen, S., Joutard, X., The Taylor Decomposition: A Unified Generalization of the Oaxaca Method to Nonlinear Models. Working papers, 2013, HAL.
10. Bengio, Y., Practical recommendations for gradient-based training of deep architectures. Neural Networks: Tricks of the Trade, second edition, 2012, 437–478.
11. Berkes, P., Wiskott, L., On the analysis and interpretation of inhomogeneous quadratic forms as receptive fields. Neural Comput. 18:8 (2006), 1868–1895.
12. Binder, A., Montavon, G., Lapuschkin, S., Müller, K.-R., Samek, W., Layer-wise relevance propagation for neural networks with local renormalization layers. Artificial Neural Networks and Machine Learning – ICANN 2016, 25th International Conference on Artificial Neural Networks, Barcelona, Spain, 6–9 September, 2016, Proceedings, Part II, 2016, 63–71.
13. Bishop, C.M., Neural Networks for Pattern Recognition. 1995, Oxford University Press, Inc., New York, NY, USA.
14. Blankertz, B., Lemm, S., Treder, M.S., Haufe, S., Müller, K.-R., Single-trial analysis and classification of ERP components — A tutorial. NeuroImage 56:2 (2011), 814–825.
15. Blankertz, B., Tomioka, R., Lemm, S., Kawanabe, M., Müller, K.-R., Optimizing spatial filters for robust EEG single-trial analysis. IEEE Signal Process. Mag. 25:1 (2008), 41–56.
16. Bojarski, M., Yeres, P., Choromanska, A., Choromanski, K., Firner, B., Jackel, L.D., Muller, U., Explaining how a deep neural network trained with end-to-end learning steers a car. CoRR arXiv:1704.07911, 2017.
17. Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., Elhadad, N., Intelligible models for healthcare: predicting pneumonia risk and hospital 30-day readmission. Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Sydney, NSW, Australia, 10–13 August, 2015, 2015, 1721–1730.
18. Chmiela, S., Tkatchenko, A., Sauceda, H.E., Poltavsky, I., Schütt, K.T., Müller, K.-R., Machine learning of accurate energy-conserving molecular force fields. Sci. Adv., 3(5), May 2017, e1603015.
19. Erhan, D., Bengio, Y., Courville, A., Vincent, P., Visualizing Higher-Layer Features of a Deep Network. Tech. Rep. 1341, Jun. 2009, University of Montreal also presented at the ICML 2009 Workshop on Learning Feature Hierarchies, Montréal, Canada.
20. Gevrey, M., Dimopoulos, I., Lek, S., Review and comparison of methods to study the contribution of variables in artificial neural network models. Ecol. Model. 160:3 (Feb 2003), 249–264.
21. Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A.C., Bengio, Y., Generative adversarial nets. Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, Montreal, Quebec, Canada, 8–13 December, 2014, 2014, 2672–2680.
22. Hansen, K., Baehrens, D., Schroeter, T., Rupp, M., Müller, K.-R., Visual interpretation of kernel-based prediction models. Mol. Inform. 30:9 (Sep 2011), 817–826.
23. Hansen, K., Biegler, F., Ramakrishnan, R., Pronobis, W., von Lilienfeld, O.A., Müller, K.-R., Tkatchenko, A., Machine learning predictions of molecular properties: accurate many-body potentials and nonlocality in chemical space. J. Phys. Chem. Lett. 6:12 (Jun 2015), 2326–2331.
24. Haufe, S., Meinecke, F.C., Görgen, K., Dähne, S., Haynes, J.-D., Blankertz, B., Bießmann, F., On the interpretation of weight vectors of linear models in multivariate neuroimaging. NeuroImage 87 (2014), 96–110.
25. Hinton, G.E., A practical guide to training restricted Boltzmann machines. Neural Networks: Tricks of the Trade, second edition, 2012, 599–619.
26. Hochreiter, S., Schmidhuber, J., LSTM can solve hard long time lag problems. Advances in Neural Information Processing Systems 9, NIPS, Denver, CO, USA, 2–5 December, 1996, 1996, 473–479.
27. Jarrett, K., Kavukcuoglu, K., Ranzato, M., LeCun, Y., What is the best multi-stage architecture for object recognition?. IEEE 12th International Conference on Computer Vision, ICCV 2009, Kyoto, Japan, 27 September – 4 October, 2009, 2009, 2146–2153.
28. Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R.B., Guadarrama, S., Darrell, T., Caffe: convolutional architecture for fast feature embedding. Proceedings of the ACM International Conference on Multimedia, MM’14, Orlando, FL, USA, 3–7 November, 2014, 2014, 675–678.
29. Khan, J., Wei, J.S., Ringnér, M., Saal, L.H., Ladanyi, M., Westermann, F., Berthold, F., Schwab, M., Antonescu, C.R., Peterson, C., Meltzer, P.S., Classification and diagnostic prediction of cancers using gene expression profiling and artificial neural networks. Nat. Med. 7:6 (Jun 2001), 673–679.
30. Kindermans, P.-J., Schütt, K.T., Alber, M., Müller, K.-R., Dähne, S., PatternNet and patternLRP – improving the interpretability of neural networks. CoRR arXiv:1705.05598, 2017.
31. Klambauer, G., Unterthiner, T., Mayr, A., Hochreiter, S., Self-normalizing neural networks. CoRR arXiv:1706.02515, 2017.
32. Krishnan, R., Sivakumar, G., Bhattacharya, P., Extracting decision trees from trained neural networks. Pattern Recognit. 32:12 (1999), 1999–2009.
33. Krizhevsky, A., Sutskever, I., Hinton, G.E., Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information, Processing Systems 2012, Proceedings of a meeting held 3–6 December, 2012, Lake Tahoe, Nevada, United States, 2012, 1106–1114.
34. Landecker, W., Thomure, M.D., Bettencourt, L.M.A., Mitchell, M., Kenyon, G.T., Brumby, S.P., Interpreting individual classifications of hierarchical networks. IEEE Symposium on Computational Intelligence and Data Mining, CIDM 2013, Singapore, 16–19 April, 2013, 2013, 32–38.
35. Lapuschkin, S., Binder, A., Montavon, G., Müller, K.-R., Samek, W., Analyzing classifiers: Fisher vectors and deep neural networks. 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas NV, USA, 27–30 June, 2016, 2016, 2912–2920.
36. Lapuschkin, S., Binder, A., Montavon, G., Müller, K.-R., Samek, W., The layer-wise relevance propagation toolbox for artificial neural networks. J. Mach. Learn. Res. 17:114 (2016), 1–5.
37. Larochelle, H., Hinton, G.E., Learning to combine foveal glimpses with a third-order Boltzmann machine. Advances in Neural Information Processing Systems, vol. 23, 2010, 1243–1251.
38. Lee, H., Grosse, R.B., Ranganath, R., Ng, A.Y., Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, Montreal, Quebec, Canada, 14–18 June, 2009, 2009, 609–616.
39. Leek, J.T., Scharpf, R.B., Bravo, H.C., Simcha, D., Langmead, B., Johnson, W.E., Geman, D., Baggerly, K., Irizarry, R.A., Tackling the widespread and critical impact of batch effects in high-throughput data. Nat. Rev. Genet. 11:10 (Sep 2010), 733–739.
40. Lemm, S., Blankertz, B., Dickhaus, T., Müller, K.-R., Introduction to machine learning for brain imaging. NeuroImage 56:2 (2011), 387–399.
41. Letham, B., Rudin, C., McCormick, T.H., Madigan, D., Interpretable classifiers using rules and Bayesian analysis: building a better stroke prediction model. Ann. Appl. Stat. 9:3 (Sep 2015), 1350–1371.
42. Li, J., Chen, X., Hovy, E.H., Jurafsky, D., Visualizing and understanding neural models in NLP. NAACL HLT 2016, the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, 12–17 June, 2016, 2016, 681–691.
43. Lipton, Z.C., The mythos of model interpretability. CoRR arXiv:1606.03490, 2016.
44. Mahendran, A., Vedaldi, A., Understanding deep image representations by inverting them. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, 7–12 June, 2015, 2015, 5188–5196.
45. Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J., Distributed representations of words and phrases and their compositionality. Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013, Proceedings of a meeting held 5–8 December, 2013, Lake Tahoe, Nevada, United States, 2013, 3111–3119.
46. Montavon, G., Lapuschkin, S., Binder, A., Samek, W., Müller, K.-R., Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recognit. 65 (2017), 211–222.
47. Montavon, G., Orr, G., Müller, K.-R., Neural Networks: Tricks of the Trade. 2nd edition, 2012, Springer Publishing Company, Inc.
48. Montavon, G., Rupp, M., Gobre, V., Vazquez-Mayagoitia, A., Hansen, K., Tkatchenko, A., Machine learning of molecular electronic properties in chemical compound space. New J. Phys., 15(9), Sep 2013, 095003.
49. Montúfar, G.F., Pascanu, R., Cho, K., Bengio, Y., On the number of linear regions of deep neural networks. Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, Montreal, Quebec, Canada, 8–13 December, 2014, 2014, 2924–2932.
50. Mordvintsev, A., Olah, C., Tyka, M., Inceptionism: going deeper into neural networks. http://googleresearch.blogspot.com/2015/06/inceptionism-going-deeper-into-neural.html, Jun. 2015.
51. Nguyen, A., Dosovitskiy, A., Yosinski, J., Brox, T., Clune, J., Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, Barcelona, Spain, 5–10 December, 2016, 2016, 3387–3395.
52. Nguyen, A., Yosinski, J., Bengio, Y., Dosovitskiy, A., Clune, J., Plug & play generative networks: conditional iterative generation of images in latent space. CoRR arXiv:1612.00005, 2016.
53. Nguyen, A., Yosinski, J., Clune, J., Multifaceted feature visualization: uncovering the different types of features learned by each neuron in deep neural networks. CoRR arXiv:1602.03616, 2016.
54. Poulin, B., Eisner, R., Szafron, D., Lu, P., Greiner, R., Wishart, D.S., Fyshe, A., Pearcy, B., Macdonell, C., Anvik, J., Visual explanation of evidence with additive classifiers. Proceedings, the Twenty-First National Conference on Artificial Intelligence and the Eighteenth Innovative Applications of Artificial Intelligence Conference, Boston, Massachusetts, USA, 16–20 July, 2006, 2006, 1822–1829.
55. Ribeiro, M.T., Singh, S., Guestrin, C., “Why should I trust you?”: explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, 13–17 August, 2016, 2016, 1135–1144.
56. Rumelhart, D.E., Hinton, G.E., Williams, R.J., Learning representations by back-propagating errors. Nature 323:6088 (Oct 1986), 533–536.
57. Samek, W., Binder, A., Montavon, G., Lapuschkin, S., Müller, K.-R., Evaluating the visualization of what a deep neural network has learned. IEEE Trans. Neural Netw. Learn. Syst. 28:11 (2017), 2660–2673.
58. Schütt, K.T., Arbabzadah, F., Chmiela, S., Müller, K.R., Tkatchenko, A., Quantum-chemical insights from deep tensor neural networks. Nat. Commun., 8, Jan 2017, 13890.
59. Selvaraju, R.R., Das, A., Vedantam, R., Cogswell, M., Parikh, D., Batra, D., Grad-CAM: why did you say that? Visual explanations from deep networks via gradient-based localization. CoRR arXiv:1610.02391, 2016.
60. Simonyan, K., Vedaldi, A., Zisserman, A., Deep inside convolutional networks: visualising image classification models and saliency maps. CoRR arXiv:1312.6034, 2013.
61. Snyder, J.C., Rupp, M., Hansen, K., Müller, K.-R., Burke, K., Finding density functionals with machine learning. Phys. Rev. Lett., 108(25), Jun 2012.
62. Soneson, C., Gerster, S., Delorenzi, M., Batch effect confounding leads to strong bias in performance estimates obtained by cross-validation. PLoS ONE, 9(6), 2014, e0100335.
63. Springenberg, J.T., Dosovitskiy, A., Brox, T., Riedmiller, M.A., Striving for simplicity: the all convolutional net. CoRR arXiv:1412.6806, 2014.
64. Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R., Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 15:1 (2014), 1929–1958.
65. Sturm, I., Lapuschkin, S., Samek, W., Müller, K.-R., Interpretable deep neural networks for single-trial EEG classification. J. Neurosci. Methods 274 (Dec 2016), 141–145.
66. Sung, A., Ranking importance of input parameters of neural networks. Expert Syst. Appl. 15 (1998), 405–411.
67. Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S.E., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A., Going deeper with convolutions. IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, 7–12 June, 2015, 2015, 1–9.
68. Taylor, B.J., Methods and Procedures for the Verification and Validation of Artificial Neural Networks. 2005, Springer-Verlag, New York, Inc., Secaucus, NJ, USA.
69. van den Oord, A., Kalchbrenner, N., Kavukcuoglu, K., Pixel recurrent neural networks. Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, 19–24 June, 2016, 2016, 1747–1756.
70. van der Maaten, L., A New Benchmark Dataset for Handwritten Character Recognition. Tech. Rep. TiCC TR 2009-002, 2009, Tilburg University.
71. Vidovic, M.M.-C., Görnitz, N., Müller, K.-R., Kloft, M., Feature importance measure for non-linear learning algorithms. CoRR arXiv:1611.07567, 2016.
72. Vidovic, M.M.-C., Kloft, M., Müller, K.-R., Görnitz, N., Ml2motif-reliable extraction of discriminative sequence motifs from learning machines. PLoS ONE, 12(3), 2017, e0174392.
73. Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A.C., Salakhutdinov, R., Zemel, R.S., Bengio, Y., Show, attend and tell: neural image caption generation with visual attention. Proceedings of the 32nd International Conference on Machine Learning, 2015, 2048–2057.
74. Zeiler, M.D., Fergus, R., Visualizing and understanding convolutional networks. Computer Vision – ECCV 2014 – 13th European Conference, Zurich, Switzerland, 6–12 September, 2014, Proceedings, Part I, 2014, 818–833.
75. Zhang, J., Lin, Z.L., Brandt, J., Shen, X., Sclaroff, S., Top-down neural attention by excitation backprop. Computer Vision – ECCV 2016 – 14th European Conference, Amsterdam, The Netherlands, 11–14 October, 2016, Proceedings, Part IV, 2016, 543–559.
76. Zhou, B., Khosla, A., Lapedriza, À., Oliva, A., Torralba, A., Learning deep features for discriminative localization. 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas NV, USA, 27–30 June, 2016, 2016, 2921–2929.
77. Zurada, J.M., Malinowski, A., Cloete, I., Sensitivity analysis for minimization of input data dimension for feedforward neural network. 1994 IEEE International Symposium on Circuits and Systems, ISCAS 1994, London, England, UK, 30 May – 2 June, 1994, 1994, 447–450.