Deep and modular neural networks

Springer Handbook of Computational Intelligence

Volumn , Issue , 2015, Pages 473-494

  Chen, Ke ^a  

^a UNIVERSITY OF MANCHESTER   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

COMPLEX NETWORKS; DEEP LEARNING; EDUCATION; LEARNING ALGORITHMS; LEARNING SYSTEMS; MOTIVATION; NETWORK ARCHITECTURE; NEURAL NETWORKS; PATTERN RECOGNITION; PROBLEM SOLVING;

BUILDING BLOCKES; HYBRID LEARNING; IN CONTEXTS; MODULAR NEURAL NETWORKS; NEURAL ARCHITECTURES; NEURAL COMPUTATIONS; PATTERN RECOGNITION TECHNIQUES; STATE OF THE ART;

DEEP NEURAL NETWORKS;

EID: 84944583733     PISSN: None     EISSN: None     Source Type: Book    
DOI: 10.1007/978-3-662-43505-2     Document Type: Chapter

References (91)

1. E.R. Kandel, J.H. Schwartz, T.M. Jessell: Principle of Neural Science, 4th edn. (McGraw-Hill, New York 2000)
2. G.M. Edelman: Neural Darwinism: Theory of Neural Group Selection (Basic Books, New York 1987)
3. J.A. Fodor: The Modularity of Mind (MIT Press, Cam-bridge 1983)
4. F. Azam: Biologically inspired modular neural networks, Ph.D. Thesis (School of Electrical and Computer Engineering, Virginia Polytechnic Institute and State University, Blacksburg 2000)
5. G. Auda, M. Kamel: Modular neural networks: A survey, Int. J. Neural Syst. 9(2), 129-151 (1999)
6. D.C. Van Essen, C.H. Anderson, D.J. Fellman: Information processing in the primate visual system, Science 255, 419-423 (1992)
7. J.H. Kaas: Why does the brain have so many visual areas?, J. Cogn. Neurosci. 1(2), 121-135 (1989)
8. G. Bugmann: Biologically plausible neural computation, Biosystems 40(1), 11-19 (1997)
9. S. Haykin: Neural Networks and Learning Machines, 3rd edn. (Prentice Hall, New York 2009)
10. M. Minsky, S. Papert: Perceptrons (MIT Press, Cam-bridge 1969)
11. D.E. Rumelhurt, G.E. Hinton, R.J. Williams: Learning internal representations by error propagation, Nature 323, 533-536 (1986)
12. Y. LeCun, L. Bottou, Y. Bengio, P. Haffner: Gradient based learning applied to document recognition, Proc. IEEE 86(9), 2278-2324 (1998)
13. G. Tesauro: Practical issues in temporal difference learning, Mach. Learn. 8(2), 257-277 (1992)
14. G. Cybenko: Approximations by superpositions of sigmoidal functions, Math. Control Signals Syst. 2(4), 302-314 (1989)
15. N. Cristianini, J. Shawe-Taylor: An Introduction to Support Vector Machines and Other Kernel-Based Learning Methods (Cambridge University Press, Cambridge 2000)
16. Y. Bengio, Y. LeCun: Scaling learning algorithms towards AI. In: Large-Scale Kernel Machines, ed. by L. Bottou, O. Chapelle, D. DeCoste, J. Weston (MIT Press, Cambridge 2006), Chap. 14
17. Y. Bengio: Learning deep architectures for AI, Found. Trends Mach. Learn. 2(1), 1-127 (2009)
18. G.E. Hinton, S. Osindero, Y. Teh: A fast learning algorithm for deep belief nets, Neural Comput. 18(9), 1527-1554 (2006)
19. Y. Bengio: Deep learning of representations for unsupervised and transfer learning, JMLR: Workshop Conf. Proc., Vol. 7 (2011) pp. 1-20
20. H. Larochelle, D. Erhan, A. Courville, J. Bergstra, Y. Bengio: An empirical evaluation of deep architectures on problems with many factors of variation, Proc. Int. Conf. Mach. Learn. (ICML) (2007) pp. 473-480
21. R. Salakhutdinov, G.E. Hinton: Learning a nonlinear embedding by preserving class neighbourhood structure, Proc. Int. Conf. Artif. Intell. Stat. (AISTATS) (2007)
22. H. Larochelle, Y. Bengio, J. Louradour, P. Lamblin: Exploring strategies for training deep neural networks, J. Mach. Learn. Res. 10(1), 1-40 (2009)
23. W.K. Wong, M. Sun: Deep learning regularized Fisher mappings, IEEE Trans. Neural Netw. 22(10), 1668-1675 (2011)
24. S. Osindero, G.E. Hinton: Modeling image patches with a directed hierarchy of Markov random field, Adv. Neural Inf. Process. Syst. (NIPS) (2007) pp. 1121-1128
25. I. Levner: Data driven object segmentation, Ph.D. Thesis (Department of Computer Science, University of Alberta, Edmonton 2008)
26. H. Mobahi, R. Collobert, J. Weston: Deep learning from temporal coherence in video, Proc. Int. Conf. Mach. Learn. (ICML) (2009) pp. 737-744
27. H. Lee, Y. Largman, P. Pham, A. Ng: Unsupervised feature learning for audio classification using con-volutional deep belief networks, Adv. Neural Inf. Process. Syst. (NIPS) (2009)
28. K. Chen, A. Salman: Learning speaker-specific characteristics with a deep neural architecture, IEEE Trans. Neural Netw. 22(11), 1744-1756 (2011)
29. K. Chen, A. Salman: Extracting speaker-specific information with a regularized Siamese deep network, Adv. Neural Inf. Process. Syst. (NIPS) (2011)
30. A. Mohamed, G.E. Dahl, G.E. Hinton: Acoustic mod-eling using deep belief networks, IEEE Trans. Audio Speech Lang. Process. 20(1), 14-22 (2012)
31. G.E. Dahl, D. Yu, L. Deng, A. Acero: Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition, IEEE Trans. Audio Speech Lang. Process. 20(1), 30-42 (2012)
32. R. Salakhutdinov, G.E. Hinton: Semantic hashing, Proc. SIGIR Workshop Inf. Retr. Appl. Graph. Model. (2007)
33. M. Ranzato, M. Szummer: Semi-supervised learning of compact document representations with deep networks, Proc. Int. Conf. Mach. Learn. (ICML) (2008)
34. A. Torralba, R. Fergus, Y. Weiss: Small codes and large databases for recognition, Proc. Int. Conf. Comput. Vis. Pattern Recogn. (CVPR) (2008) pp. 1-8
35. R. Collobert, J. Weston: A unified architecture for natural language processing: Deep neural networks with multitask learning, Proc. Int. Conf. Mach. Learn. (ICML) (2008)
36. A. Mnih, G.E. Hinton: A scalable hierarchical distributed language model, Adv. Neural Inf. Process. Syst. (NIPS) (2008)
37. J. Weston, F. Ratle, R. Collobert: Deep learning via semi-supervised embedding, Proc. Int. Conf. Mach. Learn. (ICML) (2008)
38. R. Hadsell, A. Erkan, P. Sermanet, M. Scoffier, U. Muller, Y. LeCun: Deep belief net learning in a long-range vision system for autonomous of-froad driving, Proc. Intell. Robots Syst. (IROS) (2008) pp. 628-633
39. Y. Bengio, A. Courville, P. Vincent: Representation learning: A review and new perspectives, IEEE Trans. Pattern Anal. Mach. Intell. 35(8), 1798-1827 (2013)
40. Y. Bengio, P. Lamblin, D. Popovici, H. Larochelle: Greedy layer-wise training of deep networks, Adv. Neural Inf. Process. Syst. (NIPS) (2006)
41. P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, P.A. Manzagol: Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion, J. Mach. Learn. Res. 11, 3371-3408 (2010)
42. G.E. Hinton: Training products of experts by min-imizing contrastive divergence, Neural Comput. 14(10), 1771-1800 (2002)
43. K. Kavukcuoglu, M. Ranzato, Y. LeCun: Fast inference in sparse coding algorithms with applications to object recognition. CoRR, arXiv:(2010)
44. B.A. Olshausen, D.J. Field: Sparse coding with an overcomplete basis set: A strategy employed by V1?, Vis. Res. 37, 3311-3325 (1997)
45. S. Rifai, P. Vincent, X. Muller, X. Glorot, Y. Ben-gio: Contracting auto-encoders: Explicit invariance during feature extraction, Proc. Int. Conf. Mach. Learn. (ICML) (2011)
46. S. Rifai, G. Mesnil, P. Vincent, X. Muller, Y. Ben-gio, Y. Dauphin, X. Glorot: Higher order contractive auto-encoder, Proc. Eur. Conf. Mach. Learn. (ECML) (2011)
47. M. Ranzato, C. Poultney, S. Chopra, Y. LeCun: Efficient learning of sparse representations with an energy based model, Adv. Neural Inf. Process. Syst. (NIPS) (2006)
48. M. Ranzato, Y. Boureau, Y. LeCun: Sparse feature learning for deep belief networks, Adv. Neural Inf. Process. Syst. (NIPS) (2007)
49. G.E. Hinton, R. Salakhutdinov: Reducing the dimensionality of data with neural networks, Science 313, 504-507 (2006)
50. K. Cho, A. Ilin, T. Raiko: Improved learning of Gaussian-Bernoulli restricted Boltzmann machines, Proc. Int. Conf. Artif. Neural Netw. (ICANN) (2011)
51. D. Erhan, Y. Bengio, A. Courville, P.A. Manzagol, P. Vincent, S. Bengio: Why does unsupervised pre-training help deep learning?, J. Mach. Learn. Res. 11, 625-660 (2010)
52. D.C. Ciresan, U. Meier, L.M. Gambardella, J. Schmid-huber: Deep big simple neural nets for handwritten digit recognition, Neural Comput. 22(1), 1-14 (2010)
53. M. Ranzato, A. Krizhevsky, G.E. Hinton: Factored 3-way restricted Boltzmann machines for modeling natural images, Proc. Int. Conf. Artif. Intell. Stat. (AISTATS) (2010) pp. 621-628
54. M. Ranzato, V. Mnih, G.E. Hinton: Generating more realistic images using gated MRF’s, Adv. Neural Inf. Process. Syst. (NIPS) (2010)
55. A. Courville, J. Bergstra, Y. Bengio: Unsupervised models of images by spike-and-slab RBMs, Proc. Int. Conf. Mach. Learn. (ICML) (2011)
56. H. Lee, R. Grosse, R. Ranganath, A.Y. Ng: Unsupervised learning of hierarchical representations with convolutional deep belief networks, Commun. ACM 54(10), 95-103 (2011)
57. D. Hau, K. Chen: Exploring hierarchical speech representations with a deep convolutional neural network, Proc. U.K. Workshop Comput. Intell. (UKCI) (2011)
58. Q. Le, M. Ranzato, R. Monga, M. Devin, K. Chen, G.S. Corrado, J. Dean, A.Y. Ng: Building high-level features using large scale unsupervised learning, Proc. Int. Conf. Mach. Learn. (ICML) (2012)
59. S.E. Yuksel, J.N. Wilson, P.D. Gader: Twenty years of mixture of experts, IEEE Trans. Neural Netw. Learn. Syst. 23(8), 1177-1193 (2012)
60. A. Krogh, J. Vedelsby: Neural network ensembles, cross validation, and active learning, Adv. Neural Inf. Process. Syst. (NIPS) (1995)
61. N. Ueda, R. Nakano: Generalization error of ensemble estimators, Proc. Int. Conf. Neural Netw. (ICNN) (1996) pp. 90-95
62. L.I. Kuncheva: Combining Pattern Classifiers (Wiley-Interscience, Hoboken 2004)
63. R.A. Jacobs, M.I. Jordan, S. Nowlan, G.E. Hinton: Adaptive mixture of local experts, Neural Comput. 3(1), 79-87 (1991)
64. M.I. Jordan, R.A. Jacobs: Hierarchical mixture of experts and the EM algorithm, Neural Comput. 6(2), 181-214 (1994)
65. Y. Liu, X. Yao: Simultaneous training of negatively correlated neural networks in an ensemble, IEEE Trans. Syst. Man Cybern. B 29(6), 716-725 (1999)
66. K. Chen, L. Xu, H.S. Chi: Improved learning algorithms for mixture of experts in multi-class classification, Neural Netw. 12(9), 1229-1252 (1999)
67. L.K. Hansen, P. Salamon: Neural network ensembles, IEEE Trans. Pattern Anal. Mach. Intell. 12(10), 993-1001 (1990)
68. M.P. Perrone, L.N. Cooper: Ensemble methods for hybrid neural networks. In: Artificial Neural Networks for Speech and Vision, ed. by R.J. Mammone (Chapman-Hall, New York 1993) pp. 126-142
69. T.K. Ho: The random subspace method for constructing decision forests, IEEE Trans. Pattern Anal. Mach. Intell. 20(8), 823-844 (1998)
70. K. Chen, L. Wang, H.S. Chi: Methods of combining multiple classifiers with different feature sets and their applications to text-independent speaker identification, Int. J. Pattern Recogn. Ar-tif. Intell. 11(3), 417-445 (1997)
71. J. Kittler, M. Hatef, R.P.W. Duin, J. Matas: On combining classifiers, IEEE Trans. Pattern Anal. Mach. Intell. 20(3), 226-239 (1998)
72. D.H. Wolpert: Stacked generalization, Neural Netw. 2(3), 241-259 (1992)
73. L. Xu, M.I. Jordan, G.E. Hinton: An alternative model for mixtures of experts, Adv. Neural Inf. Process. Syst. (NIPS) (1995)
74. L. Xu, A. Krzyzak, C.Y. Suen: Associative switch for combining multiple classifiers, J. Artif. Neural Netw. 1(1), 77-100 (1994)
75. K. Chen: A connectionist method for pattern classification with diverse feature sets, Pattern Recogn. Lett. 19(7), 545-558 (1998)
76. K. Chen, H.S. Chi: A method of combining multiple probabilistic classifiers through soft competition on different feature sets, Neurocomputing 20(1-3), 227-252 (1998)
77. K. Chen: On the use of different representations for speaker modeling, IEEE Trans. Syst. Man Cybern. C 35(3), 328-346 (2005)
78. Y. Yang, K. Chen: Temporal data clustering via weighted clustering ensemble with different representations, IEEE Trans. Knowl. Data Eng. 23(2), 307-320 (2011)
79. Y. Yang, K. Chen: Time series clustering via RPCL ensemble networks with different representations, IEEE Trans. Syst. Man Cybern. C 41(2), 190-199 (2011)
80. L. Xu, A. Krzyzak, C.Y. Suen: Methods of combining multiple classifiers and their applications to handwriting recognition, IEEE Trans. Syst. Man Cybern. 22(3), 418-435 (1992)
81. K. Chen, X. Yu, H.S. Chi: Combining linear discriminant functions with neural networks for supervised learning, Neural Comput. Appl. 6(1), 19-41 (1997)
82. K. Chen, L.P. Yang, X. Yu, H.S. Chi: A self-generating modular neural network architecture for supervised learning, Neurocomputing 16(1), 33-48 (1997)
83. B.L. Lu, M. Ito: Task decomposition and module combination based on class relations: A modular neural network for pattern classification, IEEE Trans. Neural Netw. Learn. Syst. 10(5), 1244-1256 (1999)
84. L. Cao: Support vector machines experts for time series forecasting, Neurocomputing 51(3), 321-339 (2003)
85. T.G. Dietterich: Ensemble learning. In: Handbook of Brain Theory and Neural Networks, ed. by M.A. Ar-bib (MIT Press, Cambridge 2002) pp. 405-408
86. Y. Freund, R.E. Schapire: Experiments with a new boosting algorithm, Proc. Int. Conf. Mach. Learn. (ICML) (1996) pp. 148-156
87. L. Breiman: Bagging predictors, Mach. Learn. 24(2), 123-140 (1996)
88. H. Schwenk, Y. Bengio: Boosting neural networks, Neural Comput. 12(8), 1869-1887 (2000)
89. J. Wang, V. Saligrama: Local supervised learning through space partitioning, Adv. Neural Inf. Process. Syst. (NIPS) (2012)
90. X.J. Zhu: Semi-supervised learning literature survey, Technical Report, School of Computer Science (University of Wisconsin, Madison 2008)
91. J. Ghosh, A. Acharya: Cluster ensembles, WIREs Data Min. Knowl. Discov. 1(2), 305-315 (2011)