Deep neural networks- A developmental perspective

APSIPA Transactions on Signal and Information Processing

Volumn 5, Issue , 2016, Pages

  Juang, Biing Hwang ^a  

^a GEORGIA INSTITUTE OF TECHNOLOGY   (United States)

Author keywords

Deep neural networks; Feed forward networks; Machine learning; Pattern recognition; Restrictive Boltzmann machine

Indexed keywords

DECISION THEORY; LEARNING SYSTEMS; PATTERN RECOGNITION; PATTERN RECOGNITION SYSTEMS;

BOLTZMANN MACHINES; EBB-AND-FLOW; FEED-FORWARD NETWORK; HISTORICAL REVIEW; LEARNING RESOURCE; PERFORMANCE IMPROVEMENTS; RESEARCH ACTIVITIES; STATISTICAL PATTERN RECOGNITION;

DEEP NEURAL NETWORKS;

EID: 84962114191     PISSN: None     EISSN: 20487703     Source Type: Journal    
DOI: 10.1017/ATSIP.2016.9     Document Type: Review

References (55)

1. Hinton, G.E., Salakhutdinov, R.R.: Reducing the dimensionality of data with neural networks. Science, 313 (5786) (2006), 504-507.
2. Hinton, G.E. et al.: Deep neural networks for acoustic modeling in speech recognition-The shared views of four research groups. IEEE Signal Process. Mag., 29 (6) (2012), 82-97.
3. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks, in Advances in Neural Information Processing Systems, vol. 25, NIPS, 2012.
4. Bengio, Y.: Learning deep architectures for AI. Found. Trends Mach. Learn., 2 (1) (2009), 1-127.
5. Deng, L., Yu, D.: Deep learning: Methods and applications. Found. Trends Signal Process., 7 (2014), 3-4.
6. Hinton, G.E.: Learningmultiple layers of representation. TrendsCognit. Sci., 11 (2007), 428-434.
7. Hinton, G.E., Osindero, S., Teh, Y.-W.: A fast learning algorithm for deep belief nets. Neural Comput., 18 (2006), 1527-1554.
8. LeCun, Y., Bengio, Y., Hinton, G.E.:Deep learning.Nature, 521 (2015), 436-444.
9. Duda, R.O., Hart, P.E., Stork, D.G.: Pattern Classification, 2nd ed., Wiley, New York, 2001, ISBN: 978-0-471-05669-0.
10. Cortes, C., Vapnik, V.: Support-vector networks.Mach. Learn., 20 (3) (1995), 273.
11. Juang, B.H., Katagiri, S.: Discriminative learning for minimum error classification. IEEE Trans. Signal Process., 40 (12) (1992), 3043-3054.
12. Rumelhart, D.E., McClelland, J.: Parallel Distributed Processing: Explorations in the Microstructure of Cognition, MIT Press, Cambridge, 1986.
13. Koch, C., Segev, I.: Methods in Neuronal Modeling: From Ions to Networks, 2nd ed., MIT Press, Cambridge, Massachusetts, 1999.
14. McCulloch, W., Pitts, W.: A logical calculus of ideas immanent in nervous activity. Bull. Math. Biophys., 5 (4) (1943), 115-133.
15. Hebb, D.: The Organization of Behavior, Wiley, New York, 1949.
16. Haykin, S.: Neural Networks: A Comprehensive Foundation, 2 ed., Prentice-Hall, New Jersey, 1998.
17. Hopfield, J.J.: Neural networks and physical systems with emergent collective computational abilities. Proc. Natl. Acad. Sci. USA, 79 (1982), 2554-2558.
18. Kohonen, T.: Adaptive, associative, and self-organizing functions in neural computing. Appl. Opt., 26 (23) (1987), 4910-4918.
19. Rosenblatt, F.: The perceptron: A probabilistic model for information storage and organization in the brain. Psychol. Rev., 65 (6) (1958), 386. Also, The Perceptron-a perceiving and recognizing automaton, Report 85-460-1, Cornell Aeronautical Laboratory, 1957.
20. Widrow, B., Hoff, Jr., M.E.: Adaptive Switching Circuits. IRE WESCOM Convention Record, Part.4, IRE, New York, 1960, 96-104.
21. McEliece, R.J., Posner, E.C., Rodemich, E.R., Venkatesh, S.S.: The capacity of theHopfield associativememory. IEEE Trans. Inf. Theory, 33 (4) (1987), 461-482.
22. Bruck, J.:On the convergence properties of theHopfieldmodel. Proc. IEEE, 78 (10) (1990), 1579-1585.
23. Ackley, D., Hinton, G., Sejnowski, T.: A learning algorithm for Boltzmann machines. Cognit. Sci., 9 (1) (1985), 147-169.
24. Smolensky, P.: Chapter 6: Information processing in dynamical systems: Foundations of harmony theory, in ParallelDistributed Processing: Explorations in theMicrostructure of Cognition, vol. 1: Foundations. D. Rumelhart; J. McLelland (ed.), MIT Press, Cambridge, MA, 1986, 194-281.
25. Olazaran, M.: A sociological study of the official history of the perceptrons controversy. Soc. Stud. Sci., 26 (3) (1996), 611-659.
26. Werbos, P.: Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences, Ph.D. thesis, Harvard University, 1974.
27. Werbos, P.: Backpropagation through time: What it does and how to do it. Proc. IEEE, 78 (10) (1990), 1550-1560.
28. Funahashi, K-I.: On the approximate realization of continuous mappings by neural networks. Neural Netw., 2 (3) (1989), 183-192.
29. Hornik, K., Stinchcombe, M., White, H.: Multilayer feedforward networks are universal approximators. Neural Netw., 2 (5) (1989), 359-366.
30. Juang, B.H., Katagiri, S.: Discriminative learning for minimum error classification pattern recognition. IEEE Trans. Signal Process., 40 (12) (1992), 3043-3054.
31. Kosko, B., Bidirectional associativememories. IEEE Trans. Syst.Man Cybern. 18 (1) (1988), 49-60.
32. Giles, C.L., Miller, C.B., Chen, D., Chen, H.H., Sun, G.Z., Lee, Y.C.: Learning and extracting finite state automata with secondorder recurrent neural networks. Neural Comput., 4 (3) (1992), 393.
33. Hammersley, J.M., Clifford, P.: Markov fields on finite graphs and lattices, 1971, available online at http://www.statslab.cam.ac.uk/grg/books/hammfest/hamm-cliff.pdf
34. Frank Spitzer: Markov random fields and Gibbs ensembles. Am. Math. Mont., 78 (2) (1971), 142-154. doi: 10.2307/2317621, JSTOR 2317621.
35. Lafferty, J.D., Mccallum, A.: Conditional random fields: Probabilistic models for segmenting and labeling sequence data, in Proc. Int. Conf. on Machine Learning (ICML), Williams College, Williamstown, MA, 2001.
36. Jordan, M.I.: Graphical models. Stat. Sci., 19 (1) (2004), 140-155.
37. Rue, H., Held, L.: Gaussian Markov Random Fields: Theory and Applications, CRC Press, Boca Raton, FL, 2005.
38. Dai, B., Ding, S., Wahba, G.: Multivariate Bernoulli Distribution, TECHNICAL REPORT NO. 1170, Department of Statistics, University ofWisconsin, June 6, 2012.
39. Cho, K.H., Raiko, T., Ilin, A.: Improved learning of Gaussian-Bernoulli restricted Boltzmann machines, Master's thesis, Aalto University, 2011.
40. Hinton, G.E.: Training products of experts byminimizing contrastive divergence. Neural Comput., 14 (2002), 1771-1800.
41. Carreira-Perpinan, M.A., Hinton, G.E.: On contrastive divergence learning, in Proc. AISTATS 2005, 10th Inter. Workshop on Artificial Intelligence and Statistics, Barbados, Jan. 2005.
42. The MNIST Digit Dataset. Available at http://yann.lecun.com/exdb/mnist/
43. Hochreiter, S.: Untersuchungen zu dynamischen neuronalen Netzen. Diploma thesis, Institut f. Informatik, TechnischeUniv.Munich, 1991.
44. Bengio, Y.: Artificial Neural Networks and their Application to Speech/Sequence Recognition. Ph.D. thesis, McGill University, Canada, 1991.
45. Bourlard, H., Kamp, Y.: Auto-association by multilayer perceptrons and singular value decomposition. Biol. Cybern., 59 (4-5) (1988), 291-294.
46. Bengio, Y.: Learning Deep Architectures for AI, Foundations and Trends in Machine Learning, 2009, available at http://www.iro. umontreal.ca/lisa/pointeurs/TR1312.pdf
47. Xu, Y., Du, J., Dai, L.-R., Lee, C.-H.: A regression approach to speech enhancement based on deep neural networks. IEEE/ACM Trans. Audio Speech Lang. Process. (TASLP), 23 (1) (2015), 7-19.
48. Seltzer, M.L., Yu, D., Wang, Y.: An investigation of deep neural networks for noise robust speech recognition, in 2013 IEEE Int. Conf. on Acoustics, Speech and Signal Processing (ICASSP), Vancouver, Canada, May 2013, 7398-7402.
49. Geman, S., Geman, D.: Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. IEEE Trans. Pattern Anal. Mach. Intell., 6 (6) (1984), 721-741.
50. Metropolis, N., Rosenbluth, A.W., Rosenbluth, M.N., Teller, A.H., Teller, E.: Equations of state calculations by fast computingmachines. J. Chem. Phys., 21 (6) (1953), 1087-1092.
51. Cressie, N.: The origins of kriging. Math. Geol., 22 (3) (1990).
52. Rabiner, L.R., Juang, B.H.: An introduction to hidden Markov models. IEEE ASSP Mag., 3 (1) (1986), 4-16.
53. Rabiner, L.R.: A tutorial on hidden Markov models and selected applications in speech recognition. IEEE Proc., (1989) 217-286.
54. Rabiner, L.R., Juang, B.H.: Fundamental of Speech Recognition, ISBN-13: 978-0130151575, Prentice-Hall, New Jersey, 1993.
55. Pan, J., Liu, C., Wang, Z., Hu, Y., Jiang, H.: Investigations of deep neural networks for large vocabulary continuous speech recognition: WhyDNNSurpassesGMMsin acoustic modelling, in Proc. Int. Symp. on Chinese Spoken Language Processing (ISCSLP'2012), Hong Kong, 2012.