Combining stochastic average gradient and Hessian-free optimization for sequence training of deep neural networks

2013 IEEE Workshop on Automatic Speech Recognition and Understanding, ASRU 2013 - Proceedings

Volumn , Issue , 2013, Pages 321-325

  Dognin, Pierre ^a   Goel, Vaibhava ^a  

^a IBM T J WATSON RESEARCH CENTER   (United States)

Author keywords

Deep Neural Network; Hessian free Optimization; Sequence Training; Stochastic Average Gradient; Stochastic Training

Indexed keywords

AUTOMATIC SPEECH RECOGNITION; AVERAGE GRADIENT; DEEP NEURAL NETWORKS; GRADIENT INFORMATIONS; MINIMUM PHONE ERROR; MODEL PARAMETERS; STOCHASTIC APPROACH; STOCHASTIC GRADIENT DESCENT;

NEURAL NETWORKS; OPTIMIZATION; SPEECH RECOGNITION;

STOCHASTIC SYSTEMS;

EID: 84893690218     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/ASRU.2013.6707750     Document Type: Conference Paper

References (7)

1. Brian Kingsbury, "lattice-based optimization of sequence classification criteria for neural-network acoustic modeling, " in ICASSP, April 2009, pp. 3761-3764.
2. Daniel Povey and Philip C.Woodland, "Minimum phone error and i-smoothing for improved discriminative training, " in ICASSP, May 2002, pp. 105-108.
3. Hang Su, Gang Li, Dong Yu, and Frank Seide, "error back propagation for sequence training of contextdependent deep networks for conversational speech transcription, " in ICASSP, May 2013, pp. 6664-6668.
4. Nicolas Le Roux, Mark Schmidt, and Francis Bach, "A stochastic gradient method with an exponential convergence rate for finite training sets, " in Advances in Neural Information Processing Systems 25, pp. 2672-2680. NIPS, 2012.
5. James Martens, "Deep learning via hessian-free optimization, " in Proceedings of the 27th International Conference on Machine Learning (ICML-10), Johannes Fürnkranz and Thorsten Joachims, Eds., Haifa, Israel, June 2010, pp. 735-742, Omnipress.
6. Ryan Kiros, "Training neural networks with stochastic hessian-free optimization, " in International Conference on Learning Representations, May 2013.
7. Léon Bottou and Yann LeCun, "Large scale online learning, " in Advances in Neural Information Processing Systems 16, Sebastian Thrun, Lawrence Saul, and Bernhard Schölkopf, Eds. MIT Press, Cambridge, MA, 2004.