Transfer learning for speech and language processing

2015 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2015

Volumn , Issue , 2016, Pages 1225-1237

  Wang, Dong ^a   Zheng, Thomas Fang ^b  

^a TSINGHUA UNIVERSITY   (China)

^b TSINGHUA NATIONAL LABORATORY FOR INFORMATION SCIENCE AND TECHNOLOGY   (China)

Author keywords

[No Author keywords available]

Indexed keywords

ABSTRACT DATA TYPES; BAYESIAN NETWORKS; INFORMATION SCIENCE; MODEL STRUCTURES;

ACOUSTIC MODEL; BAYESIAN MODEL; CROSS-LINGUAL; DATA DISTRIBUTION; LANGUAGE PROCESSING; MODEL ADAPTATION; MULTITASK LEARNING; TRANSFER LEARNING;

SPEECH RECOGNITION;

EID: 84986198297     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/APSIPA.2015.7415532     Document Type: Conference Paper

References (150)

1. J. H. Martin and D. Jurafsky, "Speech and language processing," International Edition, 2000.
2. J. Benesty, Springer handbook of speech processing. Springer Science & Business Media, 2008.
3. L. Deng and X. Li, "Machine learning paradigms for speech recognition: An overview," Audio, Speech, and Language Processing, IEEE Transactions on, vol. 21, no. 5, pp. 1060-1089, 2013.
4. S. J. Pan and Q. Yang, "A survey on transfer learning," Knowledge and Data Engineering, IEEE Transactions on, vol. 22, no. 10, pp. 1345-1359, 2010.
5. M. E. Taylor and P. Stone, "Transfer learning for reinforcement learning domains: A survey," The Journal of Machine Learning Research, vol. 10, pp. 1633-1685, 2009.
6. Y. Bengio, "Deep learning of representations for unsupervised and transfer learning," in ICML Unsupervised and Transfer Learning, 2012.
7. J. Lu, V. Behbood, P. Hao, H. Zuo, S. Xue, and G. Zhang, "Transfer learning using computational intelligence: A survey," Knowledge-Based Systems, vol. 80, pp. 14-23, 2015.
8. S. Thrun and L. Pratt, Learning to learn. Springer Science & Business Media, 2012.
9. "NIPS 95 workshop on learning to learn: Knowledge consolidation and transfer in inductive systems," 1995. [Online]. Available: http://socrates.acadiau.ca/courses/comp/dsilver/ NIPS95 LTL/transfer.workshop.1995.html
10. G. E. Hinton, O. Vinyals, and J. Dean, "Distilling the knowledge in a neural network," in NIPS 2014 Deep Learning Workshop, 2014.
11. R. Caruana, "Multitask learning," Machine learning, vol. 28, no. 1, pp. 41-75, 1997.
12. J.-L. Gauvain and C.-H. Lee, "Maximum a posteriori estimation for multivariate Gaussian mixture observations of Markov chains," IEEE Transactions on Speech and audio processing, vol. 2, no. 2, pp. 291-298, 1994.
13. C. J. Leggetter and P. Woodland, "Maximum likelihood linear regression for speaker adaptation of continuous density hidden Markov models," Computer Speech & Language, vol. 9, no. 2, pp. 171-185, 1995.
14. P. E. Utgoff, "Incremental induction of decision trees," Machine learning, vol. 4, no. 2, pp. 161-186, 1989.
15. A. Blum and T. Mitchell, "Combining labeled and unlabeled data with co-training," in Proceedings of the eleventh annual conference on Computational learning theory. ACM, 1998, pp. 92-100.
16. C. Wang and S. Mahadevan, "Heterogeneous domain adaptation using manifold alignment," in IJCAI Proceedings-International Joint Conference on Artificial Intelligence, vol. 22, no. 1, 2011, p. 1541.
17. Y. Zhu, Y. Chen, Z. Lu, S. J. Pan, G.-R. Xue, Y. Yu, and Q. Yang, "Heterogeneous transfer learning for image classification." in AAAI, 2011.
18. H.-Y. Wang and Q. Yang, "Transfer learning by structural analogy," in AAAI. Citeseer, 2011.
19. O. Arandjelovic and R. Cipolla, "Incremental learning of temporallycoherent Gaussian mixture models," Society of Manufacturing Engineers (SME) Technical Papers, pp. 1-1, 2006.
20. A. Declercq and J. H. Piater, "Online learning of Gaussian mixture models-a two-level approach." in VISAPP (1), 2008, pp. 605-611.
21. X. Zhu, "Semi-supervised learning literature survey," Computer Sciences TRP 1530, University of Wisconsin C Madison, 2005.
22. S. J. Pan, I. W. Tsang, J. T. Kwok, and Q. Yang, "Domain adaptation via transfer component analysis," Neural Networks, IEEE Transactions on, vol. 22, no. 2, pp. 199-210, 2011.
23. R. Raina, A. Battle, H. Lee, B. Packer, and A. Y. Ng, "Self-taught learning: transfer learning from unlabeled data," in Proceedings of the 24th international conference on Machine learning. ACM, 2007, pp. 759-766.
24. P. Prettenhofer and B. Stein, "Cross-lingual adaptation using structural correspondence learning," ACM Transactions on Intelligent Systems and Technology (TIST), vol. 3, no. 1, p. 13, 2011.
25. W. Dai, Y. Chen, G.-R. Xue, Q. Yang, and Y. Yu, "Translated learning: Transfer learning across different feature spaces," in Advances in neural information processing systems, 2008, pp. 353-360.
26. B. Kulis, K. Saenko, and T. Darrell, "What you saw is not what you get: Domain adaptation using asymmetric kernel transforms," in Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on. IEEE, 2011, pp. 1785-1792.
27. B. Wei and C. J. Pal, "Heterogeneous transfer learning with RBMs." in AAAI, 2011.
28. X. Shi, Q. Liu, W. Fan, P. S. Yu, and R. Zhu, "Transfer learning on heterogenous feature spaces via spectral transformation," in Data Mining (ICDM), 2010 IEEE 10th International Conference on. IEEE, 2010, pp. 1049-1054.
29. L. Duan, D. Xu, and I. Tsang, "Learning with augmented features for heterogeneous domain adaptation," arXiv preprint arXiv:1206.4660, 2012.
30. J. T. Zhou, S. J. Pan, I. W. Tsang, and Y. Yan, "Hybrid heterogeneous transfer learning through deep learning," in Twenty-Eighth AAAI Conference on Artificial Intelligence, 2014.
31. D. Gentner, "Structure-mapping: A theoretical framework for analogy," Cognitive science, vol. 7, no. 2, pp. 155-170, 1983.
32. D. Gentner and K. J. Holyoak, "Reasoning and learning by analogy: Introduction." American Psychologist, vol. 52, no. 1, p. 32, 1997.
33. J. Blitzer, R. McDonald, and F. Pereira, "Domain adaptation with structural correspondence learning," in Proceedings of the 2006 conference on empirical methods in natural language processing. Association for Computational Linguistics, 2006, pp. 120-128.
34. J. G. Carbonell, Learning by analogy: Formulating and generalizing plans from past experience. Springer, 1983.
35. D. Wang, C. Liu, Z. Tang, Z. Zhang, and M. Zhao, "Recurrent neural network training with dark knowledge transfer," arXiv preprint arXiv:1505.04630, 2015.
36. Z. Tang, D. Wang, Y. Pan, and Z. Zhang, "Knowledge transfer pretraining," arXiv preprint arXiv:1506.02256, 2015.
37. J. Baxter, "A model of inductive bias learning," J. Artif. Intell. Res.(JAIR), vol. 12, pp. 149-198, 2000.
38. J. Guinney, Q. Wu, and S. Mukherjee, "Estimating variable structure and dependence in multitask learning via gradients," Machine Learning, vol. 83, no. 3, pp. 265-287, 2011.
39. B. Romera-Paredes, A. Argyriou, N. Berthouze, and M. Pontil, "Exploiting unrelated tasks in multi-task learning," in International Conference on Artificial Intelligence and Statistics, 2012, pp. 951-959.
40. G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath et al., "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82-97, 2012.
41. L. Deng and D. Yu, "Deep learning: Methods and applications," Foundations and Trends in Signal Processing, vol. 7, no. 3-4, pp. 197-387, 2013. [Online]. Available: http://dx.doi.org/10.1561/2000000039
42. X. He, J. Gao, and L. Deng, "Deep learning for natural language processing and related applications (Tutorial at ICASSP)," in IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2014.
43. J. Hirschberg and C. D. Manning, "Advances in natural language processing," Science, vol. 349, no. 6245, pp. 261-266, 2015.
44. G. E. Hinton, S. Osindero, and Y.-W. Teh, "A fast learning algorithm for deep belief nets," Neural computation, vol. 18, no. 7, pp. 1527-1554, 2006.
45. R. Salakhutdinov and G. E. Hinton, "Deep boltzmann machines," in International Conference on Artificial Intelligence and Statistics, 2009, pp. 448-455.
46. Y. Bengio, P. Lamblin, D. Popovici, H. Larochelle et al., "Greedy layer-wise training of deep networks," Advances in neural information processing systems, vol. 19, p. 153, 2007.
47. P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.-A. Manzagol, "Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion," The Journal of Machine Learning Research, vol. 11, pp. 3371-3408, 2010.
48. G. E. Dahl, D. Yu, L. Deng, and A. Acero, "Context-dependent pretrained deep neural networks for large-vocabulary speech recognition," Audio, Speech, and Language Processing, IEEE Transactions on, vol. 20, no. 1, pp. 30-42, 2012.
49. A. Graves, A.-R. Mohamed, and G. Hinton, "Speech recognition with deep recurrent neural networks," in Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2013, pp. 6645-6649.
50. Y. Bengio and O. Delalleau, "On the expressive power of deep architectures," in Algorithmic Learning Theory. Springer, 2011, pp. 18-36.
51. R. Collobert and J. Weston, "A unified architecture for natural language processing: Deep neural networks with multitask learning," in Proceedings of the 25th international conference on Machine learning. ACM, 2008, pp. 160-167.
52. L. Deng, J. Li, J.-T. Huang, K. Yao, D. Yu, F. Seide, M. Seltzer, G. Zweig, X. He, J. Williams et al., "Recent advances in deep learning for speech research at Microsoft," in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 8604-8608.
53. S. M. Gutstein, Transfer learning techniques for deep neural nets. The University of Texas at El Paso, 2010.
54. J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng, "Multimodal deep learning," in Proceedings of the 28th international conference on machine learning (ICML-11), 2011, pp. 689-696.
55. Y. Bengio, I. J. Goodfellow, and A. Courville, Deep Learning, 2015, book in preparation for MIT Press. [Online]. Available: http://www.iro.umontreal.ca/?bengioy/dlbook
56. G. E. Hinton and R. R. Salakhutdinov, "Reducing the dimensionality of data with neural networks," Science, vol. 313, no. 5786, pp. 504-507, 2006.
57. X. Glorot, A. Bordes, and Y. Bengio, "Domain adaptation for largescale sentiment classification: A deep learning approach," in Proceedings of the 28th International Conference on Machine Learning (ICML-11), 2011, pp. 513-520.
58. M. Oquab, L. Bottou, I. Laptev, and J. Sivic, "Learning and transferring mid-level image representations using convolutional neural networks," in Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on. IEEE, 2014, pp. 1717-1724.
59. W. Zhang, R. Li, T. Zeng, Q. Sun, S. Kumar, J. Ye, and S. Ji, "Deep model based transfer and multi-task learning for biological image analysis," in Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2015, pp. 1475-1484.
60. L. Fei-Fei, R. Fergus, and P. Perona, "One-shot learning of object categories," Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 28, no. 4, pp. 594-611, 2006.
61. H. Larochelle, D. Erhan, and Y. Bengio, "Zero-data learning of new tasks." in AAAI, vol. 1, no. 2, 2008, p. 3.
62. R. Socher, M. Ganjoo, C. D. Manning, and A. Ng, "Zero-shot learning through cross-modal transfer," in Advances in neural information processing systems, 2013, pp. 935-943.
63. T. Schultz and A. Waibel, "Language-independent and languageadaptive acoustic modeling for speech recognition," Speech Communication, vol. 35, no. 1, pp. 31-51, 2001.
64. N. T. Vu, F. Kraus, and T. Schultz, "Cross-language bootstrapping based on completely unsupervised training using multilingual A-stabil," in Acoustics, Speech and Signal Processing (ICASSP), 2011 IEEE International Conference on. IEEE, 2011, pp. 5000-5003.
65. P. Swietojanski, A. Ghoshal, and S. Renals, "Unsupervised crosslingual knowledge transfer in DNN-based LVCSR," in Spoken Language Technology Workshop (SLT), 2012 IEEE. IEEE, 2012, pp. 246-251.
66. J.-T. Huang, J. Li, D. Yu, L. Deng, and Y. Gong, "Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers," in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 7304-7308.
67. G. Heigold, V. Vanhoucke, A. Senior, P. Nguyen, M. Ranzato, M. Devin, and J. Dean, "Multilingual acoustic models using distributed deep neural networks," in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 8619-8623.
68. A. Ghoshal, P. Swietojanski, and S. Renals, "Multilingual training of deep neural networks," in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 7319-7323.
69. K. Vesely, M. Karafiát, F. Grezl, M. Janda, and E. Egorova, "The language-independent bottleneck features," in Spoken Language Technology Workshop (SLT), 2012 IEEE. IEEE, 2012, pp. 336-341.
70. S. Thomas, M. L. Seltzer, K. Church, and H. Hermansky, "Deep neural network features and semi-supervised training for low resource speech recognition," in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 6704-6708.
71. Z. Tuske, J. Pinto, D. Willett, and R. Schluter, "Investigation on cross-and multilingual mlp features under matched and mismatched acoustical conditions," in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 7349-7353.
72. K. M. Knill, M. J. Gales, A. Ragni, and S. P. Rath, "Language independent and unsupervised acoustic models for speech recognition and keyword spotting," in Proc. Interspeech14, 2014.
73. P. Bell, P. Swietojanski, and S. Renals, "Multi-level adaptive networks in tandem and hybrid ASR systems," in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 6975-6979.
74. J. Gehring, Q. B. Nguyen, F. Metze, and A. Waibel, "DNN acoustic modeling with modular multi-lingual feature extraction networks," in Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on. IEEE, 2013, pp. 344-349.
75. D. Chen, B. Mak, C.-C. Leung, and S. Sivadas, "Joint acoustic modeling of triphones and trigraphemes by multi-task learning deep neural networks for low-resource speech recognition," in Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, 2014, pp. 5592-5596.
76. X. Z. Zhiyuan Tang, "Speech recognition with pronunciation vecotrs," CSLT, Tsinghua University, 2015. [Online]. Available: http://cslt.riit.tsinghua.edu.cn/publications.php?Publication-trp
77. N. Dehak, P. Kenny, R. Dehak, P. Dumouchel, and P. Ouellet, "Frontend factor analysis for speaker verification," Audio, Speech, and Language Processing, IEEE Transactions on, vol. 19, no. 4, pp. 788-798, 2011.
78. V. Ehsan, L. Xin, M. Erik, L. M. Ignacio, and G.-D. Javier, "Deep neural networks for small footprint text-dependent speaker verification," IEEE International Conference on Acoustic, Speech and Signal Processing (ICASSP), vol. 28, no. 4, pp. 357-366, 2014.
79. Y. Xu, J. Du, L.-R. Dai, and C.-H. Lee, "Cross-language transfer learning for deep neural network based speech enhancement," in Chinese Spoken Language Processing (ISCSLP), 2014 9th International Symposium on. IEEE, 2014, pp. 336-340.
80. M. Zhao, D. Wang, Z. Zhang, and X. Zhang, "Music removal by convolutional denoising autoencoder in speech recognition," in Interspeech 2015, 2015.
81. M. L. Seltzer and A. Acero, "Separating speaker and environmental variability using factored transforms." in INTERSPEECH, 2011, pp. 1097-1100.
82. D. Povey and K. Yao, "A basis representation of constrained MLLR transforms for robust adaptation," Computer Speech & Language, vol. 26, no. 1, pp. 35-51, 2012.
83. Y. Miao, F. Metze, and A. Waibel, "Learning discriminative basis coefficients for eigenspace MLLR unsupervised adaptation," in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 7927-7931.
84. K. Shinoda, "Speaker adaptation techniques for automatic speech recognition," Proc. APSIPA ASC 2011, 2011.
85. O. Abdel-Hamid and H. Jiang, "Fast speaker adaptation of hybrid NN/HMM model for speech recognition based on discriminative learning of speaker code," in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 7942-7946.
86. -, "Rapid and effective speaker adaptation of convolutional neural network based models for speech recognition." in INTERSPEECH, 2013, pp. 1248-1252.
87. S. Xue, O. Abdel-Hamid, H. Jiang, and L. Dai, "Direct adaptation of hybrid DNN/HMM model for fast speaker adaptation in LVCSR based on speaker code," in Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, 2014, pp. 6339-6343.
88. G. Saon, H. Soltau, D. Nahamoo, and M. Picheny, "Speaker adaptation of neural network acoustic models using i-vectors," in Automatic Speech Recognition and Understanding (ASRU), 2013 IEEE Workshop on. IEEE, 2013, pp. 55-59.
89. P. Karanasou, Y. Wang, M. J. Gales, and P. C. Woodland, "Adaptation of deep neural network acoustic models using factorised i-vectors," in Proc Interspeech, 2014.
90. A. Senior and I. Lopez-Moreno, "Improving DNN speaker independence with i-vector inputs," in Proc. ICASSP, 2014.
91. V. Gupta, P. Kenny, P. Ouellet, and T. Stafylakis, "I-vector-based speaker adaptation of deep neural networks for french broadcast audio transcription," in Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, 2014, pp. 6334-6338.
92. M. Rouvier and B. Favre, "Speaker adaptation of DNN-based ASR with i-vectors: Does it actually adapt models to speakers" in Fifteenth Annual Conference of the International Speech Communication Association, 2014.
93. J. Neto, L. Almeida, M. Hochberg, C. Martins, L. Nunes, S. Renals, and T. Robinson, "Speaker-adaptation for hybrid HMM-ANN continuous speech recognition system," in Proc. EUROSPEECH'95. International Speech Communication Association, 1995.
94. K. Yao, D. Yu, F. Seide, H. Su, L. Deng, and Y. Gong, "Adaptation of context-dependent deep neural networks for automatic speech recognition," in Spoken Language Technology Workshop (SLT), 2012 IEEE. IEEE, 2012, pp. 366-369.
95. R. Gemello, F. Mana, S. Scanzio, P. Laface, and R. De Mori, "Linear hidden transformations for adaptation of hybrid ANN/HMM models," Speech Communication, vol. 49, no. 10, pp. 827-835, 2007.
96. S. M. Siniscalchi, J. Li, and C.-H. Lee, "Hermitian polynomial for speaker adaptation of connectionist speech recognition systems," Audio, Speech, and Language Processing, IEEE Transactions on, vol. 21, no. 10, pp. 2152-2161, 2013.
97. P. Swietojanski and S. Renals, "Learning hidden unit contributions for unsupervised speaker adaptation of neural network acoustic models," in Spoken Language Technology Workshop (SLT), 2014 IEEE. IEEE, 2014, pp. 171-176.
98. B. Li and K. C. Sim, "Comparison of discriminative input and output transformations for speaker adaptation in the hybrid NN/HMM systems," in Interspeech'10, 2010.
99. H. Liao, "Speaker adaptation of context dependent deep neural networks," in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 7947-7951.
100. J. Xue, J. Li, D. Yu, M. Seltzer, and Y. Gong, "Singular value decomposition based low-footprint speaker adaptation and personalization for deep neural network," in Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, 2014, pp. 6359-6363.
101. S. Xue, H. Jiang, and L. Dai, "Speaker adaptation of hybrid NN/HMM model for speech recognition based on singular value decomposition," in Chinese Spoken Language Processing (ISCSLP), 2014 9th International Symposium on. IEEE, 2014, pp. 1-5.
102. D. Yu, K. Yao, H. Su, G. Li, and F. Seide, "Kl-divergence regularized deep neural network adaptation for improved large vocabulary speech recognition," in Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on. IEEE, 2013, pp. 7893-7897.
103. Y. Tang, A. Mohan, R. C. Rose, and C. Ma, "Deep neural network trained with speaker representation for speaker normalization," in Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, 2014, pp. 6329-6333.
104. Y. Miao, H. Zhang, and F. Metze, "Towards speaker adaptive training of deep neural network acoustic models," in Interspeech'14, 2014.
105. Y. Miao and F. Metze, "On speaker adaptation of long short-term memory recurrent neural networks," in Sixteenth Annual Conference of the International Speech Communication Association (INTERSPEECH)(To Appear). ISCA, 2015.
106. M. Tamura, T. Masuko, K. Tokuda, and T. Kobayashi, "Adaptation of pitch and spectrum for HMM-based speech synthesis using MLLR," in Acoustics, Speech, and Signal Processing, 2001. Proceedings.( ICASSP'01). 2001 IEEE International Conference on, vol. 2. IEEE, 2001, pp. 805-808.
107. Y.-J. Wu, Y. Nankaku, and K. Tokuda, "State mapping based method for cross-lingual speaker adaptation in HMM-based speech synthesis." in Interspeech, 2009, pp. 528-531.
108. J. Yamagishi and T. Kobayashi, "Average-voice-based speech synthesis using HSMM-based speaker adaptation and adaptive training," IEICE TRANSACTIONS on Information and Systems, vol. 90, no. 2, pp. 533-543, 2007.
109. J. Yamagishi, T. Kobayashi, Y. Nakano, K. Ogata, and J. Isogai, "Analysis of speaker adaptation algorithms for HMM-based speech synthesis and a constrained SMAPLR adaptation algorithm," Audio, Speech, and Language Processing, IEEE Transactions on, vol. 17, no. 1, pp. 66-83, 2009.
110. H. Liang, J. Dines, and L. Saheer, "A comparison of supervised and unsupervised cross-lingual speaker adaptation approaches for HMMbased speech synthesis," in Acoustics Speech and Signal Processing (ICASSP), 2010 IEEE International Conference on. IEEE, 2010, pp. 4598-4601.
111. M. Gibson and W. Byrne, "Unsupervised intralingual and cross-lingual speaker adaptation for HMM-based speech synthesis using two-pass decision tree construction," Audio, Speech, and Language Processing, IEEE Transactions on, vol. 19, no. 4, pp. 895-904, May 2011.
112. Z.-H. Ling, L. Deng, and D. Yu, "Modeling spectral envelopes using restricted boltzmann machines and deep belief networks for statistical parametric speech synthesis," Audio, Speech, and Language Processing, IEEE Transactions on, vol. 21, no. 10, pp. 2129-2139, 2013.
113. H. Zen and A. Senior, "Deep mixture density networks for acoustic modeling in statistical parametric speech synthesis," in Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, 2014, pp. 3844-3848.
114. K. Hashimoto, K. Oura, Y. Nankaku, and K. Tokuda, "The effect of neural networks in statistical parametric speech synthesis," in Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on. IEEE, 2015, pp. 4455-4459.
115. B. Potard, P. Motlicek, and D. Imseng, "Preliminary work on speaker adaptation for DNN-based speech synthesis," Idiap, Tech. Rep., 2015.
116. Z. Wu, P. Swietojanski, C. Veaux, S. Renals, and S. King, "A study of speaker adaptation for DNN-based speech synthesis," in Interspeech 2015, 2015.
117. J. Ba and R. Caruana, "Do deep nets really need to be deep" in Advances in Neural Information Processing Systems, 2014, pp. 2654-2662.
118. J. Li, R. Zhao, J.-T. Huang, and Y. Gong, "Learning small-size DNN with output-distribution-based criteria," in Proceedings of the Annual Conference of International Speech Communication Association (INTERSPEECH), September 2014. [Online]. Available: http://research.microsoft.com/apps/pubs/default.aspx?id=230080
119. W. Chan, N. R. Ke, and I. Lane, "Transferring knowledge from a RNN to a DNN," arXiv preprint arXiv:1504.01483, 2015.
120. A. Romero, N. Ballas, S. E. Kahou, A. Chassang, C. Gatta, and Y. Bengio, "Fitnets: Hints for thin deep nets," arXiv preprint arXiv:1412.6550, 2014.
121. M. Long and J. Wang, "Learning transferable features with deep adaptation networks," arXiv preprint arXiv:1502.02791, 2015.
122. Y. Lu, "Unsupervised learning of neural network outputs," arXiv preprint arXiv:1506.00990, 2015.
123. G. Durrett, A. Pauls, and D. Klein, "Syntactic transfer using a bilingual lexicon," in Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Association for Computational Linguistics, 2012, pp. 1-11.
124. L. Shi, R. Mihalcea, and M. Tian, "Cross language text classification by model translation and semi-supervised learning," in Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2010, pp. 1057-1067.
125. X. Ma, X. Wang, and D. Wang, "Recognize foreign low-frequency words with similar pairs," in Interspeech 2015, 2015.
126. P. Koehn, Statistical machine translation. Cambridge University Press, 2009.
127. W. De Smet, J. Tang, and M.-F. Moens, "Knowledge transfer across multilingual corpora via latent topics," in Advances in Knowledge Discovery and Data Mining. Springer, 2011, pp. 549-560.
128. O. Täckström, R. McDonald, and J. Uszkoreit, "Cross-lingual word clusters for direct transfer of linguistic structure," in Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2012, pp. 477-487.
129. O. Täckström, "Nudging the envelope of direct transfer methods for multilingual named entity recognition," in Proceedings of the NAACLHLT Workshop on the Induction of Linguistic Structure. Association for Computational Linguistics, 2012, pp. 55-63.
130. Y. Bengio, H. Schwenk, J.-S. Senécal, F. Morin, and J.-L. Gauvain, "Neural probabilistic language models," in Innovations in Machine Learning. Springer, 2006, pp. 137-186.
131. A. Mnih and G. E. Hinton, "A scalable hierarchical distributed language model," in NIPS, 2008, pp. 1081-1088.
132. J. Turian, L. Ratinov, and Y. Bengio, "Word representations: a simple and general method for semi-supervised learning," in Proceedings of the 48th annual meeting of the association for computational linguistics. Association for Computational Linguistics, 2010, pp. 384-394.
133. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa, "Natural language processing (almost) from scratch," The Journal of Machine Learning Research, vol. 12, pp. 2493-2537, 2011.
134. T. Mikolov, K. Chen, G. Corrado, and J. Dean, "Efficient estimation of word representations in vector space," arXiv preprint arXiv:1301.3781, 2013.
135. T. Mikolov, Q. V. Le, and I. Sutskever, "Exploiting similarities among languages for machine translation," arXiv preprint arXiv:1309.4168, 2013.
136. C. Xing, D. Wang, C. Liu, and Y. Lin, "Normalized word embedding and orthogonal transform for bilingual word translation," in NAACL'15, 2015.
137. M. Faruqui and C. Dyer, "Improving vector space word representations using multilingual correlation," in EACL'14. Association for Computational Linguistics, 2014.
138. A. Klementiev, I. Titov, and B. Bhattarai, "Inducing crosslingual distributed representations of words," in COLING'12. Citeseer, 2012.
139. K. M. Hermann and P. Blunsom, "Multilingual models for compositional distributed semantics," arXiv preprint arXiv:1404.4641, 2014.
140. S. Gouws, Y. Bengio, and G. Corrado, "Bilbowa: Fast bilingual distributed representations without word alignments," arXiv preprint arXiv:1410.2455, 2014.
141. E. Bruni, G. Boleda, M. Baroni, and N.-K. Tran, "Distributional semantics in technicolor," in Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1. Association for Computational Linguistics, 2012, pp. 136-145.
142. C. W. Leong and R. Mihalcea, "Going beyond text: A hybrid imagetext approach for measuring word relatedness." in IJCNLP, 2011, pp. 1403-1407.
143. R. Socher and L. Fei-Fei, "Connecting modalities: Semi-supervised segmentation and annotation of images using unaligned text corpora," in Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on. IEEE, 2010, pp. 966-973.
144. A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, T. Mikolov et al., "Devise: A deep visual-semantic embedding model," in Advances in Neural Information Processing Systems, 2013, pp. 2121-2129.
145. R. Kiros, R. Salakhutdinov, and R. Zemel, "Multimodal neural language models," in Proceedings of the 31st International Conference on Machine Learning (ICML-14), 2014, pp. 595-603.
146. R. Socher, A. Karpathy, Q. V. Le, C. D. Manning, and A. Y. Ng, "Grounded compositional semantics for finding and describing images with sentences," Transactions of the Association for Computational Linguistics, vol. 2, pp. 207-218, 2014.
147. N. Srivastava and R. R. Salakhutdinov, "Multimodal learning with deep boltzmann machines," in Advances in neural information processing systems, 2012, pp. 2222-2230.
148. L. Mou, G. Li, Y. Xu, L. Zhang, and Z. Jin, "Distilling word embeddings: An encoding approach," arXiv preprint arXiv:1506.04488, 2015.
149. D. Zhang, T. Luo, D. Wang, and R. Liu, "Learning from LDA using deep neural networks," arXiv preprint arXiv:1508.01011, 2015.
150. M. Long, J. Wang, G. Ding, D. Shen, and Q. Yang, "Transfer learning with graph co-regularization," Knowledge and Data Engineering, IEEE Transactions on, vol. 26, no. 7, pp. 1805-1818, 2014.