Composition of Deep and Spiking Neural Networks for Very Low Bit Rate Speech Coding

IEEE/ACM Transactions on Audio Speech and Language Processing

Volumn 24, Issue 12, 2016, Pages 2301-2312

  Cernak, Milos ^a   Lazaridis, Alexandros ^a   Asaei, Afsaneh ^a   Garner, Philip N ^a  

^a IDIAP RESEARCH INSTITUTE   (Switzerland)

Author keywords

continuous F0 coding; deep neural networks; spiking neural networks; Very low bit rate speech coding

Indexed keywords

CODES (SYMBOLS); CODING ERRORS; CONTINUOUS SPEECH RECOGNITION; DEEP NEURAL NETWORKS; HIDDEN MARKOV MODELS; IMAGE CODING; MARKOV PROCESSES; NETWORK CODING; NEURAL NETWORKS; SIGNAL ENCODING; SPEECH; SPEECH CODING;

ANALYSIS AND SYNTHESIS; CONTINUOUS F0 CODING; FUNDAMENTAL FREQUENCIES; NEURAL NETWORKS (NNS); PHONOLOGICAL FEATURES; SPEECH CODING SYSTEM; SPIKING NEURAL NETWORKS; VERY LOW BIT RATE;

SPEECH RECOGNITION;

EID: 85027056450     PISSN: 23299290     EISSN: None     Source Type: Journal    
DOI: 10.1109/TASLP.2016.2604566     Document Type: Article

References (78)

1. S. Dimolitsas, C. Ravishankar, and G. Schroder, "Current objectives in 4-kb/s wireline-quality speech coding standardization," IEEE Signal Process. Lett., vol. 1, no. 11, pp. 157-159, Nov. 1994.
2. K.-S. Lee and R. Cox, "A very low bit rate speech coder based on a recognition/synthesis paradigm," IEEE Trans. Audio, Speech, Lang. Process., vol. 9, no. 5, pp. 482-491, Jul. 2001.
3. G. V. Baudoin and F. El Chami, "Corpus based very low bit rate speech coding," in Proc. 2003 IEEE Int. Conf. Acoust. Speech Signal Process., vol. 1. Piscataway, NJ, USA: IEEE, Apr. 2003, pp. I-792-I-795.
4. J. Picone and G. R. Doddington, "A phonetic vocoder," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., vol. 1. Piscataway, NJ, USA: IEEE, May 1989, pp. 580-583.
5. K. Tokuda, T. Masuko, J. Hiroi, T. Kobayashi, and T. Kitamura, "A very low bit rate speech coder using HMM-based speech recognition/synthesis techniques," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., vol. 2. Piscataway, NJ, USA: IEEE, May 1998, vol. 2, pp. 609-612.
6. A. McCree, K. Brady, and T. F. Quatieri, "Multisensor very lowbit rate speech coding using segment quantization," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., Mar. 2008, pp. 3997-4000.
7. M. Cernak, P. N. Garner, A. Lazaridis, P. Motlicek, and X. Na, "Incremental syllable-context phonetic vocoding," IEEE/ACMTrans. Audio, Speech, Lang. Process., vol. 23, no. 6, pp. 1019-1030, Jun. 2015.
8. G. Greenberg and S. Chang, "Linguistic dissection of switchboard-corpus automatic speech recognition systems," in Proc. ISCA Autom. Speech Recognit.: Challenges New Millennium, 2000, pp. 195-202.
9. M. Cernak, S. Benus, and A. Lazaridis, "Speech vocoding for laboratory phonology," 2016. [Online]. Available: Http://arxiv.org/abs/1601.05991
10. J. Harris, English Sound Structure, 1st ed. Hoboken, NJ, USA: Wiley, Dec. 1994.
11. J. Harris and G. Lindsey, The Elements of Phonological Representation. Harlow, Essex, U.K.: Longman, 1995, pp. 34-79.
12. N. Chomsky and M. Halle, The Sound Pattern of English. NewYork, NY, USA: Harper & Row, 1968.
13. D. Yu, S. Siniscalchi, L. Deng, and C.-H. Lee, "Boosting attribute and phone estimation accuracies with deep neural networks for detectionbased speech recognition," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process. Mar. 2012, pp. 4169-4172.
14. S. M. Siniscalchi, D.-C. Lyu, T. Svendsen, and C.-H. Lee, "Experiments on cross-language attribute detection and phone recognition with minimal target-specific training data," IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 3, pp. 875-887, Mar. 2012.
15. A. Asaei, M. Cernak, and H. Bourlard, "On compressibility of neural network phonological features for low bit rate speech coding," in Proc. Interspeech, Sep. 2015, pp. 418-422.
16. Y. Bengio, "Learning deep architectures for AI," Found. Trends Mach. Learn., vol. 2, no. 1, pp. 1-127, Jan. 2009.
17. H. Bourlard and N. Morgan, Connectionist Speech Recognition: A Hybrid Approach. Boston, MA, USA: Kluwer, 1994.
18. A. Mohamed, G. E. Dahl, and G. E. Hinton, "Deep belief networks for phone recognition," in Proc. NIPS'22 Workshop Deep Learn. Speech Recognit., 2009.
19. G. Hinton, "Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups," IEEE Signal Process. Mag., vol. 29, no. 6, pp. 82-97, Nov. 2012.
20. G. Dahl, D. Yu, L. Deng, and A. Acero, "Context-dependent pretrained deep neural networks for large vocabulary speech recognition," IEEE Trans. Audio, Speech, Lang. Process., vol. 20, no. 1, pp. 30-42, Jan. 2012.
21. H. Zen, A. Senior, and M. Schuster, "Statistical parametric speech synthesis using deep neural networks," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2013, pp. 7962-7966.
22. S. Esmeir, S. Markovitch, and C. Sammut, "Anytime learning of decision trees," J. Mach. Learn. Res., vol. 8, pp. 891-933, 2007.
23. K. Yu, H. Zen, F. Mairesse, and S. Young, "Context adaptive training with factorized decision trees for HMM-based statistical parametric speech synthesis," Speech Commun., vol. 53, no. 6, pp. 914-923, Jul. 2011.
24. Y. Qian, Y. Fan, W. Hu, and F. Soong, "On the training aspects of deep neural network (DNN) for parametric TTS synthesis," in Proc. IEEE Int. Conf. Acoust., Speech Signal Process., 2014, pp. 3829-3833.
25. A. Lazaridis, B. Potard, and P. N. Garner, "DNN-based speech synthesis: Importance of input features and training data," in Proc. Int. Conf. Speech Comput., 2015, pp. 193-200.
26. B. S. Atal and J. R. Remde, "A new model of LPC excitation for producing natural-sounding speech at low bit rates," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., vol. 7. Piscataway, NJ, USA: IEEE, May 1982, pp. 614-617. [Online]. Available: Http://dx. doi.org/10.1109/icassp.1982.1171649
27. M. Schroeder and B. Atal, "Code-excited linear prediction (CELP): High-quality speech at very low bit rates," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process, vol. 10. Piscataway, NJ, USA: IEEE, Apr. 1985, pp. 937-940. [Online]. Available: Http://dx.doi.org/10.1109/icassp.1985.1168147
28. S. Morishima, H. Harashima, and Y. Katayama, "Speech coding based on a multi-layer neural network," in Proc. IEEE Int. Conf. Commun., Incl. Supercomm Tech. Sessions. Conf. Record., Apr. 1990, vol. 2, pp. 429-433. [Online]. Available: Http://dx.doi.org/10.1109/icc.1990. 117117
29. Y. Zhen, "Prediction in speech coding: The modification of the coding of LPC parameters and nonlinear estimation technique by using ANN," in Proc. 3rd Int. Conf. Signal Process., Oct. 1996, vol. 1, pp. 690-693. [Online]. Available: Http://dx.doi.org/10.1109/icsigp. 1996.567357
30. S. Hunt, "A nonlinear adaptive predictor for speech compression," in Proc. IEEE Int. Conf. Neural Netw., vol. 4. Piscataway, NJ, USA: IEEE, Jun. 1996, pp. 1998-2002. [Online]. Available: Http://dx.doi. org/10.1109/icnn.1996.549208
31. C. Chavy, B. Gas, and J. L. Zarader, "Discriminative coding with predictive neural networks," in Proc. 9th Int. Conf. Art. Neural Netw., vol. 1. Hertfordshire, U.K.: IET, 1999, pp. 216-220. [Online]. Available: Http://dx.doi.org/10.1049/cp:19991111
32. M. Faúndez-Zanuy, "Adaptive hybrid speech coding with a MLP/LPC structure," in Engineering Applications of Bio-Inspired Artificial Neural Networks: International Work-Conference on Artificial and Natural Neural Networks, IWANN'99 Alicante, Spain, June 2-4, 1999 Proceedings, Volume II. Berlin, Germany: Springer, 1999, pp. 814-823. [Online]. Available: Http://dx.doi.org/10.1007/BFb0100549
33. B.-K. Lee and J.-H. Chang, "Packet loss concealment based on deep neural networks for digital speech transmission," IEEE/ACMTrans. Audio, Speech, Lang. Process., vol. 24, no. 2, pp. 378-387, Feb. 2016. [Online]. Available: Http://dx.doi.org/10.1109/taslp.2015.2509780
34. M. Sheikhan, V. T. Vakili, and S. Garoucy, "Complexity reduction of LDCELP speech coding in prediction of gain using neural networks," World Appl. Sci. J., vol. 7, no. 7, pp. 38-44, 2009.
35. M. G. Easton and C. C. Goodyear, "A CELP codebook and search technique using a Hopfield net," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process, vol. 1. Piscataway, NJ, USA: IEEE, Apr. 1991, pp. 685-688 vol. 1. [Online]. Available: Http://dx.doi.org/10. 1109/icassp.1991.150432
36. L. Wu, M. Niranjan, and F. Fallside, "Fully vector-quantized neural network-based code-excited nonlinear predictive speech coding," IEEE Trans. Audio, Speech, Lang. Process., vol. 2, no. 4, pp. 482-489, Oct. 1994. [Online].Available: Http://dx.doi.org/10.1109/89. 326608
37. D.W. Griffin and J. S. Lim, "Multiband excitation vocoder," IEEE Trans. Audio, Speech, Lang. Process., vol. 36, no. 8, pp. 1223-1235, Aug. 1988. [Online]. Available: Http://dx.doi.org/10.1109/29.1651
38. H. Cui and H. Jiang, "A robust 800 bps MBE coder with VQ and MLP," in Proc. Int. Conf. Commun. Technol. Proc., vol. 2. Piscataway, NJ, USA: IEEE, Oct. 1998, pp. 4. [Online]. Available: Http://dx.doi.org/10.1109/icct.1998.741011
39. D. Wong, B.-H. Juang, and A. Gray, "An 800 bit/s vector quantization LPC vocoder," IEEE Trans. Acoust., Speech, Signal Process., vol. ASSP-30, no. 5, pp. 770-780, Oct. 1982. [Online]. Available: Http://dx.doi.org/10.1109/tassp.1982.1163960
40. S. Roucos, R. Schwartz, and J. Makhoul, "Segment quantization for very-low-rate speech coding," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process, vol. 7. Piscataway, NJ, USA: IEEE, May 1982, pp. 1565-1568. [Online]. Available: Http://dx.doi.org/10.1109/icassp.1982.1171472
41. S. Roucos, R. Schwartz, and J.Makhoul, "A segment vocoder at 150 b/s," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process, vol. 8. Piscataway, NJ, USA: IEEE, Apr. 1983, pp. 61-64. [Online]. Available: Http://dx.doi.org/10.1109/icassp.1983.1172241
42. D. Wong, B. Juang, and D. Cheng, "Very low data rate speech compression with LPC vector and matrix quantization," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process, vol. 8. Piscataway, NJ, USA: IEEE, Apr. 1983, pp. 65-68. [Online]. Available: Http://dx.doi.org/10.1109/icassp.1983.1172244
43. C. Tsao and R. Gray, "Matrix quantizer design for LPC speech using the generalized Llyod algorithm," IEEE Trans. Acoust., Speech, Signal Process., vol. ASSP-33, no. 3, pp. 537-545, Jun. 1985. [Online].Available: Http://dx.doi.org/10.1109/tassp.1985.1164584
44. Y. Shiraki and M. Honda, "LPC speech coding based on variablelength segment quantization," IEEE Trans. Audio, Speech, Lang. Process., vol. 36, no. 9, pp. 1437-1444, Sep. 1988. [Online]. Available: Http://dx.doi.org/10.1109/29.90372
45. J. Gibson, "Speech Compression," Information, vol. 7, no. 2, p. 32, Jun. 2016. [Online]. Available: Http://dx.doi.org/10.3390/info7020032
46. M. Cernak, B. Potard, and P. N. Garner, "Phonological vocoding using artificial neural networks," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., Apr. 2015, pp. 4844-4848.
47. R. E. Turner and M. Sahani, "Demodulation as probabilistic inference," IEEE Trans. Audio, Speech, Lang. Process., vol. 19, no. 8, pp. 2398-2411, Nov. 2011.
48. V. Leong, M. A. Stone, R. E. Turner, and U. Goswami, "A role for amplitude modulation phase relationships in speech rhythm perception." J. Acoust. Soc. Amer., vol. 136, no. 1, pp. 366-381, Jul. 2014.
49. P. Lakatos, A. S. Shah, K. H. Knuth, I. Ulbert, G. Karmos, and C. E. Schroeder, "An oscillatory hierarchy controlling neuronal excitability and stimulus processing in the auditory cortex." J. Neurophysiol., vol. 94, no. 3, pp. 1904-1911, Sep. 2005.
50. M. Cernak, A. Asaei, and H. Bourlard, On structured sparsity of phonological posteriors for linguistic parsing, 2016. [Online]. Available: Http://arxiv.org/abs/1601.05647
51. K. Yu, B. Thomson, S. Young, and T. Street, "From discontinuous to continuous F0 modelling In HMM-based speech synthesis," in Proc. 7th ISCA Speech Synthesis Workshop, 2010, pp. 94-99.
52. K. Yu and S. Young, "Continuous F0 modeling for HMM based statistical parametric speech synthesis," IEEE Trans. Audio, Speech, Lang. Process., vol. 19, no. 5, pp. 1071-1079, Jul. 2011. [Online]. Available: Http://dx.doi.org/10.1109/tasl.2010.2076805
53. K. U. Ogbureke, J. P. Cabral, and J. Carson-Berndsen, "Using noisy speech to study the robustness of a continuous F0 modelling method in HMM-based speech synthesis," in Proc. Speech Prosody, 2012, pp. 67-70.
54. A. Hyafil and M. Cernak, "Neuromorphic based oscillatory device for incremental syllable boundary detection," in Proc. Interspeech, Sep. 2015, pp. 1191-1195.
55. M. Cernak, X. Na, and P. N. Garner, "Syllable-based pitch encoding for low bit rate speech coding with recognition/synthesis architecture," in Proc. Interspeech, Aug. 2013, pp. 3449-3452.
56. M. Cernak and P. N. Garner, "PhonVoc: A phonetic and phonological vocoding toolkit," in Proc. Interspeech, 2016.
57. D. B. Paul and J. M. Baker, "The design for the wall street journal-based CSR corpus," in Proc. Workshop Speech Nat. Lang., 1992, pp. 357-362.
58. J. Garofolo, S. Lamel, L. F. Fisher, W. M. Fiscus, G. Jonathon , and D. S. Pallett, "DARPA TIMIT acoustic-phonetic continuous speech corpus CD-ROM.NIST speech disc 1-1.1.NASA STI/Recon technical report, 93, LDC93S1," Philadelphia, PA, USA: Linguistic Data Consortium, 1993.
59. A. Black, P. Taylor, and R. Caley, "The festival speech synthesis system," Human Commun. Res. Centre, Univ. of Edinburgh, Tech. Rep., 1997.
60. H. Zen, "The HMM-based speech synthesis system version 2.0," in Proc. 6th ISCA Speech Synthesis Workshop, 2007, pp. 131-136.
61. M. Wester, "Speaker adaptation and the evaluation of speaker similarity in the EMIME speech-to-speech translation project," in Proc. 7th ISCA Speech Synthesis Workshop, 2010, pp. 192-197.
62. G. E. Hinton, S. Osindero, and Y. W. Teh, "A fast learning algorithm for deep belief nets," Neural Comput., vol. 18, no. 7, pp. 1527-1554, Jul. 2006.
63. D. Povey et al., "The Kaldi speech recognition toolkit," in Proc. IEEE Workshop Autom. Speech Recognit. Understanding, Dec. 2011, IEEE Catalog No.: CFP11SRW-USB.
64. A. Hyafil, L. Fontolan, C. Kabdebon, B. Gutkin, A.-L. Giraud, and H. Brownell, "Speech encoding by coupled cortical theta and gamma oscillations," eLife, vol. 2015, no. 4, May 2015, Art. no. e06213. [Online]. Available: Http://dx.doi.org/10.7554/elife.06213
65. W. M. Fisher, tsylb2. 1996. [Online]. Available: Http://www.nist. gov/speech/tools
66. P. N. Garner, M. Cernak, and B. Potard, "A simple continuous excitation model for parametric vocoding," Idiap research report Idiap-RR-03-2015, 2015.
67. R. F. Kubichek, "Mel-cepstral distance measure for objective speech quality assessment," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process, vol. 1. Piscataway, NJ, USA: IEEE, May 1993, pp. 125-128.
68. Methods for Subjective Determination of Transmission Quality, ITU-T Rec. P.800, Geneva, Switzerland, 1996.
69. V. Grancharov andW. B. Kleijn, "Speech quality assessment," in Springer Handbook of Speech Processing, J. Benesty,M.M. Sondhi, and Y. Huang, Eds. Berlin, Germany: Springer, 2008, pp. 83-100.
70. C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, "An algorithm for intelligibility prediction of time-frequency weighted noisy speech," IEEE Trans. Audio, Speech, Lang. Process., vol. 19, no. 7, pp. 2125-2136, 2011. [Online]. Available: Http://dx.doi.org/10.1109/tasl.2011.2114881
71. S. Jørgensen, J. Cubick, and T. Dau, "Speech intelligibility evaluation for mobile phones," Acta Acust. United with Acust., vol. 105, pp. 1016-1025. [Online]. Available: Http://dx.doi.org/10.3813/aaa.918896
72. C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen, "A shorttime objective intelligibility measure for time-frequency weighted noisy speech," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., Mar. 2010, pp. 4214-4217. [Online]. Available: Http://dx.doi.org/10.1109/icassp.2010.5495701
73. One-Way Transmission Time, ITU-T Rec. G.114, Geneva, Switzerland, 2003.
74. G. Hinton, O. Vinyals, and J. Dean, "Distilling the knowledge in a neural network," Mar. 2015. [Online]. Available: Http://arxiv.org/abs/1503.02531
75. P. Nakkiran, R. Alvarez, R. Prabhavalkar, and C. Parada, "Compressing deep neural networks using a rank-constrained topology," in Proc. Interspeech, 2015, pp. 1473-1477.
76. V. Sindhwani, T. N. Sainath, and S. Kumar, "Structured transforms for small-footprint deep learning," Oct. 2015. [Online]. Available: Http://arxiv.org/abs/1510.01722
77. L. Lu and S. Renals, "Small-footprint deep neural networks with highway connections for speech recognition," CoRR, vol. abs/1512.04280, 2015. [Online]. Available: Http://arxiv.org/abs/1512.04280
78. N. D. Lane, P. Georgiev, and L. Qendro, "Deepear: Robust smartphone audio sensing in unconstrained acoustic environments using deep learning," in Proc. ACM Int. Joint Conf. Pervasive Ubiquitous Comput., 2015, pp. 283-294. [Online]. Available: Http://doi.acm. org/10.1145/2750858.2804262