Statistical multimodal integration for audio-visual speech processing

IEEE Transactions on Neural Networks

Volumn 13, Issue 4, 2002, Pages 854-866

  Nakamura, Satoshi ^a  

^a ADVANCED TELECOMMUNICATIONS RESEARCH INSTITUTE INTERNATIONAL   (Japan)

Author keywords

Audio visual; Bimodal; Hidden Markov model (HMM); Multimodal; Speech recognition; Speech synthesis; Speech translation

Indexed keywords

AUDITION; MARKOV PROCESSES; MATHEMATICAL MODELS; MODAL ANALYSIS; SPEECH RECOGNITION; SPEECH SYNTHESIS; VISION;

HIDDEN MARKOV MODELS (HMM);

SPEECH PROCESSING;

EID: 0036650148     PISSN: 10459227     EISSN: None     Source Type: Journal    
DOI: 10.1109/TNN.2002.1021886     Document Type: Article

References (101)

1. Hearing lips and seeing voices
2. Eye movement of perceivers during audiovisual speech perception
3. Visual contribution to speech intelligibility in noise
4. Which components of the face do humans and machines best speechread?
5. A comparison of models for fusion of the auditory and visual sensors in speech perception
6. Integration of acoustic and visual speech signals using neural networks
7. Audiovisual speech processing
8. Ten years after summerfield: A taxonomy of models for audio-visual fusion in speech perception
9. Seeing speech: Investigations into the synthesis and recognition of visible speech movements using automatic image processing and computer graphics
10. Automatic optically based recognition of speech
11. Neural network models of sensory integration for improved vowel recognition
12. Lip-reading of Japanese vowels using neural networks
13. Neural network vowel-recognition jointly using voice features and mouth shape image
14. Sources of structure in neural networks for speech and language
15. Neural network lipreading system for improved speech recogniton
16. Improving connected letter recognition by lipreading
17. 'Eigenlips' for robust speech recognition
18. Automatic speech recognition that includes visual speech cues
19. Toward movement-invariant automatic lip-reading and speech recognition
20. Audio-visual speech recognition compared across two architectures
21. Talking heads and speech recognizers that can see:the computer processing of visual speech signals
22. Visionary speech: Looking ahead to practical speechreading systems
23. On the integration of auditory and visual parameters in an HMM-based asr
24. On the integration of auditory and visual parameters in an HMM-based asr
25. Audio-visual sensory integration using hidden Markov models
26. Asynchronous integration of audio and visual sources in bimodal automatic speech recognition
27. Integrating audio and visual information to provide highly robust speech recognition
28. Handling disynchronization phenomena with HMM in connected speech
29. Speaker independent audiovisual database for bimodal asr
30. Improved bimodal speech recognition using tied-mixture HMMs and 5000 word audio-visual synchronous database
31. Combining noise compensation with visual information in speech recognition
32. Audio visual speech recognition and segmental master slave HMM
33. Adaptive determination of audio-visual weights for automatic speech recognition
34. Discriminative training of HMM stream exponents for audio-visual speech recognition
35. Speaker adaptation for audio-visual speech recognition
36. Intensity-and location-normalized training for HMM-based visual speech recognition
37. Perceptual interfaces for information interaction: Joint processing of audio and visual information for human-computer interaction
38. Labeling audio-visual speech corpora and training an ANN/HMM audio-visual speech recognition system
39. Stream weight optimization of speech and lip image sequence for audio-visual speech recognition
40. Audio-visual speech recognition using race-based HMMs and model-dependent stream weights
41. A cascade image transform for speaker independent automatic speechreading
42. Tracking of multiple faces for huan-computer interfaces and virtual environments
43. An adaptive integration based on product HMM for audio-visual speech recognition
44. State synchroonous modeling of audio-visual information for bi-modal speech recognition
45. Nonlinear manifold learning for visual speech recognition
46. Sensory integration in audiovisual automatic speech recognition
47. The multilayer perceptron as a tool for speech pattern processing research
48. Improving connected letter recognition by lipreading
49. Using deformable templates to infer visual speech dynamics
50. Bimodal recognition experiments with recurrent neural networks
51. Automatic bimodal speech recognition
52. Speaker independent audio-visual speech recognition
53. Bimodal speech recognition using coupled hidden Markov models
54. A new approach to integrate audio and visual features of speech
55. Asynchronous stream modeling for large vocabulary audio-visual speech recognition
56. Duration dependent input output Markov models for audio-visual event detection
57. Comparing audio- and a posteriori-probability-based stream confidence measure for audio-visual speech recognition
58. Models for audiovisual fusion in a noisy-vowel recognition task
59. Maximum likelihood weighting of dynamic speech features for CDHMM speech recognition
60. Pattern recognition using a family of design algorithms based upon the generalized probabilistic descent method
61. Stream confidence estimation for audio-visual speech recognition
62. Weighting schemes for audio-visual fusion in speech recognition
63. Audio-visual synthesis of talking faces from speech production correlates
64. Real time speech driven facial animation using formant analysis
65. Lip synchronization using linear predictive analysis
66. An intelligent facial image coding driven by speech and phoneme
67. Codebook based face point trajectory synthesis algorithm using speech input
68. A media conversion from speech to facial image for intelligent man-machine interface
69. Converting speech into lip movements: A multimedia telephone for hard of hearing people
70. Cross-modal prediction in audio-visual communication
71. Driving synthetic mouth gestures: Phonetic recognition for faceme!
72. Generation of mouthshape for a synthetic talking head
73. Speech recognition for image animation and coding
74. A new frame interpolation scheme for talking head sequences
75. Speech-to-lip movement synthesis by HMM
76. Two- and three-dimensional audio-visual speech synthesis
77. Text-to-visual speech synthesis based on parameter generation from HMM
78. Subjective evaluation for HMM-based speech-to-lip movement synthesis
79. Lip movement synthesis from speech based on hidden Markov models
80. Real-time lip-synch face animation driven by human voice
81. Text-audio-visual speech synthesis based on parameter generation from HMM
82. HMM-based text-to-audio-visual speech synthesis
83. Translingual visual speech synthesis
84. Speech-to-face movement synthesis based on HMMs
85. Speech-to-lip movement synthesis by maximizing audio-visual joint probability based on the em algorithm
86. Hidden Markov model inversion for audio-to-visual conversion in an mpeg-4 facial animation system
87. Speech-to-lip movement synthesis based on em algorithm using audio-visual HMMs
88. A 3-D model of the lips for visual speech synthesis
89. Model-based lip synchronization with a automatically translated synthetic voice toward a multimodal translation system
90. Automatic selection of visemes for image-based visual speech synthesis
91. Japanese-to-English speech translation system:atrmatrix
92. 3-D lip expression generation by using new lip parameters
93. Automatic face tracking and model match move in video sequence using 3D face model