A computational intelligibility model for assessment and compression of American sign language video

IEEE Transactions on Image Processing

Volumn 20, Issue 11, 2011, Pages 3014-3027

  Ciaramello, Frank M ^a   Hemami, Sheila S ^a,b  

^a Cornell University   (United States)

^b Department of Obstetrics Gynecology   (United States)

Author keywords

Region of interest video; video compression; video conference; video quality; visual communication

Indexed keywords

AMERICAN SIGN LANGUAGE; BIT-RATE REDUCTION; CELLULAR NETWORK; CLOSED-LOOP; COMMUNICATION TOOLS; COMPRESSED VIDEO; COMPUTATIONAL MODEL; EXPERIMENTAL DATA; GENERAL PURPOSE; GROUND TRUTH; H.264 ENCODERS; HUMAN PERCEPTION; NATURAL COMMUNICATION; REGION OF INTEREST; RELATIVE IMPACT; TEMPORAL POOLING; TWO-WAY TRANSMISSION; VIDEO CONFERENCE; VIDEO QUALITY;

CELLULAR NEURAL NETWORKS; COMPUTATIONAL METHODS; IMAGE CODING; IMAGE COMMUNICATION SYSTEMS; IMAGE COMPRESSION; OPTIMIZATION; VIDEO SIGNAL PROCESSING; VISUAL COMMUNICATION;

COMMUNICATION;

ARTICLE; COMPUTER SIMULATION; HUMAN; INFORMATION PROCESSING; METHODOLOGY; NONVERBAL COMMUNICATION; SIGN LANGUAGE; VIDEORECORDING;

COMPUTER SIMULATION; DATA COMPRESSION; HUMANS; MANUAL COMMUNICATION; SIGN LANGUAGE; VIDEOTAPE RECORDING;

EID: 80054796995     PISSN: 10577149     EISSN: None     Source Type: Journal    
DOI: 10.1109/TIP.2011.2132730     Document Type: Article

References (52)

1. E. Riskin, S. Hemami, R. Ladner, and J. Wobbrock, The MobileASL Project [Online]. Available: http://www. cs. washington. edu/research/MobileASL/index. html
2. F. Grosjean, "A study of timing in a manual and a spoken language: American Sign Language and English," J. Psycholinguistic Res., vol. 8, no. 4, pp. 379-405, Jul. 1979.
3. General Packet Radio Service GSMA, 2010 [Online]. Available: http://www. gsmworld. com/technology/gprs. htm
4. T. Wiegand, G. Sullivan, G. Bjontegaard, and A. Luthra, "Overview of the H. 264/AVC video coding standard," IEEE Trans. Circuits. Syst. Video Technol., vol. 13, no. 7, pp. 560-576, Jul. 2003.
5. A. Cavender, R. E. Ladner, and E. A. Riskin, "MobileASL: Intelligibility of sign language video as constrained by mobile phone technology," in Proc. Int. ACMSIGACCESS Conf. Comput. Accessibil. (ASSETS), 2006, pp. 71-78.
6. x264 (Rev. 736), [Online]. Available: http://developers. videolan. org/x264. html
7. R. Schumeyer, E. Heredia, and K. Barner, "Region of interest priority coding for sign language video conferencing," in Proc. IEEE Multimedia Signal Process. Workshop, 1997, pp. 531-536.
8. D. M. Saxe and R. A. Foulds, "Robust region of interest coding for improved sign language telecommunication," IEEE Trans. Inf. Technol. Biomed., vol. 6, no. 4, pp. 310-316, Dec. 2002.
9. K. Nakazono, Y. Nagashima, and A. Ichikawa, "Digital encoding applied to sign language video," IEICE Trans. Inf. Syst., vol. E89-D, no. 6, pp. 1893-1900, Jun. 2006.
10. D. Agrafiotis, N. Canagarajah, D. R. Bull, J. Kyle, H. Seers, and M. Dye, "A perceptually optimised video coding system for sign language communication at lowbit rates," Signal Process. : Image Commun., vol. 21, no. 7, pp. 531-549, 2006.
11. M. Barkowsky, J. Bialkowski, B. Eskofier, R. Bitto, and A. Kaup, "Temporal trajectory aware video quality measure," IEEE Sel. Top. Signal Process., vol. 3, no. 2, pp. 266-279, Apr. 2009.
12. K. Seshadrinathan and A. C. Bovik, "An information theoretic video quality metric based on motion models," in Proc. Workshop Video Quality Metrics, Jan. 2007, pp. 25-26.
13. A. Ninassi, O. L. Meur, P. L. Callet, and D. Barba, "Considering temporal variations of spatial visual distortions in video quality assessment," IEEE Sel. Topics Signal Process., vol. 3, no. 2, pp. 253-265, Apr. 2009.
14. Z. Wang, L. Lu, and A. C. Bovik, "Video quality assessment based on structural distortion measurement," Signal Process. : Image Commun., vol. 19, no. 2, pp. 121-132, Feb. 2004.
15. S. Quackenbush, T. Barnwell, III, and M. Clements, Objective Measures of Speech Quality. Englewood Cliffs, NJ: Prentice-Hall, 1988.
16. A. Raake, Speech Quality of VOIP. West Sussex, England: Wiley, 2006.
17. C. Valli and C. Lucas, Linguistics of American Sign Language: An Introduction. Washington, DC: Gallaudet Univ. Press, 2000.
18. S. K. Liddell and R. E. Johnson, "American Sign Language: The phonological base," Sign Lang. Stud., vol. 64, pp. 195-278, 1989.
19. C. Baker and C. A. Padden, "Focusing on the nonmanual components of American Sign Language," Understand. Lang. Through Sign Lang. Res., pp. 27-57, 1978.
20. S. K. Liddell, "Nonmanual signals and relative clauses in American Sign Language," Understand. Lang. Through Sign Lang. Res., vol. 1, pp. 59-90, 1978.
21. R. Battison, Lexical Borrowing in American Sign Language. Silver Spring, MD: Linstok Press, 1978.
22. P. Siple, "Visual constraints for sign language communication," Sign Lang. Stud., vol. 19, pp. 95-110, 1978.
23. L. J. Muir and I. E. G. Richardson, "Perception of sign language and its application to visual communications for deaf people," J. Deaf Stud. Deaf Educ., vol. 10, no. 4, pp. 390-401, Oct. 2005.
24. K. Emmorey, R. Thompson, and R. Colvin, "Eye gaze during comprehension of American Sign Language by native and beginning signers," J. Deaf Stud. Deaf Educ., vol. 14, no. 2, pp. 237-243, Apr. 2009.
25. S. Hooper, C. Miller, S. Rose, and G. Veletsianos, "The effects of digital video quality on learner comprehension in an American Sign Language assessment environment," Sign Lang. Stud., vol. 8, no. 1, pp. 42-58, 2007.
26. J. Harkins, A. Wolff, E. Korres, R. Foulds, and S. Galuska, "Intelligibility experiments with a feature extraction system designed to simulate a low-bandwidth video telephone for deaf people," in Proc. RESNA Annu. Conf., 1991, vol. 14, pp. 38-40.
27. N. Cherniavsky, A. Cavender, E. Riskin, and R. Ladner, "Variable frame rate for low power mobile sign language communication," in Proc. Int. ACM SIGACCESS Conf. Comput. Accessibil. (ASSETS), 2007, pp. 163-170.
28. G. Sperling, M. Landy, Y. Cohen, and M. Pavel, "Intelligible encoding of ASL image sequences at extremely low information rates," Comput. Vis., Graph., Image Process., vol. 31, pp. 335-391, 1985.
29. P. Letellier, M. Nadler, and J.-F. Abramatic, "The telesign project," Proc. IEEE, vol. 73, no. 4, pp. 813-827, Apr. 1985.
30. M. D. Manoranjan and J. A. Robinson, "Practical low-cost visual communication using binary images for deaf sign language," IEEE Trans. Rehabil. Eng., vol. 8, no. 1, pp. 81-88, Mar. 2000.
31. P. Kortum, W. S. Geisler, B. E. Rogowitz, and J. P. Allebach, "Implementation of a foveated image coding system for image bandwidth reduction," in Proc. SPIE: Human Vis. Electron. Imag., San Jose, CA, Apr. 1996, vol. 2657, pp. 350-360.
32. M. Yuen and H. R. Wu, "A survey of hybrid MC/DPCM/DCT video coding distortions," Signal Process., vol. 70, no. 3, pp. 247-278, 1998.
33. M. A. Changizi, A. Hsieh, R. Nijhawan, R. Kanai, and S. Shimojo, "Perceiving the present and a systematization of illusions," Cognitive Sci. : Multidisciplinary J., vol. 32, no. 3, pp. 459-503, 2008.
34. N. Habili, C. C. Lim, and A. Moini, "Segmentation of the face and hands in sign language video sequences using color and motion cues," IEEE Trans. Circuits Syst. Video Technol., vol. 14, no. 8, pp. 1086-1096, Aug. 2004.
35. G. Awad, J. Han, and A. Sutherland, "A unified system for segmentation and tracking of face and hands in sign language recognition," in Proc. Int. Conf. Pattern Recognit., vol. 1, pp. 239-242.
36. S. Phung, A. S. Bouzerdoum, and D. S. Chai, "Skin segmentation using color pixel classification: Analysis and comparison," IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 1, pp. 148-154, Jan. 2005.
37. P. Viola and M. Jones, "Rapid object detection using a boosted cascade of simple features," in Proc. Comput. Vis. Pattern Recognit., 2001, pp. 511-518.
38. F. Ciaramello and S. Hemami, "Real-time face and hand detection for video conferencing on a mobile device," in Proc. Workshop Video Quality Metrics, 2009.
39. D. Chandler and S. Hemami, "VSNR: A wavelet-based visual signal-to-noise ratio for natural images," IEEE Trans. Image Process., vol. 16, no. 9, pp. 2284-2298, Sep. 2007.
40. F. Ciaramello and S. Hemami, "Quantifying the effect of disruptions to temporal coherence on the intelligibility of compressed American Sign Language video," in Proc. SPIE: Human Vis. Electron. Imag., Feb. 2009, vol. 7240.
41. F. Ciaramello and S. Hemami, "The influence of space and time varying distortions on objective intelligibility estimators for region-of-interest video," in Proc. IEEE Int. Conf. Image Proc., 2010, pp. 1097-1100.
42. P. 800: Telephone Transmission Quality: Methods for Subjective Determination of Transmission Quality ITU, 1996.
43. J. Deller, J. Hansen, and J. Proakis, Discrete-Time Processing of Speech Signals. New York: IEEE Press, 2000, ch. 9.
44. U. Jekosch, Voice and Speech Quality Perception. Heidelberg, Germany: Springer-Verlag, 2005, ch. 10.
45. "Final report from theVideo Quality Experts Group on the validation of objective models of video quality assessment," Video Quality Experts Group (VQEG), 2000.
46. "Final report from theVideo Quality Experts Group on the validation of objective models of video quality assessment, Phase II (fr-tv2)," Video Quality Experts Group (VQEG), 2003.
47. "Final report from the Video Quality Experts Group on the validation of objective models of multimedia quality assessment, Phase I," Video Quality Experts Group (VQEG), 2008.
48. K. Seshadrinathan, R. Soundararajan, A. C. Bovik, and L. Cormack, "Study of subjective and objective quality assessment of video,", vol. 19, no. 6, pp. 1427-1441, Jun. 2010.
49. Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli, "Image quality assessment: From error visibility to structural similarity," IEEE Trans. Image Process., vol. 13, no. 4, pp. 600-612, Apr. 2004.
50. D. Rouse and S. Hemami, "Natural image utility assessment using image contours," in Proc. IEEE Int. Conf. Image Proc., 2009, pp. 2217-2220.
51. T. Wiegand, H. Schwarz, A. Joch, F. Kossentini, and G. J. Sullivan, "Rate-constrained coder control and comparison of video coding standards," IEEE Trans. Circuits Syst. Video Technol., vol. 13, no. 7, pp. 688-703, Jul. 2003.
52. F. Ciaramello and S. Hemami, "Complexity constrained rate-distortion optimization of sign language video using an objective intelligibility metric," in Proc. SPIE Visual Commun. Image Process., Jan. 2008, vol. 6822, pp. 682213-1-682213-10.