The road to immersive communication

Proceedings of the IEEE

Volumn 100, Issue 4, 2012, Pages 974-990

  Apostolopoulos, John G ^a   Chou, Philip A ^b   Culbertson, Bruce ^a   Kalker, Ton ^c   Trott, Mitchell D ^a   Wee, Susie ^d  

^a HEWLETT PACKARD LABORATORIES   (United States)

^b MICROSOFT RESEARCH   (United States)

^c Huawei Innovation Center   (United States)

^d CISCO SYSTEMS   (United States)

Author keywords

Immersive environments; signal processing; telepresence; video conferencing

Indexed keywords

AUGMENTED REALITY; CELL PROLIFERATION; DRAWING (GRAPHICS); SIGNAL PROCESSING; VISUAL COMMUNICATION;

HIGH QUALITY VIDEO; IMMERSIVE ENVIRONMENT; PERCEPTUAL MODEL; REMOTE COMMUNICATION; SOUND REPRODUCTION SYSTEMS; STATE OF THE ART; TECHNICAL CHALLENGES; TELEPRESENCE;

VIDEO CONFERENCING;

EID: 84859004156     PISSN: 00189219     EISSN: None     Source Type: Journal    
DOI: 10.1109/JPROC.2011.2182069     Document Type: Conference Paper

References (80)

1. T. Friedman, The World Is Flat: A Brief History of the Twenty-First Century, 3rd ed. New York: Farrar, Straus and Giroux, 2007.
2. T. Clark, Faraway, So Close: Doing Global iPhone Dev from New Zealand, Jan. 2011. [Online]. Available: http://arstechnica.com/gaming/2011/01/sell-in- america-live-in-newzealand. ars.
3. J. N. Bailenson, A. C. Beall, and J. Blascovich, "Mutual gaze and task performance in shared virtual environments," Personality Social Psychol. Bull., pp. 1-15, 2003.
4. A. Pentland, Honest Signals: How They Shape Our World. Cambridge, MA: MIT Press, 2008.
5. A. Vinciarelli, M. Pantic, and H. Bourland, "Social signal processing: Survey of an emerging domain," J. Image Vis. Comput., vol. 27, no. 12, pp. 1743-1759, 2009.
6. M. Zimmerman, "The nervous system in the context of information theory," in Human Physiology, R. F. Schmidt and G. Thews, Eds. New York: Springer-Verlag, 1989, pp. 166-173.
7. B. Hoang and O. Perez, Terabit Networks. [Online]. Available: http://www.ieee.org/portal/site/emergingtech/techindex.jsp? techId=1140.
8. M. Minsky, "Telepresence," OMNI Mag., pp. 44-52, Jun. 1980.
9. G. du Maurier, "Edison's telephonoscope (transmits light as well as sound)," Punch's Almanack, Dec. 9, 1878.
10. A. G. Bell, "On the possibility of seeing by electricity," in Alexander Graham Bell Family Papers. Washington, DC: Library of Congress, Apr. 10, 1891.
11. I. E. Sutherland, "The ultimate display," in Proc. IFIP Congr., 1965, pp. 506-508.
12. "S.F. man's invention to revolutionize television," San Francisco Chronicle, vol. CXXXIII, no. 50, Sep. 3, 1928.
13. C. Cruz-Niera, D. J. Sandin, T. A. DeFanti, R. V. Kenyon, and J. C. Hart, "The CAVE: Audio visual experience automatic virtual environment," Commun. ACM, vol. 35, no. 6, Jun. 1992, DOI: 10.1145/129888.129892.
14. J. Fihlo, K. M. Inkpen, and M. Czerwinski, "Image, appearance and vanity in the use of media spaces and videoconference systems," Proc. ACM Int. Conf. Supporting Group Work, May 2009, DOI: 10.1145/1531674.1531712.
15. D. Lindbergh, "The past, present, and future of video telephony: A systems perspective," Keynote, Packet Video Workshop, Dec. 2004.
16. N. Jouppi, S. Iyer, S. Thomas, and A. Slayden, "BiReality: Mutually-immersive telepresence," Proc. ACM Int. Conf. Multimedia, 2004, DOI: 10.1145/1027527. 1027725.
17. E. H. Adelson and J. R. Bergen, "The plenoptic function and the elements of early vision," in Computation Models of Visual Processing, M. Landy and J. A. Movshon, Eds. Cambridge, MA: MIT Press, 1991, pp. 3-20.
18. A. Jeans, M. Almanza-Workman, R. Cobene, R. Elder, R. Garcia, R. F. Gomez-Pancorbo, W. Jackson, M. Jam, H.-J. Him, O. Kwon, H. Luo, J. Maltabes, P. Mei, C. Perlov, M. Smith, and C. Taussig, "Advances in roll-to-roll imprint lithography for display applications," in Proc. SPIE, Feb. 2010, vol. 7637, p. 763719.
19. H. J. Kim, M. Almanza-Workman, B. Garcia, O. Kwon, F. Jeffrey, S. Braymen, J. Hauschildt, K. Junge, D. Larson, D. Stieler, A. Chaiken, B. Cobene, R. Elder, W. Jackson, M. Jam, A. Jeans, H. Luo, P. Mei, C. Perlov, and C. Taussig, "Roll-to-roll manufacturing of electronics on flexible substrates using self-aligned imprint lithography (SAIL)," J. Soc. Inf. Display, vol. 17, p. 963, 2009, DOI: 10.1889/JSID17.11.963.
20. A. Jones, M. Lang, G. Fyffe, X. Yu, J. Busch, I. McDowall, M. Bolas, and P. Debevec, "Achieving eye contact in a one-to-many 3D video teleconferencing system," ACM Trans. Graphics, vol. 28, no. 3, Jul. 2009, DOI: 10.1145/1531326.1531370.
21. E. H. Adelson and J. Y. A. Wang, "Single lens stereo with plenoptic camera," IEEE Trans. Pattern Anal. Mach. Intell., vol. 14, no. 2, pp. 99-106, Feb. 1992.
22. R. Ng, M. Levoy, M. Bredif, G. Duval, M. Horowitz, and P. Hanrahan, "Light field photography with a hand-held plenoptic camera," Stanford Univ. Comput. Sci., Stanford, CA, Tech Rep. CSTR 2005-02, Apr. 2005.
23. M. Levoy and P. Hanrahan, "Light field rendering," Proc. ACM SIGGRAPH, 1996, pp. 31-42.
24. S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. Cohen, "The lumigraph," Proc. ACM SIGGRAPH, 1996, pp. 43-54.
25. C. Zhang and T. Chen, "A survey on image-based rendering- representation, sampling and compression," in EURASIP Signal Process., Image Commun., 2004, vol. 19, pp. 1-28.
26. J. X. Chai, S. C. Chan, H. Y. Shum, and X. Tong, "Plenoptic sampling," in Proc. SIGGRAPH, 2000, pp. 307-318. (Pubitemid 32393994)
27. C. Zhang and T. Chen, "Spectral analysis for sampling image-based rendering data," IEEE Trans. Circuits Syst. Video Technol., vol. 13, no. 11, pp. 1038-1050, Nov. 2003.
28. M. N. Do, D. Marchand-Maillet, and M. Vetterli, "On the bandwidth of the plenoptic function," IEEE Trans. Image Process., vol. 21, no. 2, pp. 708-717, Feb. 2012.
29. J. Salvi, S. Fernandez, T. Pribanic, and X. Llado, "A state of the art in structured light patterns for surface profileometry," Pattern Recognit., vol. 43, no. 8, pp. 2666-2680, Aug. 2010.
30. Q. Yang, K. H. Tan, B. Culbertson, and J. Apostolopoulos, "Fusion of active and passive sensors for fast 3D capture," Proc. IEEE Int. Workshop Multimedia Signal Process., Oct. 2010, pp. 69-74.
31. A. Kolb, E. Barth, and R. Koch, "ToF-sensors: New dimensions for realism and interactivity," Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2008, DOI: 10.1109/CVPRW.2008. 4563159.
32. H. Urey, K. V. Chellappan, E. Erden, and P. Surman, "State of the art in stereoscopic and autostereeoscopic displays," Proc. IEEE, vol. 99, no. 4, pp. 540-555, Apr. 2011.
33. K.-H. Tan, I. Robinson, R. Samadani, B. Lee, D. Gelb, A. Vorbau, B. Culbertson, and J. Apostolopoulos, "Connectboard: A remote collaboration system that supports gaze-aware interaction and sharing," Proc. IEEE Int. Workshop Multimedia Signal Process., 2009, DOI: 10.1109/MMSP.2009.5293268.
34. S. C. Chan, H. Y. Shum, and K. T. Ng, "Image-based rendering and synthesis," IEEE Signal Process. Mag., vol. 24, no. 7, pp. 22-33, Nov. 2007. (Pubitemid 350201596)
35. R. Szeliski, Computer Vision: Algorithms and Applications. New York: Springer-Verlag, 2011.
36. M. Hirsch, D. Lanman, H. Holtzman, and R. Raskar, "BiDi screen: A thin, depth-sensing LCD for 3D interaction using light fields," ACM Trans. Graphics, vol. 28, no. 5, 2009.
37. L. Chiariglione, "Multimedia standards: Interfaces to innovation," Proc. IEEE, to be published.
38. V. M. Bove, "Display holography's digital second act," Proc. IEEE, vol. 100, no. 4, Apr. 2012, DOI: 10.1109/JPROC.2011. 2182071.
39. A. Vetro, T. Wiegand, and G. J. Sullivan, "Overview of the stereo and multiview video coding extensions of the H.264/MPEG-4 AVC standard," Proc. IEEE, vol. 99, no. 4, pp. 626-642, Apr. 2011.
40. B. Girod, A. M. Aaron, S. Rane, and D. Rebollo-Monedero, "Distributed video coding," Proc. IEEE, vol. 93, no. 1, pp. 71-83, Jan. 2005. (Pubitemid 40851242)
41. J. Y. A. Wang and E. Adelson, "Representing moving images with layers," IEEE Trans. Image Process., vol. 3, no. 5, pp. 625-638, Sep. 1994.
42. A. Smolic and P. Kauff, "Interactive 3D video representations and coding technologies," Proc. IEEE, vol. 93, no. 1, pp. 98-110, Jan. 2005. (Pubitemid 40851244)
43. B. L. Vercoe, W. G. Gardner, and E. D. Scheirer, "Structured audio: Creation, transmission, and rendering of parametric sound representations," Proc. IEEE, vol. 86, no. 5, pp. 922-940, May 1998.
44. P. Hinterseer, S. Hirche, S. Chaudhuri, E. Steinbach, and M. Buss, "Perception-based data reduction and transmission of haptic data in telepresence and teleaction systems," IEEE Trans. Signal Process., vol. 56, no. 2, pp. 588-597, Feb. 2008. (Pubitemid 351285056)
45. E. Steinbach, S. Hirche, J. Kammerl, I. Vittorias, and R. Chaudhari, "Haptic data compression and communication," IEEE Signal Process. Mag., vol. 28, no. 1, pp. 87-96, Jan. 2011.
46. J. N. Bailenson, A. C. Beall, and J. Blascovich, "Gaze and task performance in shared virtual environments," J. Vis. Comput. Animat., vol. 13, no. 5, pp. 313-320, 2002. (Pubitemid 135714462)
47. G. Venolia, J. Tang, R. Cervantes, S. Bly, G. Robertson, B. Lee, K. Inkpen, and S. Drucker, "Embodied social proxy: Connecting hub-and-satellite teams," Proc. Comput. Supported Cooperative Work, 2010.
48. P. E. Spector, Summated Rating Scale Construction: An Introduction. Thousand Oaks, CA: Sage Publications, 1992.
49. J. Tang and S. Minneman, "VideoWhiteboard: Video shadows to support remote collaboration," Proc. SIGHI Conf. Human Factors Comput. Syst., 1991, pp. 315-322, DOI: 10.1145/108844.108932.
50. H. Ishii and M. Kobayashi, "ClearBoard: A seamless medium for shared drawing and conversation with eye contact," Proc. SIGHI Conf. Human Factors Comput. Syst., 1992, DOI: 10.1145/142750.142977.
51. A. Tang, M. Pahud, K. Inkpen, H. Benko, J. C. Tang, and B. Buxton, "Three's company: Understanding communication channels in a three-way distributed collaboration," Proc. Comput. Supported Cooperative Work, 2010, pp. 271-280.
52. D. T. Nguyen and J. Canny, "Multiview: Improving trust in group video conferencing through spatial faithfulness," Proc. SIGHI Conf. Human Factors Comput. Syst., 2007, DOI: 10.1145/1240624.1240846. (Pubitemid 47581738)
53. Methods for Objective and Subjective Assessment of Quality, ITU-T Recommendation P.800, 1996.
54. N. S. Jayant, J. D. Johnston, and R. J. Safranek, "Signal compression based on models of human perception," Proc. IEEE, vol. 81, no. 10, pp. 1385-1422, Oct. 1993.
55. B. C. J. Moore, Psychology of Hearing, 5th ed. New York: Academic, 2003.
56. M. H. Pirenne, "Visual acuity," in The Eye, vol. 2, H. Davson, Ed. New York: Academic, 1962.
57. One-Way Transmission Time, ITU-T Recommendation G.114, 2003.
58. Relative Timing of Sound and Vision for Broadcasting, ITU-R BT.1359-1, 1998.
59. E. Geelhoed, A. Parker, D. J. Williams, and M. Groen, "Effects of latency on telepresence," HP Labs, Tech. Rep. HPL-2009-120, 2009.
60. M. Chen, "Leveraging the asymmetric sensitivity of eye contact for videoconferencing," Proc. SIGHI Conf. Human Factors Comput. Syst., 2002, DOI: 10.1145/503376.503386.
61. I. Cheng, R. Shen, X.-D. Yang, and P. Boulanger, "Perceptual analysis of level-of-detail: The JND approach," Proc. Int. Symp. Multimedia, 2006, pp. 533-540.
62. A. Benoit, P. Le Callet, P. Campisi, and R. Cousseau, "Quality assessment of stereoscopic images," EURASIP J. Image Video Process., vol. 2008, 2008, DOI:10.1155/2008/659024.
63. S. Boyne, N. Pavlovic, R. Kilgore, and M. H. Chignell, "Auditory and visual facilitation: Cross-modal fusion of information in multi-modal displays," in Meeting Proc.-Visualisation Common Operational Picture., Neuilly-sur-Seine, France, pp. 19-1-19-4, 2005, Paper 19, RTO-MP-IST-043. [Online]. Available: http://www.rto.nato.int/abstracts.asp.
64. M. Lambooij, W. IJsselsteijn, and I. Heynderickx, "Stereoscopic displays and visual comfort: A review," in SPIE Newsroom, Apr. 2, 2007, DOI: 10.1117/2. 1200703.0648.
65. V. Walsh and J. Kulikowski, Eds., Perceptual Constancy: Why Things Look as They Do. Cambridge, U.K.: Cambridge Univ. Press, 1996.
66. J. Blascovich and J. Bailenson, Infinite Reality: Avatars, Eternal Life, New Worlds, and the Dawn of the Virtual Revolution. New York: Harper-Collins, 2011.
67. B. Reeves and C. Nass, The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Places. Stanford, CA: CSLI Publications, 1996.
68. QSound Labs, Virtual Barber Shop, May 5, 2011. [Online]. Available: http://www. qsound.com/demos/binaural-audio.htm.
69. V. R. Algazi and R. O. Duda, "Headphone-based spatial sound," IEEE Signal Process. Mag., vol. 28, no. 1, pp. 33-42, Jan. 2011.
70. B. Kapralos, M. R. Jenkin, and E. Milios, "Virtual audio systems," Presence, vol. 17, no. 6, pp. 527-549, 2008.
71. I. Toshima, S. Aoki, and T. Hirahara, "An acoustical tele-presence robot: TeleHead II," in Proc. IEEE/RSJ Int. Conf. Intel. Robots Syst., Sendai, Japan, Sep. 2004, pp. 2105-2110. (Pubitemid 40275910)
72. W. Gardner, "3-D audio using loudspeakers," Ph.D. dissertation, Progr. Media Arts Sci., Schl. Archit. Planning, Massachusetts Inst. Technol., Cambridge, MA, 1997.
73. T. Grämer, "Efficient modeling of head movements and dynamic scenes in virtual acoustics," M.S. thesis, Univ. Zurich/ETH, Faculty Med., 2010.
74. D. R. Begault, "Auditory and non-auditory factors that potentially influence virtual acoustic imagery," in Proc. 16th Audio Eng. Soc. Int. Conf. Spatial Sound Reproduction, Mar. 1999.
75. A. R. A. Conway, N. Cowan, and M. F. Bunting, "The cocktail party phenomenon revisited: The importance of working memory capacity," Psychonomic Bull. Rev., vol. 8, no. 2, pp. 331-335, 2001. (Pubitemid 33604184)
76. S. Haykin and Z. Chen, "The cocktail party problem," Neural Comput., vol. 17, no. 9, pp. 1875-1902, 2005. (Pubitemid 41053627)
77. Second Life. [Online]. Available: http://www. secondlife.com
78. Yoda. [Online]. Available: http://en. wikipedia.org/wiki/Yoda
79. U.S. National Remote Working Survey, 2010. [Online]. Available: http://www.microsoft. com/presspass/download/features/2010/ NationalRemoteWorkingSummary.pdf
80. BIceland volcano ash hits European flights again,[ Agence France-Presse, May 8, 2010. [Online]. Available: http://news.ph.msn.com/top-stories/article. aspx?cp-documentid= 4076015; http://www.google.com/hostednews/afp/article/ ALeqM5gBr-mBjSK4JCW3iODp3Hhw5S5SiQ.