Overview of the stereo and multiview video coding extensions of the H.264/MPEG-4 AVC standard

Proceedings of the IEEE

Volumn 99, Issue 4, 2011, Pages 626-642

  Vetro, Anthony ^a   Wiegand, Thomas ^b,c   Sullivan, Gary J ^d  

^a MITSUBISHI ELECTRIC RESEARCH LABORATORIES   (United States)

^b TECHNISCHE UNIVERSITÄT BERLIN   (Germany)

^c FRAUNHOFER HEINRICH HERTZ INSTITUTE   (Germany)

^d MICROSOFT   (United States)

Author keywords

3 D video; Advanced video coding (AVC); Blu Ray disc; H.264; inter view prediction; MPEG 4; multiview video coding (MVC); standards; stereo video

Indexed keywords

DECODING; FORECASTING; IMAGE CODING; IMAGE COMPRESSION; ISO STANDARDS; LEGACY SYSTEMS; MOTION PICTURE EXPERTS GROUP STANDARDS; STANDARDS; STEREO IMAGE PROCESSING; VIDEO CAMERAS;

3-D VIDEOS; ADVANCED VIDEO CODING; BLU-RAY DISC; H.264; INTER-VIEW PREDICTIONS; MPEG-4; MULTIVIEW VIDEO CODING; STEREO VIDEO;

VIDEO SIGNAL PROCESSING;

EID: 79953025534     PISSN: 00189219     EISSN: None     Source Type: Journal    
DOI: 10.1109/JPROC.2010.2098830     Document Type: Conference Paper

References (61)

1. ITU-T and ISO/IEC JTC 1, Advanced video coding for generic audiovisual services, ITU-T Recommendation H.264 and ISO/IEC 14496-10 (MPEG-4 AVC), 2010.
2. M. E. Lukacs, " Predictive coding of multi-viewpoint image sets," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., Tokyo, Japan, 1986, vol. 1, pp. 521-524.
3. I. Dinstein, G. Guy, J. Rabany, J. Tzelgov, and A. Henik, " On the compression of stereo images: Preliminary results," Signal Process., Image Commun., vol. 17, no. 4, pp. 373-382, Aug. 1989.
4. M. G. Perkins, " Data compression of stereo pairs," IEEE Trans. Commun., vol. 40, no. 4, pp. 684-696, Apr. 1992.
5. ITU-T and ISO/IEC JTC 1, Generic coding of moving pictures and associated audio informationVPart 2: Video, ITU-T Recommendation H.262 and ISO/IEC 13818-2 (MPEG-2 Video), 1994.
6. ITU-T and ISO/IEC JTC 1, Final draft amendment 3, Amendment 3 to ITU-T Recommendation H.262 and ISO/IEC 13818-2 (MPEG-2 Video), ISO/IEC JTC 1/SC 29/WG 11 (MPEG) Doc. N1366, Sep. 1996.
7. A. Puri, R. V. Kollarits, and B. G. Haskell, " Stereoscopic video compression using temporal scalability," in Proc. SPIE Conf. Vis. Commun. Image Process., 1995, vol. 2501, pp. 745-756.
8. X. Chen and A. Luthra, " MPEG-2 multi-view profile and its application in 3DTV," in Proc. SPIE IS&T Multimedia Hardware Architectures, San Diego, CA, Feb. 1997, vol. 3021, pp. 212-223.
9. J.-R. Ohm, " Stereo/multiview video encoding using the MPEG family of standards," in Proc. SPIE Conf. Stereoscopic Displays Virtual Reality Syst. VI, San Jose, CA, Jan. 1999, DOI:10.1117/12.349385.
10. G. J. Sullivan, " Standards-based approaches to 3D and multiview video coding," in Proc. SPIE Conf. Appl. Digital Image Process. XXXII, San Diego, CA, Aug. 2009, DOI: 10.1117/12. 830634.
11. A. Vetro, P. Pandit, H. Kimata, A. Smolic, and Y.-K. Wang, Joint draft 8 of multiview video coding, Hannover, Germany, Joint Video Team (JVT) Doc. JVT-AB204, Jul. 2008.
12. G. J. Sullivan, A. M. Tourapis, T. Yamakage, and C. S. Lim, Draft AVC amendment text to specify constrained baseline profile, stereo high profile, and frame packing SEI message, London, U.K., Joint Video Team (JVT) Doc. JVT-AE204, Jul. 2009.
13. MPEG Requirements Sub-Group, Requirements on multi-view video coding v.7, Klagenfurt, Austria, ISO/IEC JTC 1/SC 29/WG 11 (MPEG) Doc. N8218, Jul. 2006.
14. J. Konrad and M. Halle, B3-D displays and signal processingVAn answer to 3-D Ills?[ IEEE Signal Process. Mag., vol. 24, no. 6, Nov. 2007, pp. 11-97.
15. N. A. Dodgson, " Autostereoscopic 3D displays," IEEE Computer, vol. 38, no. 8, pp. 31-36, Aug. 2005. (Pubitemid 41245041)
16. A. Smolic and P. Kauff, " Interactive 3-D video representation and coding technologies," Proc. IEEE, vol. 93, no. 1, pp. 98-110, Jan. 2005. (Pubitemid 40851244)
17. T. Fujii, T. Kimono, and M. Tanimoto, " Free-viewpoint TV system based on ray-space representation," in Proc. SPIE ITCom, 2002, vol. 4864-22, pp. 175-189.
18. P. Kauff, O. Schreer, and R. Tanger, " Virtual team user environmentsVA mixed reality approach for immersive tele-collaboration," in Proc. Int. Workshop Immersive Telepresence, Jan. 2002, pp. 1-4.
19. I. Feldmann, O. Schreer, P. Kauff, R. Scḧafer, ̀ Z. Fei, H. J. W. Belt, and O. Divorra Escoda, " Immersive multi-user 3D video communication," in Proc. Int. Broadcast Conf, Amsterdam, The Netherlands, Sep. 2009.
20. D. Florencio and C. Zhang, " Multiview video compression and streaming based on predicted viewer position," in Proc. IEEE Int. Conf. Acoust. Speech Signal Process., Taipei, Taiwan, Apr. 2009, pp. 657-660.
21. B. Wilburn, N. Joshi, V. Vaish, E.-V. Talvala, E. Antunez, A. Barth, A. Adams, M. Horowitz, and M. Levoy, " High performance imaging using large camera arrays," ACM Trans. Graph., vol. 24, no. 3, pp. 765-776, Jul. 2005.
22. R. Raskar and J. Tumblin, Computational Photography: Mastering New Techniques for Lenses, Lighting, and Sensors. London, U.K.: A K Peters, 2010978-1-56881-313-4.
23. MPEG Video Sub-Group Chair (J.-R. Ohm), Submissions received in CfP on multiview video coding, Bangkok, Thailand, ISO/IEC JTC 1/SC 29/WG 11 (MPEG) Doc. M12969, Jan. 2006.
24. MPEG Video and Test Sub-Groups, Subjective test results for the CfP on multi-view video coding, Bangkok, Thailand, ISO/IEC JTC 1/SC 29/WG 11 (MPEG) Doc. N7799, Jan. 2006.
25. K. Müller, P. Merkle, A. Smolic, and T. Wiegand, Multiview coding using AVC, Bangkok, Thailand, ISO/IEC JTC 1/SC 29/WG 11 (MPEG) Doc. M12945, Jan. 2006.
26. E. Martinian, S. Yea, and A. Vetro, Results of core experiment 1B on multiview coding, Montreux, Switzerland, ISO/IEC JTC 1/SC 29/WG 11 (MPEG) Doc. M13122, Apr. 2006.
27. E. Martinian, A. Behrens, J. Xin, A. Vetro, and H. Sun, " Extensions of H.264/AVC for multiview video compression," in Proc. IEEE Int. Conf. Image Process., Atlanta, GA, Oct. 2006, pp. 2981-2984.
28. MPEG Video Sub-Group, Technologies under study for reference picture management and high-level syntax for multiview video coding, Montreux, Switzerland, ISO/IEC JTC 1/SC 29/WG 11 (MPEG) Doc. N8018, Apr. 2006.
29. Y.-K. Wang, Y. Chen, and M. M. Hannuksela, Time-first coding for multi-view video coding, Hangzhou, China, Joint Video Team (JVT) Doc. JVT-U104, Oct. 2006.
30. A. Vetro, Y. Su, H. Kimata, and A. Smolic, Joint multiview video model 2.0, Hangzhou, China, Joint Video Team (JVT) Doc. JVT-U207, Oct. 2006.
31. Y. L. Lee, J. H. Hur, Y. K. Lee, K. H. Han, S. H. Cho, N. H. Hur, J. W. Kim, J. H. Kim, P. Lai, A. Ortega, Y. Su, P. Yin, and C. Gomila, CE11: Illumination compensation, Hangzhou, China, Joint Video Team (JVT) Doc. JVT-U052, 2006.
32. J. H. Hur, S. Cho, and Y. L. Lee, BAdaptive local illumination change compensation method for H.264/AVC-based multiview video coding," IEEE Trans. Circuits Syst. Video Technol., vol. 17, no. 11, pp. 1496-1505, Nov. 2007.
33. P. Lai, A. Ortega, P. Pandit, P. Yin, and C. Gomila, Adaptive reference filtering for MVC, San Jose, CA, Joint Video Team (JVT) Doc. JVT-W065, Apr. 2007.
34. P. Lai, A. Ortega, P. Pandit, P. Yin, and C. Gomila, " Focus mismatches in multiview systems and efficient adaptive reference filtering for multiview video coding," in Proc. SPIE Conf. Vis. Commun. Image Process., San Jose, CA, Jan. 2008, DOI: 10.1117/12. 769215.
35. H. S. Koo, Y. J. Jeon, and B. M. Jeon, MVC motion skip mode, San Jose, CA, Joint Video Team (JVT) Doc. JVT-W081, Apr. 2007.
36. H. S. Koo, Y. J. Jeon, and B. M. Jeon, " Motion information inferring scheme for multi-view video coding," IEICE Trans. Commun., pp. 1247-1250, 2008, E91-B(4).
37. E. Martinian, A. Behrens, J. Xin, and A. Vetro, BView synthesis for multiview video compression," in Proc. Picture Coding Symp., " eijing, China, 2006.
38. S. Yea and A. Vetro, BView synthesis prediction for multiview video coding," Image Commun., vol. 24, no. 1-2, pp. 89-100, Jan. 2009.
39. M. Kitahara, H. Kimata, S. Shimizu, K. Kamikura, Y. Yashima, K. Yamamoto, T. Yendo, T. Fujii, and M. Tanimoto, " Multi-view video coding using view interpolation and reference picture selection," in Proc. IEEE Int. Conf. Multimedia Expo, Toronto, ON, Canada, Jul. 2006, pp. 97-100. (Pubitemid 46679663)
40. D. Tian, P. Pandit, P. Yin, and C. Gomila, Study of MVC coding tools, Shenzhen, China, Joint Video Team (JVT) Doc. JVT-Y044, Oct. 2007.
41. T. Wiegand, G. J. Sullivan, G. Bjøntegaard, and A. Luthra, " Overview of the H.264/AVC video coding standard," IEEE Trans. Circuits Syst. Video Technol., vol. 13, no. 7, pp. 560-576, Jul. 2003.
42. G. J. Sullivan and T. Wiegand, " Video compressionVFrom concepts to the H.264/AVC standard," Proc. IEEE, vol. 93, no. 1, pp. 18-31, Jan. 2005. (Pubitemid 40851238)
43. D. Marpe, T. Wiegand, and G. J. Sullivan, " The H.264/MPEG4 advanced video coding standard and its applications," IEEE Commun. Mag., vol. 44, no. 8, pp. 134-144, Aug. 2006.
44. Y. Chen, Y.-K. Wang, K. Ugur, M. M. Hannuksela, J. Lainema, and M. Gabbouj, B3D video services with the emerging MVC standard,[ EURASIP J. Adv. Signal Process., vol. 2009, 2009, DOI: 10.1155/2009/786015.
45. H. Schwarz, D. Marpe, and T. Wiegand, " Overview of the scalable video coding extension of the H.264/AVC standard," IEEE Trans. Circuits Syst. Video Technol., vol. 17, Special Issue on Scalable Video Coding, no. 9, pp. 1103-1120, Sep. 2007. (Pubitemid 47483052)
46. Y. Chen, P. Pandit, S. Yea, and C. S. Lim, Draftreference software for MVC (JMVC 6.0), London, U.K., Joint Video Team (JVT) Doc. JVT-AE207, Jul. 2009.
47. P. Merkle, A. Smolic, K. Mueller, and T. Wiegand, " Efficient prediction structures for multiview video coding," IEEE Trans. Circuits Syst. Video Technol., vol. 17, no. 11, pp. 1461-1473, Nov. 2007. (Pubitemid 350025508)
48. Y. Su, A. Vetro, and A. Smolic, Common test conditions for multiview video coding, Hangzhou, China, Joint Video Team (JVT) Doc. JVT-U211, Oct. 2006.
49. G. Bjøntegaard, Calculation of average PSNR differences between RD-curves, Austin, TX, ITU-T SG16/Q.6, Doc. VCEG-M033, Apr. 2001
50. T. Chen, Y. Kashiwagi, C. S. Lim, and T. Nishi, Coding performance of stereo high profile for movie sequences, London, U.K., Joint Video Team (JVT) Doc. JVT-AE022, Jul. 2009.
51. M. Droese and C. Clemens, Results of CE1-D on multiview video coding, Montreux, Switzerland, ISO/IEC JTC 1/SC 29/WG 11 (MPEG) Doc. M13247, Apr. 2006.
52. L. Stelmach, W. J. Tam, D. Meegan, and A. Vincent, " Stereo image quality: Effects of mixed spatio-temporal resolution," IEEE Trans. Circuits Syst. Video Technol., vol. 10, no. 2, pp. 188-193, Mar. 2000.
53. C. Fehn, P. Kauff, S. Cho, H. Kwon, N. Hur, and J. Kim, " Asymmetric coding of stereoscopic video for transmission over T-DMB," Proc. 3DTV-CON, Kos, Greece, pp. 1-4, 2007.
54. H. Brust, A. Smolic, K. Müller, G. Tech, and T. Wiegand, BMixed resolution coding of stereoscopic video for mobile devices," in Proc. 3DTV-CON, Potsdam, Germany, May 2009, pp. 1-4.
55. Dolby Laboratories, Dolby open specification for frame-compatible 3D systems, Apr. 2010. [Online]. Available: http://www.dolby.com/uploadedFiles/ Assets/US/Doc/Professional/3DFrameCompatibleOpenStandard.pdf.
56. HDMI Founders, HDMI specification, Mar. 2010. [Online]. Available: http://www. hdmi.org/manufacturer/specification.aspx.
57. D. K. Broberg, " Infrastructures for Home Delivery, Interfacing, Captioning, and Viewing of 3-D Content," Proc. IEEE, vol. 99, no. 4, Apr. 2011, DOI: 10.1109/JPROC.2010. 2092390.
58. Cable Television Laboratories, Content Encoding Profiles 3.0 Specification OC-SP-CEP3.0-I01-100827, Aug. 2010. [Online]. Available: http://www.cablelabs. com/specifications/OC-SP-CEP3. 0-I01-100827.pdf.
59. G. J. Sullivan and W. Husak, A. Luthra for MPEG Requirements Sub-Group, Problem Statement for Scalable Resolution Enhancement of Frame-Compatible Stereoscopic 3D Video, ISO/IEC JTC 1/SC 29/WG 11 (MPEG) Doc. N11526, Geneva, Switzerland, Jul. 2010.
60. T. Schierl and S. Narasimhan, " Transport and storage systems for 3-D video using MPEG-2 systems, RTP, and ISO file formats," Proc. IEEE, vol. 99, no. 4, Apr. 2011, DOI: 10.1109/JPROC.2010.2091370.
61. K. Müller, P. Merkle, and T. Wiegand, B3-D video representation using depth maps,[ Proc. IEEE, vol. 99, no. 4, Apr. 2011, DOI: 10.1109/JPROC.2010.2091090.