A Survey of Calibration Methods for Optical See-Through Head-Mounted Displays

IEEE Transactions on Visualization and Computer Graphics

Volumn 24, Issue 9, 2018, Pages 2649-2662

  Grubert, Jens ^a   Itoh, Yuta ^b   Moser, Kenneth ^c   Swan, J Edward ^d  

^a University of Coburg   (Germany)

^b KEIO UNIVERSITY   (Japan)

^c Marxent Labs LLC   (United States)

^d MISSISSIPPI STATE UNIVERSITY   (United States)

Author keywords

Augmented reality; head mounted displays; optical see through calibration

Indexed keywords

AUGMENTED REALITY; CALIBRATION; CAMERAS; EMBEDDED SYSTEMS; GLASS; MATHEMATICAL MODELS; RENDERING (COMPUTER GRAPHICS); SURVEYS; THREE DIMENSIONAL COMPUTER GRAPHICS; THREE DIMENSIONAL DISPLAYS; TRACKING (POSITION); VIRTUAL REALITY;

AUGMENTED AND VIRTUAL REALITIES; CALIBRATION TECHNIQUES; COMPUTER GENERATED GRAPHICS; HEAD MOUNTED DISPLAYS; OPTICAL SEE-THROUGH; OPTICAL SEE-THROUGH HEAD-MOUNTED DISPLAYS; RESISTS; VIRTUAL-REALITY HEADSETS;

HELMET MOUNTED DISPLAYS;

EID: 85030697919     PISSN: 10772626     EISSN: None     Source Type: Journal    
DOI: 10.1109/TVCG.2017.2754257     Document Type: Article

References (84)

1. Y. Abdel-Aziz, "Direct linear transformation from comparator coordinates in close-range photogrammetry," in Proc. ASP Symp. Close-Range Photogrammetry Illinois, 1971, pp. 1-19.
2. M. Axholt, "Pinhole camera calibration in the presence of human noise," Ph.D. dissertation, Dept. Sci. Technol., Inst. Technol., Link-öping Univ., Linköping, Sweden, 2011.
3. M. Axholt, S. Peterson, and S. R. Ellis, "User boresight calibration precision for large-format head-up displays," in Proc. ACM Symp. Virtual Reality Softw. Technol., Oct. 2008, pp. 141-148.
4. M. Axholt, et al., "Optical see-through head mounted display direct linear transformation calibration robustness in the presence of user alignment noise," in Proc. Human Factors Ergonom. Soc. Annu. Meet., 2010, pp. 2427-2431.
5. M. Axholt, M.A. Skoglund, S. D.O'Connell, M. D. Cooper, S. R. Ellis, and A. Ynnerman, "Parameter estimation variance of the single point active alignment method in optical see-through head mounted display calibration," in Proc. IEEEVirtual Reality Conf., 2011, pp. 27-34.
6. E. Azimi, L. Qian, P. Kazanzides, and N. Navab, "Robust optical see-through head-mounted display calibration: Taking anisotropic nature of user interaction errors into account," in Proc. IEEE Virtual Reality Conf., Mar. 2017, pp. 219-220.
7. R. Azuma and G. Bishop, "Improving static and dynamic registration in an optical see-through HMD," in Proc. Annu. Conf. Comput. Graph. Interactive Techn., 1994, pp. 197-204.
8. M. Billinghurst, A. Clark, and G. Lee, "A survey of augmented reality," Found. Trends Human-Comput. Interaction, vol. 8, pp. 73-272, 2014.
9. T. P. Caudell and D. W. Mizell, "Augmented reality: An application of heads-up display technology to manual manufacturing processes," in Proc. Hawaii Int. Conf. Syst. Sci., 1992, pp. 659-669.
10. D. Dunn, et al., "Wide field of view varifocal near-eye display using see-through deformable membrane mirrors," IEEE Trans. Vis. Comput. Graph., vol. 23, no. 4, pp. 1322-1331, Apr. 2017.
11. P. J. Edwards, et al., "Design and evaluation of a system for microscope-assisted guided interventions (MAGI)," IEEE Trans. Med. Imag., vol. 19, no. 11, pp. 1082-1093, Nov. 2000.
12. O. Faugeras, Three-Dimensional Computer Vision: A Geometric Viewpoint. Cambridge, MA, USA: MIT Press, 1993.
13. M. Figl, et al., "A fully automated calibration method for an optical see-through head-mounted operating microscope with variable zoom and focus," IEEE Trans. Med. Imag., vol. 24, no. 11, pp. 1492-1499, Nov. 2005.
14. A. Fuhrmann, D. Schmalstieg, and W. Purgathofer, "Fast calibration for augmented reality," in Proc. ACM Symp. Virtual Reality Softw. Technol., 1999, pp. 166-167.
15. C. Gao, H. Hua, and N. Ahuja, "Easy calibration of a headmounted projective display for augmented reality systems," in Proc. IEEE Virtual Reality Conf., 2003, pp. 53-60.
16. Y. Genc, F. Sauer, F. Wenzel, M. Tuceryan, and N. Navab, "Optical see-through HMD calibration: A stereo method validated with a video see-through system," in Proc. IEEE ACM Int. Symp. Augmented Reality, 2000, pp. 165-174.
17. Y. Genc, M. Tuceryan, A. Khamene, and N. Navab, "Optical seethrough calibration with vision-based trackers: Propagation of projection matrices," in Proc. IEEE ACM Int. Symp. Augmented Reality, 2001, pp. 147-156.
18. Y. Genc, M. Tuceryan, and N. Navab, "Practical solutions for calibration of optical see-through devices," in Proc. Int. Symp. Mixed Augmented Reality, 2002, Art. no. 169.
19. S. J. Gilson, A. W. Fitzgibbon, and A. Glennerster, "Spatial calibration of an optical see-through head-mounted display," J. Neurosci. Methods, vol. 173, no. 1, pp. 140-146, 2008.
20. J. Grubert, et al., "Extended investigations of user-related issues in mobile industrial AR," in Proc. 9th IEEE Int. Symp. Mixed Augmented Reality, 2010, pp. 229-230.
21. J. Grubert, J. Tuemle, R. Mecke, and M. Schenk, "Comparative user study of two see-through calibration methods," in Proc. IEEE Virtual Reality Conf., 2010, pp. 269-270.
22. J. Grubert, J. Tümler, and R. Mecke, "Untersuchungen zur optimierung der see-through-kalibrierung fuer mobile augmented reality assistenzsysteme," Michael Schenk, Hrsg, vol. 6, no. 7, pp. 64-67, 2008.
23. S. G. Hart and L. E. Staveland, "Development of NASA-TLX (task load index): Results of empirical and theoretical research," Advances Psychology, vol. 52, pp. 139-183, 1988.
24. R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision. Cambridge, U.K.: Cambridge Univ. Press, 2003.
25. R. Hartley and A. Zisserman, "Multiple view geometry in computer vision," Robotica, vol. 23, no. 2, pp. 271-271, 2005.
26. R. Horaud and F. Dornaika, "Hand-eye calibration," Int. J. Robot. Res., vol. 14, no. 3, pp. 195-210, 1995.
27. X. Hu and H. Hua, "Design and assessment of a depth-fused multi-focal-plane display prototype," J. Display Technol., vol. 10, no. 4, pp. 308-316, 2014.
28. X. Hu and H. Hua, "High-resolution optical see-through multifocal-plane head-mounted display using freeform optics," Opt. Exp., vol. 22, no. 11, pp. 13 896-13 903, 2014.
29. H. Hua, C. Gao, and N. Ahuja, "Calibration of a head-mounted projective display for augmented reality systems," in Proc. Int. Symp. Mixed Augmented Reality, 2002, pp. 176-185.
30. H. Hua, X. Hu, and C. Gao, "A high-resolution optical seethrough head-mounted display with eyetracking capability," Opt. Exp., vol. 21, no. 25, pp. 30 993-30 998, 2013.
31. J. F. Hughes, A. Van Dam, J. D. Foley, and S. K. Feiner, Computer Graphics: Principles and Practice. London, U.K.: Pearson Education, 2013.
32. A. D. Hwang and E. Peli, "An augmented-reality edge enhancement application for Google glass," Optometry Vis. Sci.: Official Publication Amer. Academy Optometry, vol. 91, no. 8, 2014, Art. no. 1021.
33. Y. Itoh, T. Amano, D. Iwai, and G. Klinker, "Gaussian light field: Estimation of viewpoint-dependent blur for optical see-through head-mounted displays," IEEE Trans. Vis. Comput. Graph., vol. 22, no. 11, pp. 2368-2376, Nov. 2016.
34. Y. Itoh, M. Dzitsiuk, T. Amano, and G. Klinker, "Semi-parametric color reproduction method for optical see-through head-mounted displays," IEEE Trans. Vis. Comput. Graph., vol. 21, no. 11, pp. 1269-1278, Nov. 2015.
35. Y. Itoh and G. Klinker, "Interaction-free calibration for optical seethrough head-mounted displays based on 3D eye localization," in Proc. IEEE Symp. 3D User Interfaces, 2014, pp. 75-82.
36. Y. Itoh and G. Klinker, "Performance and sensitivity analysis of INDICA: Interaction-free display calibration for optical seethrough head-mounted displays," in Proc. Int. Symp. Mixed Augmented Reality, 2014, pp. 171-176.
37. Y. Itoh and G. Klinker, "Light-field correction for spatial calibration of optical see-through head-mounted displays," IEEE Trans. Vis. Comput. Graph., vol. 21, no. 4, pp. 471-480, Apr. 2015.
38. Y. Itoh and G. Klinker, "Simultaneous direct and augmented view distortion calibration of optical see-through head-mounted displays," in Proc. Int. Symp. Mixed Augmented Reality, 2015, pp. 43-48.
39. Y. Itoh and G. Klinker, "Vision enhancement: Defocus correction via optical see-through head-mounted displays," in Proc. 6th Augmented Human Int. Conf., 2015, pp. 1-8.
40. A. L. Janin, D. W. Mizell, and T. P. Caudell, "Calibration of headmounted displays for augmented reality applications," in Proc. IEEE Virtual Reality Annu. Int. Symp., 1993, pp. 246-255.
41. J. A. Jones, D. Edewaard, R. A. Tyrrell, and L. F. Hodges, "A schematic eye for virtual environments," in Proc. IEEE Symp. 3D User Interfaces, 2016, pp. 221-230.
42. H. Jun and G. Kim, "A calibration method for optical see-through head-mounted displays with a depth camera," in Proc. IEEE Virtual Reality, 2016, pp. 103-111.
43. H. Kato and M. Billinghurst, "Marker tracking and HMD calibration for a video-based augmented reality conferencing system," in Proc. IEEE ACM Int. Workshop Augmented Reality, Jun. 1999, pp. 85-94.
44. F. Kellner, et al., "Geometric calibration of head-mounted displays and its effects on distance estimation," IEEE Trans. Vis. Comput. Graph., vol. 18, no. 4, pp. 589-596, Apr. 2012.
45. R. S. Kennedy, N. E. Lane, K. S. Berbaum, and M. G. Lilienthal, "Simulator sickness questionnaire: An enhanced method for quantifying simulator sickness," Int. J. Aviation Psychology, vol. 3, no. 3, pp. 203-220, 1993.
46. G. Klinker, D. Stricker, and D. Reiners, "Augmented reality: A balance act between high quality and real-time constraints," Mixed Reality-Merging Real Virtual Worlds, pp. 325-346, 1999.
47. T. Langlotz, M. Cook, and H. Regenbrecht, "Real-time radiometric compensation for optical see-through head-mounted displays," IEEE Trans. Vis. Comput. Graph., vol. 22, no. 11, pp. 2385-2394, Nov. 2016.
48. S. Lee and H. Hua, "A robust camera-based method for optical distortion calibration of head-mounted displays," in Proc. IEEE Virtual Reality, 2013, pp. 27-30.
49. P. Lincoln, et al., "From motion to photons in 80 microseconds: Towards minimal latency for virtual and augmented reality," IEEE Trans. Vis. Comput. Graph., vol. 22, no. 4, pp. 1367-1376, Apr. 2016.
50. S. Liu, H. Hua, and D. Cheng, "A novel prototype for an optical see-through head-mounted display with addressable focus cues," IEEE Trans. Vis. Comput. Graph., vol. 16, no. 3, pp. 381-393, May/Jun. 2010.
51. G. Luo, N. Rensing, E. Weststrate, and E. Peli, "Registration of an on-axis see-through head-mounted display and camera system," Opt. Eng., vol. 44, no. 2, pp. 024 002-024 002, 2005.
52. Y. Ma, S. Soatto, J. Kosecka, and S. S. Sastry, An Invitation to 3-D Vision: From Images to Geometric Models. Berlin, Germany: Springer, 2012.
53. P. Maier, A. Dey, C. A. Waechter, C. Sandor, M. Tönnis, and G. Klinker, "An empiric evaluation of confirmation methods for optical see-through head-mounted display calibration," in Proc. Int. Symp. Mixed Augmented Reality, 2011, pp. 267-268.
54. A. Maimone and H. Fuchs, "Computational augmented reality eyeglasses," in Proc. Int. Symp. Mixed Augmented Reality, 2013, pp. 29-38.
55. A. Maimone, A. Georgiou, and J. S. Kollin, "Holographic near-eye displays for virtual and augmented reality," ACM Trans. Graph., vol. 36, no. 4, pp. 85:1-85:16, Jul. 2017. [Online]. Available: http://doi.acm.org/10.1145/3072959.3073624
56. N. Makibuchi, H. Kato, and A. Yoneyama, "Vision-based robust calibration for optical see-through head-mounted displays," in Proc. IEEE Int. Conf. Image Process., 2013, pp. 2177-2181.
57. N. Matsuda, A. Fix, and D. Lanman, "Focal surface displays," ACM Trans. Graph., vol. 36, no. 4, pp. 86:1-86:14, Jul. 2017. [Online]. Available: http://doi.acm.org/10.1145/3072959.3073590
58. E. McGarrity, Y. Genc, M. Tuceryan, C. Owen, and N. Navab, "A new system for online quantitative evaluation of optical seethrough augmentation," in Proc. IEEE ACM Int. Symp. Augmented Reality, 2001, pp. 157-166.
59. E. McGarrity, M. Tuceryan, C. Owen, Y. Genc, and N. Navab, "Evaluation of optical see-through systems," in Proc. Int. Conf. Augmented Virtual Environ. 3D Imag., 2001, pp. 18-21.
60. J. J. Moré, "The levenberg-marquardt algorithm: Implementation and theory," in Numerical Analysis. Berlin, Germany: Springer, 1978, pp. 105-116.
61. K. Moser, Y. Itoh, K. Oshima, J. E. Swan, G. Klinker, and C. Sandor, "Subjective evaluation of a semi-automatic optical see-through head-mounted display calibration technique," IEEE Trans. Vis. Comput. Graph., vol. 21, no. 4, pp. 491-500, Apr. 2015.
62. K. R. Moser, M. Axholt, and J. E. Swan, "Baseline SPAAM calibration accuracy and precision in the absence of human postural sway error," in Proc. IEEE Virtual Reality, 2014, pp. 99-100.
63. K. R. Moser and J. E. Swan, "Evaluation of user-centric optical seethrough head-mounted display calibration using a leap motion controller," in Proc. IEEE Symp. 3D User Interfaces, 2016, pp. 159-167.
64. N. Navab, S. Zokai, Y. Genc, and E. M. Coelho, "An on-line evaluation system for optical see-through augmented reality," in Proc. IEEE Virtual Reality, 2004, pp. 245-246.
65. T. Oishi and S. Tachi, "Methods to calibrate projection transformation parameters for see-through head-mounted displays," Presence: Teleoperators Virtual Environ., vol. 5, no. 1, pp. 122-135, 1996.
66. M. O'Loughlin and C. Sandor, "User-centric calibration for optical see-through augmented reality," Honours Thesis, Division of Information Technology, Engineering and the Environment, University of South Australia, 2013.
67. C. B. Owen, J. Zhou, A. Tang, and F. Xiao, "Display-relative calibration for optical see-through head-mounted displays," in Proc. Int. Symp. Mixed Augmented Reality, 2004, pp. 70-78.
68. A. Plopski, Y. Itoh, C. Nitschke, K. Kiyokawa, G. Klinker, and H. Takemura, "Corneal-imaging calibration for optical seethrough head-mounted displays," IEEE Trans. Vis. Comput. Graph., vol. 21, no. 4, pp. 481-490, Apr. 2015.
69. A. Plopski, J. Orlosky, Y. Itoh, C. Nitschke, K. Kiyokawa, and G. Klinker, "Automated spatial calibration of HMD systems with unconstrained eye-cameras," in Proc. Int. Symp. Mixed Augmented Reality, 2016, pp. 94-99.
70. J. Ponce, et al., "Computer vision: A modern approach," Comput., vol. 16, no. 11, 2011.
71. L. Priese, F. Schmitt, and P. Lemke, "Automatische see-through kalibrierung," Fachbereich Informatik, Universität Koblenz Landau, Koblenz, Germany, Tech. Rep. 7/2007, 2007.
72. L. Qian, A. Winkler, B. Fuerst, P. Kazanzides, and N. Navab, "Modeling physical structure as additional constraints for stereoscopic optical see-through head-mounted display calibration," in Proc. IEEE Int. Symp. Mixed Augmented Reality, Sep. 2016, pp. 154-155.
73. L. Qian, A. Winkler, B. Fuerst, P. Kazanzides, and N. Navab, "Reduction of interaction space in single point active alignment method for optical see-through head-mounted display calibration," in Proc. IEEE Int. Symp. Mixed Augmented Reality, Sep. 2016, pp. 156-157.
74. W. Robinett and R. Holloway, "The visual display transformation for virtual reality," Presence: Teleoperators Virtual Environ., vol. 4, no. 1, pp. 1-23, 1995.
75. I. E. Sutherland, "Three-dimensional data input by tablet," Proc. IEEE, vol. 62, no. 4, pp. 453-461, Apr. 1974.
76. A. Tang, J. Zhou, and C. Owen, "Evaluation of calibration procedures for optical see-through head-mounted displays," in Proc. Int. Symp. Mixed Augmented Reality, 2003, Art. no. 161.
77. E. Trucco and A. Verri, Introductory Techniques for 3-D Computer Vision. Englewood Cliffs, NJ, USA: Prentice Hall, 1998.
78. R. Tsai, "A versatile camera calibration technique for highaccuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses," IEEE J. Robot. Autom., vol. 3, no. 4, pp. 323-344, Aug. 1987.
79. M. Tuceryan, Y. Genc, and N. Navab, "Single-point active alignment method (SPAAM) for optical see-through HMD calibration for augmented reality," Presence: Teleoperators Virtual Environ., vol. 11, no. 3, pp. 259-276, 2002.
80. M. Tuceryan and N. Navab, "Single point active alignment method (SPAAM) for optical see-through HMD calibration for AR," in Proc. Int. Symp. Mixed Augmented Reality, 2000, pp. 149-158.
81. J. Vince, Mathematics for Computer Graphics. Berlin, Germany: Springer, 2013.
82. Z. Zhang, D. Weng, J. Guo, Y. Liu, and Y. Wang, "An accurate calibration method for optical see-through head-mounted displays based on actual eye-observation model," in Proc. Int. Symp. Mixed Augmented Reality, 2017, pp. 245-250.
83. Z. Zhang, D. Weng, Y. Liu, Y. Wang, and X. Zhao, "RIDE: Regioninduced data enhancement method for dynamic calibration of optical see-through head-mounted displays," in Proc. IEEE Virtual Reality, 2017, pp. 245-246.
84. Z. Zhang, D. Weng, Y. Liu, and L. Xiang, "3D optical see-through head-mounted display based augmented reality system and its application," in Proc. Int. Conf. Opt. Photon. Eng., 2015, pp. 952 428-952 428.