Display-camera calibration using eye reflections and geometry constraints

Computer Vision and Image Understanding

Volumn 115, Issue 6, 2011, Pages 835-853

  Nitschke, Christian ^a   Nakazawa, Atsushi ^a,b   Takemura, Haruo ^a  

^a OSAKA UNIVERSITY   (Japan)

^b JAPAN SCIENCE AND TECHNOLOGY AGENCY   (Japan)

Author keywords

Calibration; Catadioptric imaging; Corneal reflection; Display camera system; Eye pose estimation; Human eye model; Image analysis; Visual system

Indexed keywords

CATADIOPTRIC IMAGING; CORNEAL REFLECTION; DISPLAY-CAMERA SYSTEM; EYE POSE ESTIMATION; HUMAN EYE MODEL; VISUAL SYSTEM;

CALIBRATION; CAMERAS; ESTIMATION; GEOMETRY; HUMAN COMPUTER INTERACTION; IMAGE ANALYSIS; KNOWLEDGE MANAGEMENT;

DISPLAY DEVICES;

EID: 79952851278     PISSN: 10773142     EISSN: 1090235X     Source Type: Journal    
DOI: 10.1016/j.cviu.2011.02.008     Document Type: Article

References (68)

1. A.F. Abate, M. Nappi, D. Riccio, and G. Sabatino 2D and 3D face recognition: a survey Pattern Recognit. Lett. 28 14 2007 1885 1906 (Pubitemid 47259204)
2. M. Backes, T. Chen, M. Dürmuth, H.P.A. Lensch, M. Welk, Tempest in a teapot: compromising reflections revisited, in: Proceedings of the IEEE Symposium on Security and Privacy (SP), 2009, pp. 315-327.
3. M. Backes, M. Dürmuth, D. Unruh, Compromising reflections - or - how to read LCD monitors around the corner, in: Proceedings of the IEEE Symposium on Security and Privacy (SP), 2008, pp. 158-169.
4. S. Baker, and S.K. Nayar A theory of single-viewpoint catadioptric image formation Int. J. Comput. Vis. 35 2 1999 175 196 (Pubitemid 32210850)
5. O. Bimber, G. Wetzstein, A. Emmerling, C. Nitschke, Enabling view-dependent stereoscopic projection in real environments, in: Proceedings of the IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR), 2005, pp. 14-23. (Pubitemid 44733987)
6. S. Bogan, G. Waring, O. Ibrahim, C. Drews, and L. Curtis Classification of normal corneal topography based on computer-assisted videokeratography Arch. Ophthalmol. 108 7 1990 945 949 (Pubitemid 20216011)
7. T. Bonfort, P. Sturm, P. Gargallo, General specular surface triangulation, in: Proceedings of the Asian Conference on Computer Vision (ACCV), 2006, pp. 872-881. (Pubitemid 43849478)
8. J.-Y. Bouguet, Camera Calibration Toolbox for Matlab, 2010. < http://www.vision.caltech.edu/bouguetj/calib-doc/ > (last accessed on 18.02.11).
9. K.W. Bowyer, K. Hollingsworth, and P.J. Flynn Image understanding for iris biometrics: a survey Comput. Vis. Image Underst. 110 2 2008 281 307
10. G. Bradski, and A. Kaehler Learning OpenCV first ed. 2008 O'Reilly Media, Inc. Sebastopol
11. J.J. Clark Photometric stereo using LCD displays Image Vision Comput. 28 4 2010 704 714
12. F.L. Coutinho, C.H. Morimoto, Free head motion eye gaze tracking using a single camera and multiple light sources, in: Proceedings of the Brazilian Symposium on Computer Graphics and Image Processing (SIBGRAPI), 2006, pp. 171-178. (Pubitemid 47515406)
13. F.L. Coutinho, C.H. Morimoto, A depth compensation method for cross-ratio based eye tracking, in: Proceedings of the ACM Symposium on Eye Tracking Research and Applications (ETRA), 2010, pp. 137-140.
14. J. Daugman How iris recognition works IEEE Trans. Circuits Syst. Video Technol. 14 1 2004 21 30
15. A.T. Duchowski A breadth-first survey of eye tracking applications Beh. Res. Methods Instr. Comput. 34 4 2002 455 470
16. A.T. Duchowski Eye Tracking Methodology: Theory and Practice second ed. 2007 Springer-Verlag New York
17. Y. Francken, T. Cuypers, T. Mertens, J. Gielis, P. Bekaert, High quality mesostructure acquisition using specularities, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008, pp. 1-7.
18. Y. Francken, C. Hermans, P. Bekaert, Screen-camera calibration using a spherical mirror, in: Proceedings of the IEEE Canadian Conference on Computer and Robot Vision (CRV), 2007, pp. 11-20.
19. Y. Francken, C. Hermans, P. Bekaert, Screen-camera calibration using gray codes, in: Proceedings of the IEEE Canadian Conference on Computer and Robot Vision (CRV), 2009, pp. 155-161.
20. Y. Francken, C. Hermans, T. Cuypers, P. Bekaert, Fast normal map acquisition using an LCD screen emitting gradient patterns, in: Proceedings of the IEEE Canadian Conference on Computer and Robot Vision (CRV), 2008, pp. 189-195.
21. N. Funk, Y.-H. Yang, Using a raster display for photometric stereo, in: Proceedings of the IEEE Canadian Conference on Computer and Robot Vision (CRV), 2007, pp. 201-207. (Pubitemid 47432471)
22. C. Geyer, and K. Daniilidis Catadioptric projective geometry Int. J. Comput. Vis. 45 3 2001 223 243
23. E.D. Guestrin, M. Eizenman, Remote point-of-gaze estimation requiring a single-point calibration for applications with infants, in: Proceedings of the ACM Symposium on Eye Tracking Research & Applications (ETRA), 2008, pp. 267-274.
24. M.A. Halstead, B.A. Barsky, S.A. Klein, R.B. Mandell, Reconstructing curved surfaces from specular reflection patterns using spline surface fitting of normals, in: Proceedings of the ACM SIGGRAPH, 1996, pp. 335-342.
25. R.I. Hammoud Passive Eye Monitoring: Algorithms Applications and Experiments 2008 Springer-Verlag Berlin/Heidelberg
26. D. Hansen, and Q. Ji In the eye of the beholder: a survey of models for eyes and gaze IEEE Trans. Pattern Anal. Mach. Intell. 32 3 2010 478 500
27. R. Hartley, and A. Zisserman Multiple View Geometry in Computer Vision second ed. 2003 Cambridge University Press New York
28. D.M. Hoffman, A.R. Girshick, K. Akeley, and M.S. Banks Vergence-accommodation conflicts hinder visual performance and cause visual fatigue J. Vis. 8 3 2008 1 30
29. I. Ihrke, K.N. Kutulakos, H.P.A. Lensch, M. Magnor, and W. Heidrich State of the art in transparent and specular object reconstruction STAR Proc. Eurogr. 2008 87 108
30. N. Iqbal, S.-Y.A. Lee, A study on human gaze estimation using screen reflection, in: Proceedings of the International Conference on Intelligent Data Engineering and Automated Learning (IDEAL), 2008, pp. 104-111.
31. D.R.A. Iskander A parametric approach to measuring limbus corneae from digital images IEEE Trans. Biomed. Eng. 53 6 2006 1134 1140
32. A. Jaimes, and N. Sebe Multimodal human-computer interaction: a survey Comput. Vis. Image Underst. 108 1-2 2007 116 134 (Pubitemid 47333395)
33. M.K. Johnson, H. Farid, Exposing digital forgeries through specular highlights on the eye, in: Proceedings of the Information Hiding: International Workshop (IH), 2007, pp. 311-325.
34. J.J. Kang, M. Eizenman, E.D. Guestrin, and E. Eizenman Investigation of the cross-ratios method for point-of-gaze estimation IEEE Trans. Biomed. Eng. 55 9 2008 2293 2302
35. P.L. Kaufman, A. Alm, Adler's Physiology of the Eye: Clinical Application, 10th ed., Mosby, St. Louis, 2003.
36. R.K. Kumar, A. Ilie, J.-M. Frahm, M. Pollefeys, Simple calibration of non-overlapping cameras with a mirror, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008, pp. 1-7.
37. Y.J. Ko, E.C. Lee, and K.R. Park A robust gaze detection method by compensating for facial movements based on corneal specularities Pattern Recognit. Lett. 29 10 2008 1474 1485
38. S. Kuthirummal, S.K. Nayar, Multiview radial catadioptric imaging for scene capture, in: Proceedings of the ACM SIGGRAPH, 2006, pp. 916-923. (Pubitemid 44481053)
39. K.N. Kutulakos, and E. Steger A theory of refractive and specular 3D shape by light-path triangulation Int. J. Comput. Vis. 76 1 2008 13 29 (Pubitemid 350234085)
40. M. Lambooij, W. IJsselsteijn, M. Fortuin, and I. Heynderickx Visual discomfort and visual fatigue of stereoscopic displays: a review J. Imag. Sci. Technol. 53 3 2009 53 030201-030201-14
41. R.B. Mandell A guide to videokeratography Int. Contact Lens Clin. 23 6 1996 205 228 (Pubitemid 126168504)
42. C.H. Morimoto, and M.R. Mimica Eye gaze tracking techniques for interactive applications Comput. Vis. Image Underst. 98 1 2005 4 24
43. N.J.W. Morris, K.N. Kutulakos, Reconstructing the surface of inhomogeneous transparent scenes by scatter-trace photography, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2007, pp. 1-8.
44. D. Nehab, T. Weyrich, S. Rusinkiewicz, Dense 3D reconstruction from specularity consistency, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008, pp. 1-7.
45. K. Nishino, P.N. Belhumeur, S.K. Nayar, Using eye reflections for face recognition under varying illumination, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2005, pp. 519-526. (Pubitemid 44055000)
46. K. Nishino, S.K. Nayar, Eyes for relighting, in: Proceedings of the ACM SIGGRAPH, 2004, pp. 704-711.
47. K. Nishino, and S.K. Nayar Corneal imaging system: environment from eyes Int. J. Comput. Vis. 70 1 2006 23 40 (Pubitemid 43855865)
48. C. Nitschke, A. Nakazawa, H. Takemura, Display-camera calibration from eye reflections, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2009, pp. 1226-1232.
49. C. Nitschke, A. Nakazawa, H. Takemura, Eye reflection analysis and application to display-camera calibration, in: Proceedings of the IEEE International Conference on Image Processing (ICIP), 2009, pp. 3449-3452.
50. M. Porta Vision-based user interfaces: methods and applications Int. J. Hum. Comput. Stud. 57 1 2002 27 73
51. W.H. Press, W.T. Vetterling, S.A. Teukolsky, and B.P. Flannery Numerical Recipes in C++: The Art of Scientific Computing second ed. 2002 Cambridge University Press New York
52. G. Schindler, Photometric stereo via computer screen lighting for real-time surface reconstruction, in: Proceedings of the International Symposium on 3D Data Processing, Visualization and Transmission (3DPVT), 2008.
53. D. Schnieders, X. Fu, K.-Y.K. Wong, Reconstruction of display and eyes from a single image, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010, pp. 1442-1449.
54. R.S. Snell, and M.A. Lemp Clinical Anatomy of the Eye second ed. 1998 Blackwell Publishing Malden
55. P. Sturm, T. Bonfort, How to compute the pose of an object without a direct view? in: Proceedings of the Asian Conference on Computer Vision (ACCV), 2006, pp. 21-31. (Pubitemid 43849394)
56. B. Sun, and R. Ramamoorthi Affine double- and triple-product wavelet integrals for rendering ACM Trans. Graph. 28 2 2009 1 17
57. R. Swaminathan, M.D. Grossberg, and S.K. Nayar Non-single viewpoint catadioptric cameras: geometry and analysis Int. J. Comput. Vis. 66 3 2006 211 229
58. M. Tarini, H.P. Lensch, M. Goesele, and H. Seidel 3D acquisition of mirroring objects using striped patterns Graph. Models 67 4 2005 233 259 (Pubitemid 40684005)
59. N. Tsumura, M.N. Dang, T. Makino, Y. Miyake, Estimating the directions to light sources using images of eye for reconstructing 3D human face, in: Proceedings of the IS&T/SID's Color Imaging Conference, Color Science, Systems and Application, 2003, pp. 77-81.
60. M. Turk Computer vision in the interface Commun. ACM 47 1 2004 60 67
61. A. Villanueva, and R. Cabeza Models for gaze tracking systems J. Image Video Process. 4 2007 1 16 2007
62. H. Wang, S. Lin, X. Ye, and W. Gu Separating corneal reflections for illumination estimation Neurocomputing 71 10-12 2008 1788 1797
63. J.-G. Wang, E. Sung, R. Venkateswarlu, Eye gaze estimation from a single image of one eye, in: Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2003, pp. 136-143.
64. R.J. Woodham Photometric method for determining surface orientation from multiple images Opt. Eng. 19 1 1980 139 144
65. D.H. Yoo, and M.J. Chung A novel non-intrusive eye gaze estimation using cross-ratio under large head motion Comput. Vis. Image Underst. 98 1 2005 25 51
66. L. Young, and D. Sheena Survey of eye movement recording methods Behav. Res. Methods Instrum. 7 5 1975 397 429
67. W. Zhao, R. Chellappa, P.J. Phillips, and A. Rosenfeld Face recognition: a literature survey ACM Comput. Surv. 35 4 2003 399 458
68. D.E. Zongker, D.M. Werner, B. Curless, D.H. Salesin, Environment matting and compositing, in: Proceedings of the ACM SIGGRAPH, (1999) 205-214.