Efficient and precise interactive hand tracking through joint, continuous optimization of pose and correspondences

ACM Transactions on Graphics

Volumn 35, Issue 4, 2016, Pages

  Taylor, Jonathan ^a   Bordeaux, Lucas ^a   Cashman, Thomas ^a   Corish, Bob ^a   Keskin, Cem ^a   Sharp, Toby ^a   Soto, Eduardo ^b   Sweeney, David ^a   Valentin, Julien ^c   Luff, Benjamin ^d   Topalian, Arran ^d   Wood, Erroll ^e   Khamis, Sameh ^a   Kohli, Pushmeet ^a   Izadi, Shahram ^a   Banks, Richard ^a   Fitzgibbon, Andrew ^a   Shotton, Jamie ^a  

^a MICROSOFT RESEARCH   (United States)

^b MCMASTER UNIVERSITY   (Canada)

^c UNIVERSITY OF OXFORD   (United Kingdom)

^d UNIVERSITY OF ABERTAY DUNDEE   (United Kingdom)

^e UNIVERSITY OF CAMBRIDGE   (United Kingdom)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; CAMERAS; INTERACTIVE COMPUTER GRAPHICS; LEARNING SYSTEMS; NONLINEAR PROGRAMMING; OPTIMIZATION;

ARTICULATED HAND TRACKING; CONTINUOUS OPTIMIZATION; JOINT OPTIMIZATION; NON-LINEAR OPTIMIZATION; NUMBER OF ITERATIONS; OBJECTIVE FUNCTIONS; QUANTITATIVE ASSESSMENTS; SMOOTH SURFACE MODELING;

PALMPRINT RECOGNITION;

EID: 84979965021     PISSN: 07300301     EISSN: 15577368     Source Type: Journal    
DOI: 10.1145/2897824.2925965     Document Type: Conference Paper

References (61)

1. 3GEAR SYSTEMS INC, 2013. Gesture recognizer. http:// threegear.com, Jan.
2. ATHITSOS, V., AND SCLAROFF, S. 2003. Estimating 3D hand pose from a cluttered image. In Proc. CVPR, vol. 2, II-432.
3. BALLAN, L., TANEJA, A., GALL, J., GOOL, L. V., AND POLLEFEYS, M. 2012. Motion capture of hands in action using discriminative salient points. In Proc. ECCV, 640-653.
4. BRAY, M., KOLLER-MEIER, E., AND VAN GOOL, L. 2004. Smart particle filtering for 3D hand tracking. In Proc. Automatic Face and Gesture Recognition, 675-680.
5. DE LA GORCE, M., FLEET, D. J., AND PARAGIOS, N. 2011. Model-Based 3D Hand Pose Estimation from Monocular Video. IEEE Trans. PAMI 33, 9, 1793-1805.
6. DIPIETRO, L., SABATINI, A. M., AND DARIO, P. 2008. A survey of glove-based systems and their applications. IEEE Trans. Systems, Man, and Cybernetics, Part C: Applications and Reviews 38, 4, 461-482.
7. EROL, A., BEBIS, G., NICOLESCU, M., BOYLE, R. D., AND TWOMBLY, X. 2007. Vision-based hand pose estimation: A review. CVIU 108, 1-2, 52-73.
8. FLEISHMAN, S., KLIGER, M., LERNER, A., AND KUTLIROFF, G. 2015. ICPIK: Inverse kinematics based articulated-ICP. In Proc. CVPR Workshops, 28-35.
9. GEMAN, S., AND MCCLURE, D. E. 1987. Statistical methods for tomographic image reconstruction. Bulletin of the International Statistical Institute 52, 4, 5-21.
10. GUZMA N-RIVERA, A., KOHLI, P., GLOCKER, B., SHOTTON, J., SHARP, T., FITZGIBBON, A. W., AND IZADI, S. 2014. Multi-output learning for camera relocalization. In Proc. CVPR, 1114-1121.
11. HEAP, T., AND HOGG, D. 1996. Towards 3D hand tracking using a deformable model. In Proc. Automatic Face and Gesture Recognition, 140-145.
12. INTEL CORPORATION, 2016. RealSense SDK. http://software. intel.com/realsense, Jan.
13. JACOBSON, A., DENG, Z., KAVAN, L., AND LEWIS, J. 2014. Skinning: Real-time shape deformation. In ACM SIGGRAPH 2014 Courses, #24.
14. KESKIN, C., KIRAÇ, F., KARA, Y. E., AND AKARUN, L. 2012. Hand pose estimation and hand shape classification using multilayered randomized decision forests. In Proc. ECCV, 852-863.
15. KHAMIS, S., TAYLOR, J., SHOTTON, J., KESKIN, C., IZADI, S., AND FITZGIBBON, A. 2015. Learning an efficient model of hand shape variation from depth images. In Proc. CVPR, 2540-2548.
16. KIM, D., HILLIGES, O., IZADI, S., BUTLER, A. D., CHEN, J., OIKONOMIDIS, I., AND OLIVIER, P. 2012. Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor. In Proc. UIST, 167-176.
17. KRUPKA, E., BAR HILLEL, A., KLEIN, B., VINNIKOV, A., FREEDMAN, D., AND STACHNIAK, S. 2014. Discriminative ferns ensemble for hand pose recognition. In Proc. CVPR, 3670-3677.
18. LEAP MOTION INC, 2013. Motion Controller. http://leapmotion.com/product, Jan.
19. LEAP MOTION INC, 2015. Orion. http://developer. leapmotion.com/orion, Feb.
20. LI, P., LING, H., LI, X., AND LIAO, C. 2015. 3D hand pose estimation using randomized decision forest with segmentation index points. In Proc. ICCV, 819-827
21. LOOP, C. T. 1987. Smooth Subdivision Surfaces Based on Triangles. Master's thesis, University of Utah.
22. LOPER, M., MAHMOOD, N., ROMERO, J., PONS-MOLL, G., AND BLACK, M. J. 2015. SMPL: a skinned multi-person linear model. ACM Trans. Graphics 34, 6, #248.
23. MAKRIS, A., KYRIAZIS, N., AND ARGYROS, A. 2015. Hierarchical particle filtering for 3D hand tracking. In Proc. CVPR Workshops, 8-17.
24. MELAX, S., KESELMAN, L., AND ORSTEN, S. 2013. Dynamics based 3D skeletal hand tracking. In Proceedings of the 2013 Graphics Interface Conference, 63-70.
25. MITCHELL, D. P. 1991. Spectrally optimal sampling for distribution ray tracing. In Proc. SIGGRAPH, 157-164.
26. MONNAI, Y., HASEGAWA, K., FUJIWARA, M., YOSHINO, K., INOUE, S., AND SHINODA, H. 2014. HaptoMime: Mid-air haptic interaction with a floating virtual screen. In Proc. UIST, 663-667.
27. NEVEROVA, N., WOLF, C., NEBOUT, F., AND TAYLOR, G. 2015. Hand pose estimation through weakly-supervised learning of a rich intermediate representation. arXiv preprint 1511.06728.
28. OBERWEGER, M., WOHLHART, P., AND LEPETIT, V. 2015. Training a feedback loop for hand pose estimation. In Proc. ICCV, 3316-3324.
29. OIKONOMIDIS, I., KYRIAZIS, N., AND ARGYROS, A. 2011. Efficient model-based 3D tracking of hand articulations using Kinect. In Proc. BMVC, 101.1-101.11.
30. POIER, G., RODITAKIS, K., SCHULTER, S., MICHEL, D., BISCHOF, H., AND ARGYROS, A. A. 2015. Hybrid one-shot 3D hand pose estimation by exploiting uncertainties. In Proc. BMVC, 182.1-182.14.
31. QIAN, C., SUN, X., WEI, Y., TANG, X., AND SUN, J. 2014. Realtime and robust hand tracking from depth. In Proc. CVPR, 1106-1113.
32. REHG, J. M., AND KANADE, T. 1994. Visual tracking of high DOF articulated structures: an application to human hand tracking. In Proc. ECCV, 35-46.
33. SHARP, T., KESKIN, C., ROBERTSON, D., TAYLOR, J., SHOTTON, J., KIM, D., RHEMANN, C., LEICHTER, I., VINNIKOV, A., WEI, Y., FREEDMAN, D., KOHLI, P., KRUPKA, E., FITZGIBBON, A., AND IZADI, S. 2015. Accurate, robust, and flexible realtime hand tracking. In Proc. CHI, 3633-3642.
34. SHOTTON, J., FITZGIBBON, A., COOK, M., SHARP, T., FINOCCHIO, M., MOORE, R., KIPMAN, A., AND BLAKE, A. 2011. Real-time human pose recognition in parts from a single depth image. In Proc. CVPR, 1297-1304.
35. SHOTTON, J., SHARP, T., KOHLI, P., NOWOZIN, S., WINN, J., AND CRIMINISI, A. 2013. Decision jungles: Compact and rich models for classification. In NIPS, 234-242.
36. SRIDHAR, S., OULASVIRTA, A., AND THEOBALT, C. 2013. Interactive markerless articulated hand motion tracking using RGB and depth data. In Proc. ICCV, 2456-2463.
37. SRIDHAR, S., RHODIN, H., SEIDEL, H.-P., OULASVIRTA, A., AND THEOBALT, C. 2014. Real-time hand tracking using a sum of anisotropic Gaussians model. In Proc. 3DV, 319-326.
38. SRIDHAR, S., MUELLER, F., OULASVIRTA, A., AND THEOBALT, C. 2015. Fast and robust hand tracking using detection-guided optimization. In Proc. CVPR, 3213-3221.
39. STENGER, B., MENDONÇ A, P. R., AND CIPOLLA, R. 2001. Modelbased 3D tracking of an articulated hand. In Proc. CVPR, vol. 2, II-310.
40. SUN, X., WEI, Y., LIANG, S., TANG, X., AND SUN, J. 2015 Cascaded hand pose regression. In Proc. CVPR, 824-832.
41. TAGLIASACCHI, A., SCHRö DER, M., TKACH, A., BOUAZIZ, S., BOTSCH, M., AND PAULY, M. 2015. Robust articulated-ICP for real-time hand tracking. Computer Graphics Forum 34, 5, 101-114.
42. TAN, D. J., CASHMAN, T., TAYLOR, J., FITZGIBBON, A., TARLOW, D., KHAMIS, S., IZADI, S., AND SHOTTON, J. 2016. Fits like a glove: Rapid and reliable hand shape personalization. In Proc. CVPR.
43. TANG, D., YU, T.-H., AND KIM, T.-K. 2013. Real-time articulated hand pose estimation using semi-supervised transductive regression forests. In Proc. ICCV, 3224-3231.
44. TANG, D., TAYLOR, J., KOHLI, P., KESKIN, C., KIM, T.-K., AND SHOTTON, J. 2015. Opening the black box: Hierarchical sampling optimization for estimating human hand pose. In Proc. ICCV, 3325-3333.
45. TAYLOR, J., SHOTTON, J., SHARP, T., AND FITZGIBBON, A. 2012. The Vitruvian Manifold: Inferring dense correspondences for one-shot human pose estimation. In Proc. CVPR, 103-110.
46. TAYLOR, J., STEBBING, R., RAMAKRISHNA, V., KESKIN, C., SHOTTON, J., IZADI, S., HERTZMANN, A., AND FITZGIBBON, A. 2014. User-specific hand modeling from monocular depth sequences. In Proc. CVPR, 644-651.
47. TEJANI, A., TANG, D., KOUSKOURIDAS, R., AND KIM, T.-K. 2014. Latent-class Hough forests for 3D object detection and pose estimation. In Proc. ECCV, 462-477.
48. TOMPSON, J., STEIN, M., LECUN, Y., AND PERLIN, K. 2014. Real-time continuous pose recovery of human hands using convolutional networks. ACM Trans. Graphics 33, 5, #169.
49. TRIGGS, W., MCLAUCHLAN, P., HARTLEY, R., AND FITZGIBBON, A. 2000. Bundle adjustment-A modern synthesis. In Vision Algorithms: Theory and Practice, LNCS. 298-372.
50. TZIONAS, D., BALLAN, L., SRIKANTHA, A., APONTE, P., POLLEFEYS, M., AND GALL, J. 2015. Capturing hands in action using discriminative salient points and physics simulation. arXiv preprint 1506.02178.
51. ULTRAHAPTICS LTD, 2013. Haptics System. http://ultrahaptics.com, Jan.
52. VALENTIN, J., DAI, A., NIESSNER, M., KOHLI, P., TORR, P., IZADI, S., AND KESKIN, C. 2016. Learning to navigate the energy landscape. arXiv preprint 1603.05772.
53. VICENTE, S., AND AGAPITO, L. 2013. Balloon shapes: reconstructing and deforming objects with volume from images. In Proc. 3DV, 223-230.
54. WANG, R. Y., AND POPOVIć, J. 2009. Real-time hand-tracking with a color glove. ACM Trans. Graphics 28, 3, #63.
55. WANG, R., PARIS, S., AND POPOVIć, J. 2011. 6D hands. In Proc. UIST, 549-558.
56. WANG, Y., MIN, J., ZHANG, J., LIU, Y., XU, F., DAI, Q., AND CHAI, J. 2013. Video-based hand manipulation capture through composite motion control. ACM Trans. Graphics 32, 4 (July), 43:1-43:14.
57. WU, Y., AND HUANG, T. S. 2000. View-independent recognition of hand postures. In Proc. CVPR, vol. 2, 88-94.
58. WU, Y., LIN, J. Y., AND HUANG, T. S. 2001. Capturing natural hand articulation. In Proc. ICCV, vol. 2, 426-432.
59. XU, C., AND CHENG, L. 2013. Efficient hand pose estimation from a single depth image. In Proc. ICCV, 3456-3462.
60. ZACH, C. 2014. Robust bundle adjustment revisited. In Proc. ECCV, 772-787.
61. ZHAO, W., CHAI, J., AND XU, Y.-Q. 2012. Combining markerbased mocap and RGB-D camera for acquiring high-fidelity hand motion data. In Proc. Symposium on Computer Animation, 33-42.