Soli: Ubiquitous gesture sensing with millimeter wave radar

ACM Transactions on Graphics

Volumn 35, Issue 4, 2016, Pages

  Lien, Jaime ^a   Gillian, Nicholas ^a   Karagozler, M Emre ^a   Amihood, Patrick ^a   Schwesig, Carsten ^a   Olson, Erik ^a   Raja, Hakim ^a   Poupyrev, Ivan ^a  

^a GOOGLE INC   (United States)

Author keywords

Gestures; Interaction; Radar; RF; Sensors

Indexed keywords

ABSTRACTING; COMPUTER HARDWARE; GESTURE RECOGNITION; HARDWARE; INTERACTIVE COMPUTER GRAPHICS; MILLIMETER WAVES; RADAR; RECONFIGURABLE HARDWARE; SENSORS;

GESTURE VOCABULARIES; GESTURES; HARDWARE ABSTRACTION LAYERS; HIGH TEMPORAL RESOLUTION; INTERACTION; MILLIMETER WAVE RADAR; SENSOR ARCHITECTURES; SYSTEM ARCHITECTURES;

HUMAN COMPUTER INTERACTION;

EID: 84979966849     PISSN: 07300301     EISSN: 15577368     Source Type: Journal    
DOI: 10.1145/2897824.2925953     Document Type: Conference Paper

References (85)

1. AXELSSON, S. R. 2004. Noise radar using random phase and frequency modulation. IEEE Transactions on Geoscience and Remote Sensing 42, 11, 2370-2384.
2. AZEVEDO, S., AND MCEWAN, T. 1996. Micropower impulse radar. Science and Technology Review, 17-29.
3. BARNES, S. B. 1997. Douglas Carl Engelbart: Developing the underlying concepts for contemporary computing. IEEE Annals of the History of Computing 19, 3, 16-26.
4. BAUM, C. E., ROTHWELL, E. J., CHEN, K.-M., AND NYQUIST, D. P. 1991. The singularity expansion method and its application to target identification. Proceedings of the IEEE 79, 10, 1481-1492.
5. BENEZETH, Y., JODOIN, P. M., EMILE, B., LAURENT, H., AND ROSENBERGER, C. 2008. Review and evaluation of commonlyimplemented background subtraction algorithms. In ICPR 2008.
6. BISHOP, C. M. 2006. Pattern recognition. Machine Learning.
7. BOLT, R. A. 1980. Put-that-there. SIGGRAPH '80, 262-270.
8. BREIMAN, L. 2001. Random forests. Machine Learning 45, 1, 5-32.
9. BROOKNER, E. 1985. Phased-array radars. Scientific American 252, 2, 94-102.
10. BROWN, L. 1999. A Radar History of World War II: Technical and Military Imperatives. Institute of Physics Publishing.
11. BRUNNBAUER, M., MEYER, T., OFNER, G., MUELLER, K., AND HAGEN, R. 2008. Embedded wafer level ball grid array (eWLB). In IEMT 2008, IEEE, 1-6.
12. CARD, S., MORAN, T., AND NEWELL, A. 1983. The Psychology of Human-Computer Interaction. L. Erlbaum Associates.
13. CHAN, L., CHEN, C.-H. H. Y.-L., AND YANG, S. 2015. Cyclops: Wearable and single-piece full-body gesture input devices. In CHI 2015, ACM, 3001-3009.
14. COMSCORE INC. 2014. The US mobile app report. Tech. rep.
15. COOPERSTOCK, J. R., FELS, S. S., BUXTON, W., AND SMITH, K. C. 1997. Reactive environments. Communications of the ACM 40, 9 (Sep), 65-73.
16. DARDAS, N. H., AND GEORGANAS, N. D. 2011. Real-time hand gesture detection and recognition using bag-of-features and support vector machine techniques. IEEE Transactions on Instrumentation and Measurement 60, 11, 3592-3607.
17. DIETZ, P., AND LEIGH, D. 2001. Diamond Touch. In UIST '01, 219.
18. DORFMULLER-ULHAAS, K., AND SCHMALSTIEG, D. 2001. Finger tracking for interaction in augmented environments. ISMAR 2001, 55-64.
19. DUFFNER, S., BERLEMONT, S., LEFEBVRE, G., AND GARCIA, C. 2014. 3D gesture classification with convolutional neural networks. In ICASSP 2014, IEEE, 5432-5436.
20. FCC, 2016. FCC online table of frequency allocations.
21. GENG, S., KIVINEN, J., ZHAO, X., AND VAINIKAINEN, P. 2009. Millimeter-wave propagation channel characterization for shortrange wireless communications. IEEE Transactions on Vehicular Technology 58, 1 (Jan), 3-13.
22. GILLIAN, N., KNAPP, R. B., AND O'MODHRAIN, S. 2011. Recognition of multivariate temporal musical gestures using ndimensional dynamic time warping. In Proceedings of the 2011 International Conference on New Interfaces for Musical Expression (NIME 11), Oslo, Norway.
23. GILLIAN, N. 2011. Gesture Recognition for Musician Computer Interaction. PhD thesis, School of Music and Sonic Arts, Queen's University Belfast.
24. GUSTAFSON, S., HOLZ, C., AND BAUDISCH, P. 2011. Imaginary phone: Learning imaginary interfaces by transferring spatial memory from a familiar device. UIST '11, 283-292.
25. GUYON, I., AND ELISSEEFF, A. 2003. An introduction to variable and feature selection. The Journal of Machine Learning Research 3.
26. HANSEN, C. J. 2011. WiGiG: Multi-gigabit wireless communications in the 60 GHz band. IEEE Wireless Communications 18, 6, 6-7.
27. HARRISON, C., TAN, D., AND MORRIS, D. 2010. Skinput: Appropriating the body as an input surface. CHI 2010, 453-462.
28. HOLLEIS, P., SCHMIDT, A., PAASOVAARA, S., PUIKKONEN, A., AND HAKKILA, J. 2008. Evaluating capacitive touch input on clothes. In MobileHCI '08, ACM Press, New York, New York, USA, 81-90.
29. HOLZ, C., AND WILSON, A. 2011. Data miming: Inferring spatial object descriptions from human gesture. CHI 2011, 811-820.
30. HUSSAIN, M. G. 1998. Ultra-wideband impulse radar-an overview of the principles. IEEE Aerospace and Electronic Systems Magazine 13, 9, 9-14.
31. JI, S., XU, W., YANG, M., AND YU, K. 2013. 3D convolutional neural networks for human action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 1, 221-231.
32. JUNKER, H., AMFT, O., LUKOWICZ, P., AND TRöSTER, G. 2008. Gesture spotting with body-worn inertial sensors to detect user activities. Pattern Recognition 41, 6, 2010-2024.
33. KELLER, J. B. 1962. Geometrical theory of diffraction. JOSA 52, 2, 116-130.
34. KELLOGG, B., TALLA, V., AND GOLLAKOTA, S. 2014. Bringing gesture recognition to all devices. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14), 303-316.
35. KESKIN, C., KIRAÇ, F., KARA, Y. E., AND AKARUN, L. 2013. Real time hand pose estimation using depth sensors. In Consumer Depth Cameras for Computer Vision. Springer, 119-137.
36. KHOSHELHAM, K., AND ELBERINK, S. O. 2012. Accuracy and resolution of Kinect depth data for indoor mapping applications. Sensors 12, 2, 1437-1454.
37. KIM, D., HILLIGES, O., IZADI, S., BUTLER, A. D., CHEN, J., OIKONOMIDIS, I., AND OLIVIER, P. 2012. Digits. UIST '12, 167-176.
38. KNOTT, E. F. 2012. Radar Cross Section Measurements. Springer Science & Business Media.
39. KURAKIN, A., ZHANG, Z., AND LIU, Z. 2012. A real time system for dynamic hand gesture recognition with a depth sensor. In 2012 Proceedings of the 20th European Signal Processing Conference (EUSIPCO), 1975-1979.
40. LEE, S., AND BUXTON, W. 1985. A multi-touch three dimensional touch-sensitive tablet. In CHI'85, 21-25.
41. LEVANON, N. 2000. Multifrequency complementary phase-coded radar signal. IEE Proceedings-Radar, Sonar and Navigation 147, 6, 276-284.
42. LOPRESTI, L., LACASCIA, M., SCLAROFF, S., AND CAMPS, O. 2015. Gesture modeling by Hanklet-based hidden Markov model. In Computer Vision-ACCV 2014, D. Cremers, I. Reid, H. Saito, and M.-H. Yang, Eds., vol. 9005 of Lecture Notes in Computer Science. Springer International Publishing, 529-546.
43. MELGAREJO, P., ZHANG, X., RAMANATHAN, P., AND CHU, D. 2014. Leveraging directional antenna capabilities for finegrained gesture recognition. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing, ACM, 541-551.
44. MOLCHANOV, P., GUPTA, S., KIM, K., AND KAUTZ, J. 2015. Hand gesture recognition with 3D convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 1-7.
45. MOLCHANOV, P., GUPTA, S., KIM, K., AND PULLI, K. 2015. Multi-sensor system for driver's hand-gesture recognition. In IEEE Conference on Automatic Face and Gesture Recognition.
46. MOLCHANOV, P., GUPTA, S., KIM, K., AND PULLI, K. 2015. Short-range FMCW monopulse radar for hand-gesture sensing. In IEEE Radar Conference (RadarCon 2015), IEEE, 1491-1496.
47. NASR, I., KARAGOZLER, E., POUPYREV, I., AND TROTTA, S. 2015. A highly integrated 60-GHz 6-channel transceiver chip in 0.35 um SiGe technology for smart sensing and short-range communications. In IEEE CSICS 2015, IEEE, 1-4.
48. NASR, I., JUNGMAIER, R., BAHETI, A., NOPPENEY, D., BAL, J. S., WOJNOWSKI, M., KARAGOZLER, E., RAJA, H., LIEN, J., POUPYREV, I., AND TROTTA, S. 2016. A highly integrated 60 GHz 6-channel transceiver with antenna in package for smart sensing and short range communications. submitted to IEEE Journal of Solid State Circuits.
49. NATHANSON, F. E., REILLY, J. P., AND COHEN, M. N. 1991. Radar Design Principles-Signal Processing and the Environment, vol. 91. NOVELDA. Novelda Xethru NVA620x IC.
50. OTERO, M. 2005. Application of a continuous wave radar for human gait recognition. Proceedings of SPIE 5809, 538-548.
51. PARADISO, J., ABLER, C., HSIAO, K.-Y., AND REYNOLDS, M. 1997. The Magic Carpet: Physical sensing for immersive environments. In CHI'97 Extended Abstract: s on Human Factors in Computing Systems, ACM, 277-278.
52. PARADISO, J. A. 1999. The Brain Opera technology: New instruments and gestural sensors for musical interaction and performance. Journal of New Music Research 28, 2, 130-149.
53. PARK, J.-I., AND KIM, K.-T. 2010. A comparative study on ISAR imaging algorithms for radar target identification. Progress In Electromagnetics Research 108, 155-175.
54. POTTER, L. C., CHIANG, D.-M., CARRIERE, R., AND GERRY, M. J. 1995. A GTD-based parametric model for radar scattering. IEEE Transactions on Antennas and Propagation 43, 10, 1058-1067.
55. PU, Q., GUPTA, S., GOLLAKOTA, S., AND PATEL, S. 2013. Whole-home gesture recognition using wireless signals. In Proceedings of the 19th Annual International Conference on Mobile Computing & Networking, ACM, 27-38.
56. RAHMAN, T., ADAMS, A. T., ZHANG, M., AND CHOUDHURY, T. 2015. DoppleSleep: A contactless unobtrusive sleep sensing system using short-range doppler radar. In Ubicomp '15, ACM.
57. RALSTON, T. S., CHARVAT, G. L., AND PEABODY, J. E. 2010. Real-time through-wall imaging using an ultrawideband MIMO phased array radar system. In IEEE International Symposium on Phased Array Systems and Technology (ARRAY), IEEE.
58. REKIMOTO, J. 2001. GestureWrist and GesturePad: Unobtrusive wearable interaction devices. ISWC 2001, 21-27.
59. RICHARDS, M. A., SCHEER, J., HOLM, W. A., ET AL. 2010. Principles of modern radar: basic principles. SciTech Pub.
60. ROMERO, J., KJELLSTRöM, H., EK, C. H., AND KRAGIC, D. 2013. Non-parametric hand pose estimation with object context. Image and Vision Computing 31, 8, 555-564.
61. RUSSELL, D., STREITZ, N., AND WINOGRAD, T. 2005. Building disappearing computers. Communications of the ACM 48, 3, 42-48.
62. SAPONAS, T. S., TAN, D. S., MORRIS, D., BALAKRISHNAN, R., TURNER, J., AND LANDAY, J. A. 2009. Enabling alwaysavailable input with muscle-computer interfaces. UIST 2009, 167-176.
63. SHARP, T., KESKIN, C., ROBERTSON, D., TAYLOR, J., SHOTTON, J., KIM, D., RHEMANN, C., LEICHTER, I., VINNIKOV, A., WEI, Y., FREEDMAN, D., KOHLI, P., KRUPKA, E., FITZGIBBON, A., AND IZADI, S. 2015. Accurate, robust, and flexible real-time hand tracking. CHI 2015, 3633-3642.
64. SHOTTON, J., SHARP, T., KIPMAN, A., FITZGIBBON, A., FINOCCHIO, M., BLAKE, A., COOK, M., AND MOORE, R. 2013. Real-time human pose recognition in parts from single depth images. Communications of the ACM 56, 1, 116-124.
65. SHUEY, D., BAILEY, D., AND MORRISSEY, T. P. 1986. PHIGS: A standard, dynamic, interactive graphics interface. IEEE Computer Graphics and Applications 6, 8, 50-57.
66. SKOLNIK, M. I. 1962. Introduction to radar. Radar Handbook 2.
67. SMITH, C., AND GOGGANS, P. 1993. Radar target identification. IEEE Antennas and Propagation Magazine 35, 2 (April), 27-38.
68. SMITH, J., WHITE, T., DODGE, C., PARADISO, J., GERSHENFELD, N., AND ALLPORT, D. 1998. Electric field sensing for graphical interfaces. IEEE Computer Graphics and Applications 18, June, 54-59.
69. SMULDERS, P. 2002. Exploiting the 60 GHz band for local wireless multimedia access: Prospects and future directions. IEEE Communications Magazine 40, 1, 140-147.
70. SONG, J., Sö Rö S, G., PECE, F., FANELLO, S. R., IZADI, S., KESKIN, C., AND HILLIGES, O. 2014. In-air gestures around unmodified mobile devices. UIST 2014, 319-329.
71. STORCHEUS, D., ROSTAMIZADEH, A., AND KUMAR, S. 2015. A survey of modern questions and challenges in feature extraction. In Proceedings of the 1st International Workshop on Feature Extraction: Modern Questions and Challenges, NIPS, 1-18.
72. STOVE, A. G. 1992. Linear FMCW radar techniques. In IEEE Proceedings on Radar and Signal Processing, vol. 139, IET, 343-350.
73. STRICKON, J., AND PARADISO, J. 1998. Tracking hands above large interactive surfaces with a low-cost scanning laser rangefinder. CHI'98, 231-232.
74. SUTHERLAND, I. E., BLACKWELL, A., AND RODDEN, K. 1963. Sketchpad: A man-machine graphical communication system. Tech. Rep. TK296, Lincoln Laboratory, MIT, Lexington, MA.
75. WAN, Q., LI, Y., LI, C., AND PAL, R. 2014. Gesture recognition for smart home applications using portable radar sensors. In 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2014), IEEE, 6414-6417.
76. WANG, Y., AND FATHY, A. E. 2011. Micro-doppler signatures for intelligent human gait recognition using a UWB impulse radar. In IEEE Antennas and Propagation Society, AP-S International Symposium (Digest), 2103-2106.
77. WANG, R. Y., AND POPOVIć, J. 2009. Real-time hand-tracking with a color glove. In SIGGRAPH 2009, ACM Press, New York, New York, USA, 1.
78. WATSON-WATT, R. 1945. Radar in war and in peace. Nature 155, 3935, 319-324.
79. WEI, T., AND ZHANG, X. 2015. mTrack: High-precision passive tracking using millimeter wave radios. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking, ACM, 117-129.
80. WEICHERT, F., BACHMANN, D., RUDAK, B., AND FISSELER, D. 2013. Analysis of the accuracy and robustness of the Leap Motion controller. Sensors 13, 5, 6380-6393.
81. WEIGEL, M., LU, T., BAILLY, G., OULASVIRTA, A., MAJIDI, C., AND STEIMLE, J. 2015. iSkin: Flexible, stretchable and visually customizable on-body touch sensors for mobile computing. In CHI 2015, ACM, 1-10.
82. WU, J., KONRAD, J., AND ISHWAR, P. 2013. Dynamic time warping for gesture-based user identification and authentication with Kinect. In ICASSP 2013, 2371-2375.
83. YATANI, K., AND TRUONG, K. N. 2012. BodyScope: A wearable acoustic sensor for activity recognition. In Ubicomp '12, ACM, 341-350.
84. ZHAI, S., MILGRAM, P., AND BUXTON, W. 1996. The influence of muscle groups on performance of multiple degree-of-freedom input. CHI '96, 308-315.
85. ZHUANG, Y., SONG, C., WANG, A., LIN, F., LI, Y., GU, C., LI, C., AND XU, W. 2015. SleepSense: Non-invasive sleep event recognition using an electromagnetic probe. In IEEE 12th International Conference on Wearable and Implantable Body Sensor Networks (BSN 2015), 1-6.