Implementation of a Virtual Training Simulator Based on 360° Multi-View Human Action Recognition

IEEE Access

Volumn 5, Issue , 2017, Pages 12496-12511

  Kwon, Beom ^a   Kim, Junghwan ^a   Lee, Kyoungoh ^a   Lee, Yang Koo ^b   Park, Sangjoon ^b   Lee, Sanghoon ^a  

^a Yonsei University   (South Korea)

^b Electronics and Telecommunications Research Institute (ETRI)   (South Korea)

Author keywords

Human action recognition; Kinect sensor; virtual training simulator

Indexed keywords

ACCELEROMETERS; EXTRACTION; FEATURE EXTRACTION; MILITARY APPLICATIONS; MONITORING; MUSCULOSKELETAL SYSTEM; PERSONNEL TRAINING; SIMULATORS; VIRTUAL REALITY;

HUMAN-ACTION RECOGNITION; KINECT SENSORS; SENSOR SYSTEMS; SKELETON; VIRTUAL TRAINING SIMULATORS;

E-LEARNING;

EID: 85023190889     PISSN: None     EISSN: 21693536     Source Type: Journal    
DOI: 10.1109/ACCESS.2017.2723039     Document Type: Article

References (65)

1. D. Lee, K. Baek, J. Lee, and H. Lim, "A development of virtual reality game utilizing Kinect, oculus rift and smartphone, " Int. J. Appl. Eng. Res., vol. 11, no. 2, pp. 829-833, 2016.
2. Z. Merchant, E. T. Goetz, L. Cifuentes, W. Keeney-Kennicutt, and T. J. Davis, "Effectiveness of virtual reality-based instruction on students' learning outcomes in K-12 and higher education: A meta-analysis, " Com-put. Edu., vol. 70, pp. 29-40, Jan. 2014.
3. D. Meldrum, A. Glennon, S. Herdman, D. Murray, and R. McConn-Walsh, "Virtual reality rehabilitation of balance: Assessment of the usability of the NintendoWii Fit Plus, " Disability Rehabil. Assistive Technol., vol. 7, no. 3, pp. 205-210, 2012.
4. K. E. Kopecky and E. H. Winer, "MetaTracker: Unifying and abstracting 3-D motion tracking data from multiple heterogenous hardware systems, " IEEE Access, vol. 4, pp. 189-203, 2016.
5. H. Oh and S. Lee, "Visual presence: Viewing geometry visual information of UHD S3D entertainment, " IEEE Trans. Image Process., vol. 25, no. 7, pp. 3358-3371, Jul. 2016.
6. H. Kim, S. Lee, and A. C. Bovik, "Saliency prediction on stereoscopic videos, " IEEE Trans. Image Process., vol. 23, no. 4, pp. 1476-1490, Apr. 2014.
7. B. W. Knerr, "Immersive simulation training for the dismounted soldier, " U. S. Army Res. Inst. Behavioral Social Sci., Tech. Rep., Feb. 2007.
8. G. S. Taylor and J. S. Barnett, "Evaluation of wearable simulation interface for military training, " Human Factors, J. Human Factors Ergonom. Soc., vol. 55, no. 3, pp. 672-690, Jun. 2013.
9. Quantum 3D Product Literature, Quantum3D Incorporated, San Jose, CA, USA. [Online]. Available: http://www. quantum3d. com
10. J. Han, L. Shao, D. Xu, and J. Shotton, "Enhanced computer vision with microsoft Kinect sensor: A review, " IEEE Trans. Cybern., vol. 43, no. 5, pp. 1318-1334, Oct. 2013.
11. Z.-C. Wang, C.-C. Tsai, and M.-C. Chien, "Design of an intelligent soldier combat training system, " Int. J. Autom. Smart Technol., vol. 2, no. 4, pp. 309-317, 2012.
12. A. Stork et al., "Enabling virtual assembly training in and beyond the automotive industry, " in Proc. IEEE 18th Int. Conf. Virtual Syst. Multimedia (VSMM), Sep. 2012, pp. 347-352.
13. C. Park and J. Moon, "Using game technology to develop snowboard training simulator, " in Proc. Int. Conf. Human-Comput. Interact., Jul. 2013, pp. 723-726.
14. F. Cassola, L. Morgado, F. de Carvalho, H. Paredes, B. Fonseca, and P. Martins, "Online-Gym: A 3D virtual gymnasium using Kinect interaction, " Procedia Technol., vol. 13, pp. 130-138, 2014.
15. D. Bogatinov, P. Lameski, V. Trajkovik, and K. M. Trendova, "Firearms training simulator based on low cost motion tracking sensor, " Multimedia Tools Appl., vol. 76, no. 1, pp. 1403-1418, 2017.
16. Z. He, Z. Liu, L. Jin, L.-X. Zhen, and J.-C. Huang, "Weightlessness feature-A novel feature for single tri-axial accelerometer based activity recognition, " in Proc. IEEE 19th Int. Conf. Pattern Recognit. (ICPR), Dec. 2008, pp. 1-4.
17. A. M. Khan, Y.-K. Lee, S. Y. Lee, and T.-S. Kim, "Atriaxial accelerometerbased physical-activity recognition via augmented-signal features and a hierarchical recognizer, " IEEE Trans. Inf. Technol. Biomed., vol. 14, no. 5, pp. 1166-1172, Sep. 2010.
18. A. M. Khan, Y.-K. Lee, S. Lee, and T.-S. Kim, "Accelerometer's position independent physical activity recognition system for long-term activity monitoring in the elderly, " Med. Biol. Eng. Comput., vol. 48, no. 12, pp. 1271-1279, Dec. 2010.
19. L. Bao and S. S. Intille, "Activity recognition from user-annotated acceleration data, " in Proc. Int. Conf. Pervasive Comput., Apr. 2004, pp. 1-17.
20. E. M. Tapia et al., "Real-time recognition of physical activities and their intensities using wireless accelerometers and a heart rate monitor, " in Proc. 11th IEEE Int. Symp. Wearable Comput., Boston, MA, USA, Oct. 2007, pp. 37-40.
21. U. Maurer, A. Smailagic, D. P. Siewiorek, and M. Deisher, "Activity recognition and monitoring using multiple sensors on different body positions, " in Proc. IEEE Int. Workshop Wearable Implant. Body Sensor Netw. (BSN), Apr. 2006, pp. 1-4.
22. A. Y. Yang, S. Iyengar, P. Kuryloski, and R. Jafari, "Distributed segmentation and classification of human actions using a wearable motion sensor network, " in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), Jun. 2008, pp. 1-8.
23. C. Thurau and V. Hlavá, "Pose primitive based human action recognition in videos or still images, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2008, pp. 1-8.
24. W. Yang, Y. Wang, and G. Mori, "Recognizing human actions from still images with latent poses, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2010, pp. 2030-2037.
25. S. Maji, L. Bourdev, and J. Malik, "Action recognition from a distributed representation of pose and appearance, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2011, pp. 3177-3184.
26. A. Gupta, A. Kembhavi, and L. S. Davis, "Observing human-object interactions: Using spatial and functional compatibility for recognition, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 10, pp. 1775-1789, Oct. 2009.
27. B. Yao, X. Jiang, A. Khosla, A. L. Lin, L. Guibas, and L. Fei-Fei, "Human action recognition by learning bases of action attributes and parts, " in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Nov. 2011, pp. 1331-1338.
28. D. T. Le, R. Bernardi, and J. Uijlings, "Exploiting language models to recognize unseen actions, " in Proc. 3rd ACM Conf. Int. Conf. Multimedia Retr., Apr. 2013, pp. 231-238.
29. T. Ojala, M. Pietikäinen, and T. Mäenpää, "Multiresolution gray-scale and rotation invariant texture classification with local binary patterns, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 7, pp. 971-987, Jul. 2002.
30. P. Scovanner, S. Ali, and M. Shah, "A 3-dimensional sift descriptor and its application to action recognition, " in Proc. ACM Multimedia, Sep. 2007, pp. 357-360.
31. H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, "Speeded-up robust features (SURF), " Comput. Vis. Image Understand., vol. 110, no. 3, pp. 346-359, 2008.
32. I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld, "Learning realistic human actions from movies, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Anchorage, AK, USA, Jun. 2008, pp. 1-8.
33. A. Klaser, M. Marszalek, and C. Schmid, "A spatio-temporal descriptor based on 3D-gradients, " in Proc. 19th Brit. Mach. Vis. Conf. (BMVS), Sep. 2008, pp. 275-284.
34. H. Wang, D. Oneata, J. Verbeek, and C. Schmid, "A robust and efficient video representation for action recognition, " Int. J. Comput. Vis., vol. 119, no. 3, pp. 219-238, Sep. 2016.
35. L. Liu, L. Shao, X. Li, and K. Lu, "Learning spatio-temporal representations for action recognition: A genetic programming approach, " IEEE Trans. Cybern., vol. 46, no. 1, pp. 158-170, Jan. 2016.
36. L. Liu and L. Shao, "Learning discriminative representations from RGBD video data, " in Proc. 23th Int. Joint Conf. Artif. Intell. (IJCAI), vol. 4. Aug. 2013, pp. 1493-1500.
37. S. Ji, W. Xu, M. Yang, and K. Yu, "3D convolutional neural networks for human action recognition, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 1, pp. 221-231, Jan. 2013.
38. L. Liu, Y. Zhou, and L. Shao. (2016). "DAP3D-Net: Where, what and how actions occur in videos?" [Online]. Available: https://arxiv. org/ abs/1602. 03346
39. Z. Liu, C. Zhang, and Y. Tian, "3d-based deep convolutional neural network for action recognition with depth sequences, " Image Vis. Comput., vol. 55, pp. 93-100, Nov. 2016.
40. J. Carreira, P. Agrawal, K. Fragkiadaki, and J. Malik, "Human pose estimation with iterative error feedback, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 4733-4742.
41. A. Newell, K. Yang, and J. Deng, "Stacked hourglass networks for human pose estimation, " in Proc. Eur. Conf. Comput. Vis. (ECCV), Oct. 2016, pp. 483-499.
42. X. Zhou et al., "Sparseness meets deepness: 3D human pose estimation from monocular video, " in Proc. IEEE Conf. Comput. Vis. Pattern Recog-nit. (CVPR), Jun. 2016, pp. 4966-4975.
43. F. Zhou and F. De la Torre, "Spatio-temporal matching for human pose estimation in video, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 8, pp. 1492-1504, Aug. 2016.
44. V. Bloom, D. Makris, and V. Argyriou, "G3D: A gaming action dataset and real time action recognition evaluation framework, " in Proc. IEEE Com-put. Soc. Conf. Comput. Vis. Pattern Recognit. Workshops, Providence, RI, USA, Jun. 2012, pp. 7-12.
45. G. T. Papadopoulos, A. Axenopoulos, and P. Daras, "Real-time skeletontracking-based human action recognition using Kinect data, " in Proc. Int. Conf. Multimedia Modeling, Jan. 2014, pp. 473-483.
46. V. Bloom, V. Argyriou, and D. Makris, "Hierarchical transfer learning for online recognition of compound actions, " Comput. Vis. Image Under-stand., vol. 144, pp. 62-72, Mar. 2016.
47. L. Xia, C.-C. Chen, and J. K. Aggarwal, "View invariant human action recognition using histograms of 3D joints, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), Jun. 2012, pp. 20-27.
48. N. A. Azis, H.-J. Choi, and Y. Iraqi, "Substitutive skeleton fusion for human action recognition, " in Proc. IEEE Int. Conf. Big Data Smart Comput. (BigComp), Feb. 2015, pp. 170-177.
49. N. A. Azis, Y. S. Jeong, H. J. Choi, and Y. Iraqi, "Weighted averaging fusion for multi-view skeletal data and its application in action recognition, " IET Comput. Vis., vol. 10, no. 2, pp. 134-142, 2016.
50. M. Li, W. Song, L. Song, K. Huang, Y. Xi, and K. Cho, "A wireless Kinect sensor network system for virtual reality applications, " in Proc. Int. Conf. Comput. Sci. Appl., Dec. 2016, pp. 61-65.
51. Z. Zhang, "Microsoft Kinect sensor and its effect, " IEEE Multimedia, vol. 19, no. 2, pp. 4-10, Feb. 2012.
52. J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, "A benchmark for the evaluation of RGB-D SLAM systems, " in Proc. IEEE/RSJ Int. Conf. Intell. Robot Syst. (IROS), Vilamoura, Portugal, Oct. 2012, pp. 573-580.
53. C. Raposo, J. P. Barreto, and U. Nunes, "Fast and accurate calibration of a Kinect sensor, " in Proc. IEEE Int. Conf. 3D Vision-3DV, Jun. 2013, pp. 342-349.
54. J. Kim, I. Lee, J. Kim, and S. Lee, "Implementation of an omnidirectional human motion capture system using multiple Kinect sensors, " IEICE Trans. Fundam. Electron., Commun. Comput. Sci., vol. E98-A, no. 9, pp. 2004-2008, 2015.
55. S. Yonemoto, A. Matsumoto, D. Arita, and R.-I. Taniguchi, "A real-time motion capture system with multiple camera fusion, " in Proc. IEEE Int. Conf. Image Anal. Process., Sep. 1999, pp. 600-605.
56. R. Y. Wang, "Practical color-based motion capture, " Massachusetts Inst. Technol., Cambridge, MA, USA, Tech. Rep., 2011.
57. J. Liu, S. Chen, H. Sun, Y. Qin, and X. Wang, "Real time tracking method by using color markers, " in Proc. IEEE Int. Conf. Virtual Reality Vis. (ICVRV), Sep. 2013, pp. 106-111.
58. M. A. Livingston, J. Sebastian, Z. Ai, and J. W. Decker, "Performance measurements for the Microsoft Kinect skeleton, " in Proc. IEEE Virtual Reality Workshops, Mar. 2012, pp. 119-120.
59. J. Gu, X. Ding, S. Wang, and Y. Wu, "Action and gait recognition from recovered 3-D human joints, " IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 40, no. 4, pp. 1021-1033, Aug. 2010.
60. J. Wang, Z. Liu, Y. Wu, and J. Yuan, "Mining actionlet ensemble for action recognition with depth cameras, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2012, pp. 1290-1297.
61. W. Zhu et al. (2016). "Co-occurrence feature learning for skeleton based action recognition using regularized deep LSTM networks. " [Online]. Available: https://arxiv. org/abs/1603. 07772
62. A. Ramey, V. González-Pacheco, and M. A. Salichs, "Integration of a lowcost RGB-D sensor in a social robot for gesture recognition, " in Proc. ACM/IEEE Int. Conf. Human-Robot Interact., 2011, pp. 229-230.
63. M. Zanr, M. Leordeanu, and C. Sminchisescu, "The moving pose: An efficient 3D kinematics descriptor for low-latency action recognition and detection, " in Proc. IEEE Int. Conf. Comput. Vis., Dec. 2013, pp. 2752-2759.
64. W. Ding, K. Liu, F. Cheng, and J. Zhang, "STFC: Spatio-temporal feature chain for skeleton-based human action recognition, " J. Vis. Commun. Image Represent., vol. 26, pp. 329-337, Jan. 2015.
65. S. Nowozin and J. Shotton, "Action points: A representation for lowlatency online human action recognition, " Microsoft Res., Cambridge, U. K., Tech. Rep. MSR-TR-2012-68, 2012.