Classification of motor errors to provide real-time feedback for sports coaching in virtual reality — A case study in squats and Tai Chi pushes

Computers and Graphics (Pergamon)

Volumn 76, Issue , 2018, Pages 47-59

  Hülsmann, Felix ^a   Göpfert, Jan Philip ^a   Hammer, Barbara ^a   Kopp, Stefan ^a   Botsch, Mario ^a  

^a BIELEFELD UNIVERSITY   (Germany)

Author keywords

Auto generated augmented feedback; Human motion analysis; Motor learning environments; Motor performance quality; Sports coaching in virtual reality

Indexed keywords

COMPUTER AIDED INSTRUCTION; DECISION TREES; NEAREST NEIGHBOR SEARCH; NEURAL NETWORKS; PIPELINES; SPORTS;

AUGMENTED FEEDBACK; AUTOMATICALLY GENERATED; CLASSIFICATION QUALITY; CONVOLUTIONAL NEURAL NETWORK; HUMAN ACTIVITY RECOGNITION; HUMAN MOTION ANALYSIS; MOTOR LEARNING; MOTOR PERFORMANCE;

VIRTUAL REALITY;

EID: 85053300543     PISSN: 00978493     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.cag.2018.08.003     Document Type: Article

References (64)

1. Kyan, M., Sun, G., Li, H., Zhong, L., Muneesawang, P., Dong, N., et al. An approach to ballet dance training through MS kinect and visualization in a cave virtual reality environment. ACM Trans Intell Syst Technol (TIST), 6(2), 2015, 23.
2. de Kok, I., Hough, J., Hülsmann, F., Botsch, M., Schlangen, D., Kopp, S., A multimodal system for real-time action instruction in motor skill learning. Proceedings of the international conference on multimodal interaction, 2015, ACM, 355–362.
3. Sigrist, R., Rauter, G., Marchal-Crespo, L., Riener, R., Wolf, P., Sonification and haptic feedback in addition to visual feedback enhances complex motor task learning. Exp Brain Res 233:3 (2015), 909–925.
4. Chan, J.C., Leung, H., Tang, J.K., Komura, T., A virtual reality dance training system using motion capture technology. IEEE Trans Learn Technol 4:2 (2011), 187–195.
5. Miles, H.C., Pop, S.R., Watt, S.J., Lawrence, G.P., John, N.W., A review of virtual environments for training in ball sports. Comput Graph 36:6 (2012), 714–726.
6. Schack, T., Bertollo, M., Koester, D., Maycock, J., Essig, K., 2014, Routledge, 953–965.
7. Hough, J., de Kok, I., Schlangen, D., Kopp, S., Timing and grounding in motor skill coaching interaction: consequences for the information state. Proceedings of the 19th SemDial workshop on the semantics and pragmatics of dialogue (goDIAL), 2015, 86–94.
8. de Kok, I., Hough, J., Frank, C., Schlangen, D., Kopp, S., Dialogue structure of coaching sessions. Proceedings of the SemDial workshop on the semantics and pragmatics of dialogue (DialWatt), 2014, 167–169.
9. de Kok, I., Hough, J., Schlangen, D., Kopp, S., Deictic gestures in coaching interactions. Proceedings of the workshop on multimodal analyses enabling artificial agents in human-machine interaction, 2016, ACM, 10–14.
10. Xi, X., Keogh, E., Shelton, C., Wei, L., Ratanamahatana, C.A., Fast time series classification using numerosity reduction. Proceedings of the 23rd international conference on Machine learning, 2006, ACM, 1033–1040.
11. Bagnall, A., Lines, J., An experimental evaluation of nearest neighbour time series classification. technical report# CMP-C14-01. Department of Computing Sciences, University of East Anglia, Tech. Rep., 2014, 1–7.
12. Núñez, J.C., Cabido, R., Pantrigo, J.J., Montemayor, A.S., Vélez, J.F., Convolutional neural networks and long short-term memory for skeleton-based human activity and hand gesture recognition. Pattern Recognit 76 (2018), 80–94.
13. Houmanfar, R., Karg, M., Kulić D., Movement analysis of rehabilitation exercises: distance metrics for measuring patient progress. IEEE Syst J 10:3 (2016), 1014–1025.
14. Rector, K., Bennett, C.L., Kientz, J.A., Eyes-free yoga: an exergame using depth cameras for blind & low vision exercise. Proceedings of the international ACM SIGACCESS conference on computers and accessibility, 2013, 12–19.
15. Taylor, P.E., Almeida, G.J., Kanade, T., Hodgins, J.K., Classifying human motion quality for knee osteoarthritis using accelerometers. Proceedings of the annual international conference of the IEEE engineering in medicine and biology, 2010, 339–343.
16. Yurtman, A., Barshan, B., Automated evaluation of physical therapy exercises using multi-template dynamic time warping on wearable sensor signals. Comput Methods Programs Biomed 117:2 (2014), 189–207.
17. Parisi, G.I., Magg, S., Wermter, S., Human motion assessment in real time using recurrent self-organization. Proceedings of the 25th IEEE international symposium on robot and human interactive communication (RO-MAN), 2016, 2016, IEEE, 71–76.
18. O'Reilly, M., Whelan, D., Chanialidis, C., Friel, N., Delahunt, E., Ward, T., et al. Evaluating squat performance with a single inertial measurement unit. Proceedings of the international conference on wearable and implantable body sensor networks (BSN), 2015, IEEE, 1–6.
19. Giggins, O., Kelly, D., Caulfield, B., Evaluating rehabilitation exercise performance using a single inertial measurement unit. Proceedings of the international conference on pervasive computing technologies for healthcare, 2013, 49–56.
20. Giggins, O.M., Sweeney, K.T., Caulfield, B., Rehabilitation exercise assessment using inertial sensors: a cross-sectional analytical study. J Neuroeng Rehabil, 11(1), 2014, 1.
21. Kianifar, R., Lee, A., Raina, S., Kulić D., Classification of squat quality with inertial measurement units in the single leg squat mobility test. Proceedings of the annual international conference of the engineering in medicine and biology society (EMBC), 2016, IEEE, 6273–6276.
22. Zou, H., Hastie, T., Tibshirani, R., Sparse principal component analysis. J Comput Graph Stat 15:2 (2006), 265–286.
23. Hossner, E.-J., Schiebl, F., Göhner, U., A functional approach to movement analysis and error identification in sports and physical education. Front Psychol, 6, 2015, 1339.
24. Sigrist, R., Rauter, G., Riener, R., Wolf, P., Augmented visual, auditory, haptic, and multimodal feedback in motor learning: a review. Psychon Bull Rev 20:1 (2013), 21–53.
25. Liu, D., Todorov, E., Evidence for the flexible sensorimotor strategies predicted by optimal feedback control. J Neurosci 27:35 (2007), 9354–9368.
26. Wilson, A.D., Bobick, A.F., Parametric hidden Markov models for gesture recognition. IEEE Trans Pattern Anal Mach Intell 21:9 (1999), 884–900.
27. Rodríguez, J.J., Alonso, C.J., Interval and dynamic time warping-based decision trees. Proceedings of the ACM symposium on applied computing, 2004, 548–552.
28. Wu, Y., Chang, E.Y., Distance-function design and fusion for sequence data. Proceedings of the thirteenth ACM international conference on Information and knowledge management, 2004, 324–333.
29. Nanopoulos, A., Alcock, R., Manolopoulos, Y., Feature-based classification of time-series data. Intl J Comput Res 10:3 (2001), 49–61.
30. Adistambha, K., Ritz, C.H., Burnett, I.S., Motion classification using dynamic time warping. Proceedings of the IEEE workshop on multimedia signal processing, 2008, 622–627.
31. Petitjean, F., Forestier, G., Webb, G.I., Nicholson, A.E., Chen, Y., Keogh, E., Dynamic time warping averaging of time series allows faster and more accurate classification. Proceedings of the International Conference on Data Mining, 2014, IEEE, 470–479.
32. Li, W., Wen, L., Chang, M.-C., Lim, S.N., Lyu, S., Adaptive rnn tree for large-scale human action recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, 1444–1452.
33. Liu, J., Shahroudy, A., Xu, D., Chichung, A.K., Wang, G., Skeleton-based action recognition using spatio-temporal LSTM network with trust gates. IEEE Trans Pattern Anal Mach Intell, 2017, 14.
34. Liu, J., Li, Y., Song, S., Xing, J., Lan, C., Zeng, W., Multi-modality multi-task recurrent neural network for online action detection. IEEE Trans Circuits Syst Video Technol, 2018, 14.
35. Liu, J., Wang, G., Hu, P., Duan, L.-Y., Kot, A.C., Global context-aware attention LSTM networks for 3D action recognition. Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2017, 1647–1656.
36. Simonyan, K., Vedaldi, A., Zisserman, A., Deep inside convolutional networks: visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013, 1–8.
37. Clark, M.A., Lucett, S., Sutton, B.G., NASM essentials of personal fitness training, 2008, Lippincott Williams & Wilkins.
38. Chan, S., Luk, T., Hong, Y., Kinematic and electromyographic analysis of the push movement in Tai Chi. Br J Sports Med 37:4 (2003), 339–344.
39. Wei, S.-E., Ramakrishna, V., Kanade, T., Sheikh, Y., Convolutional pose machines. Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2016, 4724–4732.
40. Cao, Z., Simon, T., Wei, S.-E., Sheikh, Y., Realtime multi-person 2D pose estimation using part affinity fields. Proceedings of the CVPR, 2017.
41. Mehta, D., Rhodin, H., Casas, D., Fua, P., Sotnychenko, O., Xu, W., et al. Monocular 3D human pose estimation in the wild using improved CNN supervision. Proceedings of the International Conference on 3D Vision (3DV), 2017, IEEE, 506–516.
42. Mehta, D., Sridhar, S., Sotnychenko, O., Rhodin, H., Shafiei, M., Seidel, H.-P., et al. VNECT: real-time 3D human pose estimation with a single RGB camera. ACM Trans Grap (TOG), 36(4), 2017, 44.
43. Fernández-Delgado, M., Cernadas, E., Barro, S., Amorim, D., Do we need hundreds of classifiers to solve real world classification problems. J Mach Learn Res 15:1 (2014), 3133–3181.
44. Bishop, C.M., Pattern recognition and machine learning, 2006, Springer New York, Inc. 0-387-31073-8.
45. Müller, M., Information retrieval for music and motion, 2007, Springer.
46. Weston, J., Mukherjee, S., Chapelle, O., Pontil, M., Poggio, T., Vapnik, V., Feature selection for SVMs. Adv Neural Inf Process Syst (NIPS), 2000, 668–674.
47. Chen, Y.-W., Lin, C.-J., Combining svms with various feature selection strategies. Guyon, I., Nikravesh, M., Gunn, S., Zadeh, L.A., (eds.) Feature Extraction: Foundations and Applications, 207, 2006, Springer Berlin Heidelberg, Berlin, Heidelberg 978-3-540-35488-8, 315–324.
48. Guyon, I., Elisseeff, A., An introduction to variable and feature selection. J Mach Learn Res 3:Mar (2003), 1157–1182.
49. Genuer, R., Poggi, J.-M., Tuleau-Malot, C., Variable selection using random forests. Pattern Recognit Lett 31:14 (2010), 2225–2236.
50. Svetnik, V., Liaw, A., Tong, C., Wang, T., Application of breiman's random forest to modeling structure-activity relationships of pharmaceutical molecules. Proceedings of the International Workshop on Multiple Classifier Systems, 2004, Springer, 334–343.
51. Gregorutti, B., Michel, B., Saint-Pierre, P., Correlation and variable importance in random forests. Stat Comput 27:3 (2017), 659–678.
52. Breiman, L., Random forests. Mach Learn 45:1 (2001), 5–32.
53. Biau, G., Analysis of a random forests model. J Mach Learn Res 13:Apr (2012), 1063–1095.
54. Breiman, L., Friedman, J., Stone, C.J., Olshen, R.A., Classification and regression trees, 1984, CRC Press.
55. Bi, J., Bennett, K., Embrechts, M., Breneman, C., Song, M., Dimensionality reduction via sparse support vector machines. J Mach Learn Res 3:Mar (2003), 1229–1243.
56. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., et al. Scikit-learn: machine learning in Python. J Mach Learn Res 12 (2011), 2825–2830.
57. Hülsmann, F., Richter, A., Kopp, S., Botsch, M., Accurate online alignment of human motor performances. Proceedings of the ACM motion in games, 2017, ACM pp.7:1–7:6.
58. Goodfellow, I., Bengio, Y., Courville, A., Bengio, Y., Deep learning, 1, 2016, MIT Press Cambridge.
59. Zeiler, M.D., Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701, 2012, 1–6.
60. Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., et al. Tensorflow: a system for large-scale machine learning. OSDI, 16, 2016, 265–283.
61. Waltemate, T., Hülsmann, F., Pfeiffer, T., Kopp, S., Botsch, M., Realizing a low-latency virtual reality environment for motor learning. Proceedings of the 21st ACM symposium on virtual reality software and technology, 2015, 139–147.
62. Achenbach, J., Waltemate, T., Latoschik, M.E., Botsch, M., Fast generation of realistic virtual humans. Proceedings of the 23rd ACM symposium on virtual reality software and technology, 2017, ACM, 12.
63. Waltemate, T., Gall, D., Roth, D., Botsch, M., Latoschik, M.E., The impact of avatar personalization and immersion on virtual body ownership, presence, and emotional response. IEEE Trans Vis Comput Graph 24:4 (2018), 1643–1652.
64. Li, J., Cheng, K., Wang, S., Morstatter, F., Trevino, R.P., Tang, J., et al. Feature selection: a data perspective. ACM Comput Surv (CSUR), 50(6), 2017, 94.