Anticipating Human Activities Using Object Affordances for Reactive Robotic Response

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 38, Issue 1, 2016, Pages 14-29

  Koppula, Hema S ^a   Saxena, Ashutosh ^a  

^a Cornell University ^*  (United States)

Author keywords

3D Activity Understanding; Human Activity Anticipation; Machine Learning; RGBD Data; Robotics Perception

Indexed keywords

ARTIFICIAL INTELLIGENCE; LEARNING SYSTEMS; ROBOTICS;

3D ACTIVITY UNDERSTANDING; CONDITIONAL RANDOM FIELD; DETECTION ACCURACY; HUMAN ACTIVITIES; POSSIBLE FUTURES; REACTIVE ROBOTICS; RGBD DATA; SPATIAL TEMPORALS;

COMPUTER AIDED DESIGN;

ALGORITHM; ANTICIPATION; HUMAN; HUMAN ACTIVITIES; MACHINE LEARNING; MOVEMENT (PHYSIOLOGY); PERCEPTION; ROBOTICS; STATISTICAL MODEL; THREE DIMENSIONAL IMAGING; VIDEORECORDING;

ALGORITHMS; ANTICIPATION, PSYCHOLOGICAL; HUMAN ACTIVITIES; HUMANS; IMAGING, THREE-DIMENSIONAL; MACHINE LEARNING; MODELS, STATISTICAL; MOVEMENT; PERCEPTION; ROBOTICS; VIDEO RECORDING;

EID: 84961588783     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2015.2430335     Document Type: Article

References (83)

1. K. Tang, L. Fei-Fei, and D. Koller, "Learning latent temporal structure for complex event detection," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 1250-1257.
2. M. Rohrbach, S. Amin, M. Andriluka, and B. Schiele, "A database for fine grained activity detection of cooking activities," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 1194-1201.
3. H. Pirsiavash and D. Ramanan, "Detecting activities of daily living in first-person camera views," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 2847-2854.
4. J.-K. Min and S.-B. Cho, "Activity recognition based on wearable sensors using selection/fusion hybrid ensemble," in Proc. IEEE Int. Conf. Syst. Man, Cybern., 2011, pp. 1319-1324.
5. H. S. Koppula, R. Gupta, and A. Saxena, "Learning human activities and object affordances from RGB-D videos," Int. J. Robot. Res., vol. 32, pp. 951-970, 2013.
6. J. Sung, C. Ponce, B. Selman, and A. Saxena, "Unstructured human activity detection from RGBD images," in Proc. IEEE Int. Conf. Robot. Autom., 2012, pp. 842-849.
7. B. Ni, G. Wang, and P. Moulin, "RGBD-HuDaAct: A color-depth video database for human daily activity recognition," in Proc. IEEE Int. Conf. Comput. Vis. Workshop, 2011, pp. 1147-1153.
8. E. Guizzo and E. Ackerman, "The rise of the robot worker," IEEE Spectr., vol. 49, no. 10, pp. 34-41, Oct. 2012.
9. S. Nikolaidis and J. Shah, "Human-robot cross-training: Computational formulation, modeling and evaluation of a human team training strategy," in Proc. IEEE 8th Int. Conf. Human-Robot Interact., 2013, pp. 33-40.
10. J. Gibson, The Ecological Approach to Visual Perception. Boston, MA, USA: Houghton Mifflin, 1979.
11. S. Maji, L. Bourdev, and J. Malik, "Action recognition from a distributed representation of pose and appearance," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2011, pp. 3177-3184.
12. Z. Xing, J. Pei, G. Dong, and P. S. Yu, "Mining sequence classifiers for early prediction," in Proc. SIAM Int. Conf. Data Mining, 2008, pp. 644-655.
13. I. Laptev, M. Marszalek, C. Schmid, and B. Rozenfeld, "Learning realistic human actions from movies," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2008, pp. 1-8.
14. J. Niebles, C. Chen, and L. Fei-fei, "Modeling temporal structure of decomposable motion segments for activity classification," in Proc. Eur. Conf. Comput. Vis., 2010, pp. 392-405.
15. A. Gaidon, Z. Harchaoui, and C. Schmid, "Actom sequence models for efficient action detection," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2011, pp. 3201-3208.
16. B. Laxton, L. Jongwoo, and D. Kriegman, "Leveraging temporal, contextual and ordering constraints for recognizing complex activities in video," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2007, pp. 1-8.
17. R. Chaudhry, A. Ravichandran, G. Hager, and R. Vidal, "Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2009, pp. 1932-1939.
18. J. Liu, B. Kuipers, and S. Savarese, "Recognizing human actions by attributes," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2011, pp. 3337-3344.
19. S. Sadanand and J. Corso, "Action bank: A high-level representation of activity in video," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 1234-1241.
20. M. Rohrbach, M. Regneri, M. A., S. Amin, M. Pinkal, and B. Schiele, "Script data for attribute-based recognition of composite activities," in Proc. Eur. Conf. Comput. Vis., 2012, pp. 144-157.
21. Y. Fu, T. M. Hospedales, T. Xiang, and S. Gong, "Attribute learning for understanding unstructured social activity," in Proc. Eur. Conf. Comput. Vis., 2012, pp. 530-543.
22. M. Ryoo, "Human activity prediction: Early recognition of ongoing activities from streaming videos," in Proc. Int. Conf. Comput. Vis., 2011, pp. 1036-1043.
23. M. Hoai and F. De la Torre, "Max-margin early event detectors," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 2863-2870.
24. J. K. Aggarwal and M. S. Ryoo, "Human activity analysis: A review," ACM Comput. Surveys, vol. 43, no. 3, article 16, 2011.
25. W. Yang, Y. Wang, and G. Mori, "Recognizing human actions from still images with latent poses," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2010, pp. 2030-2037.
26. A. Gupta, A. Kembhavi, and L. S. Davis, "Observing humanobject interactions: Using spatial and functional compatibility for recognition," IEEE Trans. Pattern Analysis Mach. Intell., vol. 31, no. 10, pp. 1775-1789, Oct. 2009.
27. B. Yao and L. Fei-Fei, "Modeling mutual context of object and human pose in human-object interaction activities," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2010, pp. 17-24.
28. Y. Jiang, M. Lim, and A. Saxena, "Learning object arrangements in 3d scenes using human context," in Proc. 29th Int. Conf. Mach. Learn., 2012, pp. 1543-1550.
29. S. Tran and L. S. Davis, "Event modeling and recognition using Markov logic networks," in Proc. 10th Eur. Conf. Comput. Vis., 2008, pp. 610-623.
30. H. S. Koppula, A. Anand, T. Joachims, and A. Saxena, "Semantic labeling of 3d point clouds for indoor scenes," in Proc. Conf. Neural Inform. Process. Syst., 2011, pp. 244-252.
31. Z. Jia, A. Gallagher, A. Saxena, and T. Chen, "3d-based reasoning with blocks, support, and stability," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 1-8.
32. J. Shotton, R. Girshick, A. Fitzgibbon, T. Sharp, M. Cook, M. Finocchio, R. Moore, P. Kohli, A. Criminisi, A. Kipman, and A. Blake, "Efficient human pose estimation from single depth images," IEEE Trans. Pattern Analysis Mach. Intell., vol. 35, no. 12, pp. 2821-2840, Dec. 2013.
33. D. L. Ly, A. Saxena, and H. Lipson, "Co-evolutionary predictors for kinematic pose inference from rgbd images," in Proc. 14th Int. Conf. Genetic Evol. Comput. Conf., 2012, pp. 967-974.
34. H. Zhang and L. E. Parker, "4-dimensional local spatio-temporal features for human activity recognition," in Proc. Int. Conf. Intell. Robot Syst., 2011, pp. 2044-2049.
35. O. Duchenne, I. Laptev, J. Sivic, F. Bach, and J. Ponce, "Automatic annotation of human actions in video," in Proc. Int. Conf. Comput. Vis., 2009, pp. 1491-1498.
36. S. Satkin and M. Hebert, "Modeling the temporal extent of actions," in Proc. Eur. Conf. Comput. Vis., 2010, pp. 536-548.
37. Y. Ke, R. Sukthankar, and M. Hebert, "Event detection in crowded videos," in Proc. Int. Conf. Comput. Vis, Oct. 2007, pp. 1-8.
38. M. H. Nguyen, L. Torresani, F. De la Torre, and C. Rother, "Weakly supervised discriminative localization and classification: A joint learning process," in Proc. Int. Conf. Comput. Vis., 2009, pp. 1925-1932.
39. T. Simon, M. H. Nguyen, F. De la Torre, and J. F. Cohn, "Action unit detection with segment-based SVMs," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2010, 2737-2744.
40. X. Xuan and K. Murphy, "Modeling changing dependency structure in multivariate time series," in Proc. Int. Conf. Mach Learn., 2007, pp. 1055-1062.
41. Z. Harchaoui, F. Bach, and E. Moulines, "Kernel change-point analysis," in Proc. Neural Inform. Process. Syst., 2008, pp. 609-616.
42. S. Oh, J. Rehg, T. Balch, and F. Dellaert, "Learning and inferring motion patterns using parametric segmental switching linear dynamic systems," Int. J. Comput. Vis., vol. 77, pp. 103-124, 2008.
43. E. B. Fox, E. B. Sudderth, M. I. Jordan, and A. S. Willsky, "Nonparametric Bayesian learning of switching linear dynamical systems," in Proc. Neural Inform. Process. Syst. 21, 2009, pp. 457-464.
44. M. Hoai, Z. Lan, and F. De la Torre, "Joint segmentation and classification of human actions in video," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2011, pp. 3265-3272.
45. M. Hoai and F. De la Torre, "Maximum margin temporal clustering," in Proc. Int. Conf. Artif. Intell. Statist., 2012, pp. 520-528.
46. Y. Zhu, N. M. Nayak, and A. K. Roy-Chowdhury, "Context-aware modeling and recognition of activities in video," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 2491-2498.
47. Z. Wang, Q. Shi, C. Shen, and A. van den Hengel, "Bilinear programming for human activity recognition with unknown MRF graphs," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 1690-1697.
48. B. D. Ziebart, N. Ratliff, G. Gallagher, C. Mertz, K. Peterson, J. A. Bagnell, M. Hebert, A. Dey, and S. Srinivasa, "Planning-based prediction for pedestrians," in Proc. Int. Conf. Intell. Robots Syst., 2009, pp. 3931-3936.
49. K. Kitani, B. D. Ziebart, J. A. Bagnell, and M. Hebert, "Activity forecasting," in Proc. Eur. Conf. Comput. Vis., 2012, pp. 201-214.
50. M. Kuderer, H. Kretzschmar, C. Sprunk, and W. Burgard, "Feature-based prediction of trajectories for socially compliant navigation," presented at the Conf. Robot. Sci. Syst., Sydney, NSW, Australia, 2012.
51. Z. Wang, M. Deisenroth, H. B. Amor, D. Vogt, B. Scholkopf, and J. Peters, "Probabilistic modeling of human movements for intention inference," presented at the Conf. Robot. Sci. Syst., Sydney, NSW, Australia, 2012.
52. A. Dragan and S. Srinivasa, "Formalizing assistive teleoperation," presented at the Conf. Robot. Sci. Syst., Sydney, NSW, Australia, 2012.
53. S. Hongeng and R. Nevatia, "Large-scale event detection using semi-hidden Markov models," in Proc. Int. Conf. Comput. Vis., 2003, pp. 1455-1462.
54. P. Natarajan and R. Nevatia, "Coupled hidden semi Markov models for activity recognition," presented at the IEEE Workshop Motion Video Comput., Austin, TX, USA, 2007.
55. S. Gong and T. Xiang, "Recognition of group activities using dynamic probabilistic networks," in Proc. Int. Conf. Comput. Vis., 2003, pp. 742-749.
56. A. Quattoni, S. Wang, L.-P. Morency, M. Collins, and T. Darrell, "Hidden-state conditional random fields," IEEE Trans. Pattern Analysis Mach. Intell., vol. 29, no. 10, pp. 1848-1852, Oct. 2007.
57. C. Sminchisescu, A. Kanaujia, Z. Li, and D. Metaxas, "Conditional models for contextual human motion recognition," in Proc. Int. Conf. Comput. Vis., 2005, pp. 1808-1815.
58. S. Sarawagi and W. W. Cohen, "Semi-Markov conditional random fields for information extraction," in Proc. Neural Inform. Process. Syst., 2004, pp. 1185-1192.
59. Q. Shi, L. Wang, L. Cheng, and A. Smola, "Human action segmentation and recognition using discriminative semi-Markov models," Int. J. Comput. Vis., vol. 93, pp. 22-32, 2011.
60. Z. Khan, T. Balch, and F. Dellaert, "MCMC-based particle filtering for tracking a variable number of interacting targets," IEEE Trans. Pattern Analysis Mach. Intell., vol. 27, no. 11, pp. 1805-1819, Nov. 2005.
61. R. Hess and A. Fern, "Discriminatively trained particle filters for complex multi-object tracking," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2009, pp. 240-247.
62. D. Fox, "Kld-sampling: Adaptive particle filters," presented at the Neural Inform. Process. Syst., Vancouver, BC, USA, 2001.
63. P. Jensfelt, D. Austin, O. Wijk, and M. Andersson, "Feature based condensation for mobile robot localization," in Proc. IEEE Int. Conf. Robot. Autom., 2000, pp. 2531-2537.
64. D. Schulz, D. Fox, and J. Hightower, "People tracking with anonymous and ID-sensors using Rao-Blackwellised particle filters," in Proc. Int. Joint Conf. Artif. Intell., 2003, pp. 921-926.
65. A. Doucet, N. d. Freitas, K. Murphy, and S. Russell, "Rao-blackwellised particle filtering for dynamic Bayesian networks," in Proc. 16th Conf. Uncertainty Artif. Intell., 2000, pp. 176-183.
66. F. Gustafsson, F. Gunnarsson, N. Bergman, U. Forssell, J. Jansson, R. Karlsson, and P.-J. Nordlund, "Particle filters for positioning, navigation, and tracking," IEEE Trans. Signal Process., vol. 50, no. 2, pp. 425-437, Feb. 2002.
67. M. Montemerlo, S. Thrun, D. Koller, and B. Wegbreit, "Fastslam: A factored solution to the simultaneous localization and mapping problem," in Proc. AAAI Nat. Conf. Artif. Intell., 2002, pp. 593-598.
68. Openni [Online]. Available: https://github.com/OpenNI/ OpenNI, 2012.
69. M. Montemerlo, S. Thrun, and W. Whittaker, "Conditional particle filters for simultaneous mobile robot localization and peopletracking," in Proc. Int. Conf. Robot. Autom., 2002, pp. 695-701.
70. H. Kjellstrom, J. Romero, and D. Kragic, "Visual object-action recognition: Inferring object affordances from human demonstration," Comput. Vis. Image Understanding, vol. 115, no. 1, pp. 81-90, 2011.
71. J. Sun, J. L. Moore, A. Bobick, and J. M. Rehg, "Learning visual object categories for robot affordance prediction," Int. J. Robotic Res., vol. 29, pp. 174-197, 2009.
72. T. Hermans, J. M. Rehg, and A. Bobick, "Affordance prediction via learned object attributes," presented at the Int. Conf. Robot. Autom, Workshop Semantic Perception, Mapping, Exploration, Shanghai, China, 2011.
73. H. Grabner, J. Gall, and L. Van Gool, "What makes a chair a chair" in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2011, pp. 1529-1536.
74. A. Gupta, S. Satkin, A. Efros, and M. Hebert, "From 3d scene geometry to human workspace," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2011, pp. 1961-1968.
75. V. Delaitre, D. Fouhey, I. Laptev, J. Sivic, A. Gupta, and A. Efros, "Scene semantics from long-term observation of people," in Proc. Eur. Conf. Comput. Vis., 2012, pp. 284-298.
76. Y. Jiang, H. S. Koppula, and A. Saxena, "Hallucinated humans as the hidden context for labeling 3d scenes," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 2993-3000.
77. J. J. Faraway, M. P. Reed, and J. Wang, "Modelling three-dimensional trajectories by using Bezier curves with application to hand motion," JRSS Ser. C, vol. 56, pp. 571-585, 2007.
78. H. S. Koppula, A. Anand, T. Joachims, and A. Saxena, "Semantic labeling of 3d point clouds for indoor scenes," in Proc. Neural Inform. Process. Syst., 2011, pp. 244-252.
79. C. Rother, V. Kolmogorov, V. Lempitsky, and M. Szummer, "Optimizing binary mrfs via extended roof duality," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2007, pp. 1-8.
80. P. Hammer, P. Hansen, and B. Simeone, "Roof duality, complementation and persistency in quadratic 0-1 optimization," Math. Prog., vol. 28, no. 2, pp. 121-155, 1984.
81. P. F. Felzenszwalb and D. Huttenlocher, "Efficient graph-based image segmentation," Int. J. Comput. Vis., vol. 59, no. 2, pp. 167-181, 2004.
82. T. Joachims, T. Finley, and C. Yu, "Cutting-plane training of structural SVMs," Mach. Learn., vol. 77, no. 1, pp. 27-59, 2009.
83. R. Diankov, "Automated construction of robotic manipulation programs," Ph.D. dissertation, Robotics Institute, CMU, Pittsburgh, PA, USA, Aug. 2010.