A survey of quantitative team performance metrics for human-robot collaboration

41st International Conference on Environmental Systems 2011, ICES 2011

Volumn , Issue , 2011, Pages

  Singer, Sharon M ^a   Akin, David L ^a  

^a UNIVERSITY OF MARYLAND   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

SPACE RESEARCH; SURVEYS;

HUMAN-ROBOT COLLABORATION; PERFORMANCE MODEL; ROBOT TEAMS; ROBOTIC SPACE EXPLORATION; TEAM CONFIGURATION; TEAM MEMBERS; TEAM PERFORMANCE;

ROBOTS;

EID: 85085719817     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.2514/6.2011-5248     Document Type: Conference Paper

References (71)

1. J.A. Arnold. Towards a framework for architecting heterogeneous teams of humans and robots for space exploration. Master's thesis, Massachusetts Institute of Technology, 2006.
2. A. Bechar, Y. Edan, and J. Meyer. Optimal collaboration in human-robot target recognition systems. In Systems, Man and Cybernetics, 2006. SMC'06. IEEE International Conference on, volume 5, pages 4243-4248. IEEE, 2007.
3. A. Bechar, J. Meyer, and Y. Edan. An objective funcion to evaluate performance of human-robot collaboration in target recognition tasks. Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on, 39(6):611-620, 2009.
4. L. Billman and M. Steinberg. Human system performance metrics for evaluation of mixed-initiative heterogneous autonomous systems. In Proceedings of the 2007 Workshop on Performance Metrics for Intelligent Systems, pages 120-126. ACM, August 2007.
5. P. Bobko, P. Roth, and M. Buster. the usefulness of unit weights in creating composite scores: a literature review, application to content validity, and meta-analysis. Organizational Research Methods, 10(4):689-709, 2007.
6. D.J. Bruemmer, D.A. Few, R.L. Boring, J.L. Marble, M.C. Walton, and C.W. Nielsen. Shared understanding for collaborative control. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Sytems and Humans, 35(4):494-504, July 2005.
7. J.L. Burke, R.R. Murphy, D.R. Riddle, and T. Fincannon. Task performance metrics in human-robot interaction: Taking a systems approach. Performance Metrics for Intelligent Systems, 2004.
8. J.W. Crandall and M.L. Cummings. Developing performance metrics for the supervisory control of multiple robots. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction pages 33-40. ACM, 2007.
9. J.W. Crandall and ML Cummings. A predictive model for human-unmanned vehicles systems. Technical Report HAL2008- 05, Massachusetts Institute of Technology, 2008.
10. J.W. Crandall, M.A. Goodrich, D.R. Olsen, and C.W. Nielsen. Validating human-robot interaction schemes in multi-tasking environments. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Sytems and Humans, 35(4):438-449, July 2005.
11. J.W. Crandall, C.W. Nielsen, and M.A. Goodrich. Towards predicting robot team performance. In Systems, Man and Cybernetics, 2003 IEEE International Conference on, volume 1, pages 906-911. IEEE,2003.
12. ML Cummings, P. Pina, and J.W. Crandall. A metric taxonomy for supervisory control of unmanned vehicles. In AUVSI, 2008.
13. K. Dautenhahn and I. Werry. A quantitative technique for analysing robot-human interactions. In Proceedings of the 2002 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2002.
14. E. De Visser, R. Parasuraman, A. Freedy, E. Freedy, and G. Weltman. A comprehensive methodology for assessing human-robot team performance for use in training and simulation. In Human Factors and Ergonomics Society Annual Meeting Proceedings, volume 50, pages 2639-2643. Human Factors and Ergonomics Society, 2006.
15. B. Donmez, P. Pina, and M.L. Cummings. Evaluation criteria for human-automation performance metrics. In Proceedings of the Performance Metrics for Intelligent Systems Workshop, 2008.
16. M.R. Elara, C.A.A. Calderon, C. Zhou, and W.S. Wijesoma. False alarm demand: A new metric for measuring robot performance in human robot teams. In Autonomous Robots and Agents, 2009. ICARA 2009. 4th International Conference on, pages 436-441. IEEE, 2009.
17. A. Elfes, C.R. Weisbin, H. Hua, J.H. Smith, J. Mrozinski, and K. Shelton. The huron task allocation and scheduling system: Planning human and robot activities for lunar missions. In Automation Congress, 2008. WAC 2008. World, pages 1-8. IEEE, 2008.
18. C.A. Ellis, S.J. Gibbs, and G. Rein. Groupware: some issues and experiences. Communications of the ACM, 34(1):39-58, 1991.
19. T. Fong, A. Abercromby, M.G. Bualat, M.C. Deans, K.V. Hodges, J.M. Hurtado, R. Landis, P. Lee, and D. Schreck- enghost. Assessment of robotic recon for human exploration on the moon. Acta Astronautica, 67(9-10):1176-1188, November- December 2010.
20. T. Fong, I. Nourbakhsh, C. Kunz, L. Fluckiger, J. Schreiner, R. Ambrose, R. Burridge, R. Simmons, L.M. Hiatt, A. Schultz, et al. The peer-to-peer human-robot interaction project. Space, 6750(AIAA 2005-6750), September 2005.
21. T. Fong, J. Scholtz, J.A. Shah, L. Fluckiger, C. Kunz, D. Lees, J. Schreiner, M. Siegel, L.M. Hiatt, and I. Nourbakhsh. A preliminary study of peer-to-peer human-robot interaction. In Systems, Man and Cybernetics, 2006. SMC'06. IEEE International Conference on, volume 4, pages 3198-3203. IEEE, 2007.
22. A. Freedy, E. DeVisser, G. Weltman, and N. Coeyman. Measurement of trust in human-robot collaboration. In Collaborative Technologies and Systems, 2007. CTS 2007. International Symposium on, pages 106-114. IEEE, 2008.
23. B.P. Gerkey and M.J. Mataric. A formal analysis and taxonomy of task allocation in multi-robot systems. International Journal of Robotics Research, 23(9):939-954, 2004.
24. D.F. Glas, T. Kanda, H. Ishiguro, and N. Hagita. Simultaneous teleoperation of multiple social robots. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction, pages 311-318. ACM, 2008.
25. M. Goodrich and D. Olsen. Seven principles of efficient human robot interaction. In IEEE International Conference on Systems, Man, and Cybernetics, volume 4, pages 3943-3948, 2003.
26. M.A. Goodrich, T.W. McLain, J.D. Anderson, J. Sun, and J.W. Crandall. Managing autonomy in robot teams: observations from four experiments. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction, pages 25-32. ACM, March 2007.
27. S.G. Hart and L.E. Staveland. Development of nasa-tlx (task load index): Results of empirical and theoretical research. Human Mental Workload, 1:139-183, 1988.
28. F. Heger and S. Singh. Sliding autonomy for complex coordinated multi-robot tasks: analysis and experiments. In Proceedings of Robotics: Science and Systems, August 2006.
29. A. Howard. A synergistic approach for maximizing human-automation system performance. Draper fiscal year 2006 final report, Georgia Institute of Technology, July 2006.
30. A. Howard. A systematic approach to predict performance of human-automation systems. IEEE Transactions on Systems, Man, and Cybernetics - Part C: Applications and Reviews, 37(4):594-601, July 2007.
31. A. Howard and G. Cruz. Adapting human leadership approaches for role allocation in human-robot navigation scenarios. In Automation Congress, 2006. WAC'06. World, pages 1-8. IEEE, 2007.
32. J.H. Hwang, K.W. Lee, and D.S. Kwon. A formal method of measuring interactivity in hri. In Robot and Human interactive Communication, 2007. RO-MAN 2007. The 16th IEEE International Symposium on, pages 738-743. IEEE, 2007.
33. B. Kannan and L.E. Parker. Fault-tolerance based metrics for evaluating system performance in multi-robot teams. In Proceedings of Performance Metrics for Intelligent Systems Workshop, 2006.
34. T. Kaupp and A. Makarenko. Measuring human-robot team effectiveness to determine an appropriate autonomy level. In Robotics and Automation, 2008. ICRA 2008. IEEE International Conference on, pages 2146-2151. IEEE, 2008.
35. T. Kaupp, A. Makarenko, and H. Durrant-Whyte. Human-robot communication for collaborative decision making - a probabilistic approach. Robotics and Autonomous Systems, 58:444-456, 2010.
36. J. Keller. Human performance modeling for discrete-event simulation: human performance modeling for discrete-event simulation: workload. In Proceedings of the 34th Conference on Winter Simulation: Exploring New Frontiers, pages 157-162, Winter Simulation Conference, 2002.
37. A. Lampe and R. Chatila. Performance measure for the evaluation of mobile robot autonomy. Proceedings of the 2006 IEEE International Conference on Robotics and Automation, May 2006.
38. I.S. MacKenzie. A note on the information-theoretic basis for fitts' law. Journal of Motor Behavior, 1989.
39. G.A. Mann. Quantitative evaluation of human-robot options for maintenance tasks during analogue surface operations. In C. Pain, editor, Proceedings of the 8th Australian Mars Exploration Conference, pages 26-34, 2008.
40. J.L. Marble, D.J. Bruemmer, D.A. Few, and D.D. Dudenhoeffer. Evaluation of supervisory vs. peer-peer interaction with human-robot teams. In Proceedings of the 37th Annual Hawaii International Conference on Systems Sciences, pages 4-8, 2004.
41. C.A. Miller and R. Parasuraman. Who's in charge?: Intermediate levels of control for robots we can live with. Systems, Man, and Cybernetics, 2003. IEEE International Conference on, 1:462-467, 2003.
42. C. E. Nehme. Modeling Human Supervisory Control in Heterogeneous Unmanned Vehicle Systems. PhD thesis, Massachusetts Institute of Technology, February 2009.
43. U. Nehmzow. Quantitative analysis of robot-environment interaction-towards scientific mobile robotics. Robotics and Autonomous Systems, 44:55-68, 2003.
44. D.R. Olsen and M.A. Goodrich. Metrics for evaluating human-robot interactions. In Proceedings of PERMIS, 2003.
45. D.R. Olsen, B. Wood, and J. Turner. Metrics for human driving of multiple robots. In International Conference on Robotics and Automation. IEEE, 2004.
46. D.R. Olsen and S.B. Wood. Fan-out: measuring human control of multiple robots. In Proceedings of the Special Interest Group on Computer Human Interaction Conference on Human Factors in Computing Systems, pages 231-238. ACM, 2004.
47. Y. Oren. Performance analysis of human-robot cooperation in target recognition tasks. Master's thesis, Ben-Gurion University of the Negev, Aug. 2008.
48. R. Parasuraman. Designing automation for human use: empirical studies and quantitative models. Ergonomics, 43(7):931-951, July 2000.
49. R. Parasuraman, T.B. Sheridan, and C.D. Wickens. A model for types and levels of human interaction with automation. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Sytems and Humans, 30(3):286-297, May 2000.
50. P. Pina, ML Cummings, J.W. Crandall, and M.D. Penna. Identifying generalizable metric classes to evaluate human-robot teams. In 3rd Annual Conference on Human-Robot Interaction: Metrics for Human-Robot Interaction Workshop, 2008.
51. P. Pina, B. Donmez, and M.L. Cummings. Selecting metrics to evaluate human supervisory control applications. HAL Report HAL2008-04, Humans and Automation Laboratory, Massachusetts Institute of Technology, 2008.
52. S.S. Ponda, H.L. Choi, and J.P. How. Predictive planning for heterogeneous human-robot teams. AIAA Infotech@ Aerospace, 2010.
53. M. Prewett, R. Johnson, K. Saboe, L. Elliott, and M. Coovert. Managing workload in human-robot interaction: a review of empirical studies. Computers in Human Behavior, 26(5):840-856, September 2010.
54. G. Rodriguez and C.R. Weisbin. A new method to evaluate human-robot system performance. Autonomous Robots, 14(2-3):165-178, 2003.
55. F. Rohrmuller, O. Kourakos, M. Rambow, D. Brscic, D. Wollherr, S. Hirche, and M. Buss. Interconnected performance optimization in complex robotic systems. In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 4113-4118. IEEE, 2010.
56. J.A. Saleh and F. Karray. Towards generalized performance metrics for human-robot interaction. In Autonomous and Intelligent Systems, 2010 International Conference on, pages 1-6. IEEE, 2010.
57. J. Scholtz. Theory and evaluation of human robot interactions. In Proceedings of the 36th Hawaii International Conference on System Sciences, 2002.
58. J. Scholtz, B. Antonishek, and J.D. Young. Implementation of a situation awareness assessment tool for evaluation of human-robot interfaces. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Sytems and Humans, 35(4):450-459, July 2005.
59. D. Schreckenghost, T. Fong, H. Utz, and T. Milam. Measuring robot performance in real-time for nasa robotic reconnaissance operations. In Proceedings of NIST PERMIS Workshop, 2009.
60. D. Schreckenghost, T. Milam, and T. Fong. Ai space odyssey: Measuring performance in real time during remote human-robot operations with adjustable autonomy. IEEE Intelligent Systems, 25(5):36-44, 2010.
61. J.A. Shah, J.H. Saleh, and J.A. Hoffman. Review and synthesis of considerations in architecting heterogeneous teams of humans and robots for optimal space exploration. IEEE Transactions on Systems, Man, and Cybernetics - Part C: Applications and Reviews, 37(5):779-, September 2007.
62. J.A. Shah, J.H. Saleh, and J.A. Hoffman. Analytical basis for evaluating the effect of unplanned interventions on the effectiveness of a human-robot system. Reliability Engineering and System Safety, 93(8):1280-1286, 2008.
63. S.M. Singer and D. Akin. Role definition and task allocation for a cooperative EVA and robotic team. SAE International, 2009.
64. B. Stanton, B. Antonishek, and J. Scholtz. Development of an evaluation method for acceptable usability. In Proceedings of PERMIS, 2006.
65. A. Steinfeld, T. Fong, D. Kaber, M. Lewis, J. Scholtz, A. Schultz, and M. Goodrich. Common metrics for humanrobot interaction. In Proceedings of the 1st ACM SIGCHI/SIGART Conference on Human-Robot Interaction, pages 33-40 Association for Computing Machinery, 2006.
66. J.G. Trafton, N.L. Cassimastis, M.D. Bugajska, D.P. Brock, F.E. Mintz, and A. Schultz. Enabling effective human-robot interaction using perspective-taking in robots. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Sytems and Humans, 35(4):460-470, July 2005.
67. E. Tunstel. Operational performance metrics for mars exploration rovers. Journal of Field Robotics, 24(8-9):651-670, 2007.
68. H. Wang, M. Lewis, P. Velagapudi, P. Scerri, and K. Sycara. How search and its subtasks scale in n robots. In Proceedings of the 4th ACM/IEEE International Conference on Human Robot Interaction, 2009.
69. J. Wang and M. Lewis. Assessing cooperation in human control of heterogeneous robots. In Proceedings of the 3rd ACM/IEEE International Conference on Human Robot Interaction, March 2008.
70. H.A. Yanco and J.L. Drury. A taxonomy for human-robot interaction. In Proceedings of the AAAI Fall Symposium on Human-Robot Interaction, 2002.
71. H.A. Yanco, J.L. Drury, and J. Scholtz. Beyond usability evaluation: Analysis of human-robot interaction at a major robotics competition. Human-Computer Interaction, 19(1):117-149, 2009.