Evaluation criteria for human-automation performance metrics

Performance Evaluation and Benchmarking of Intelligent Systems

Volumn , Issue , 2009, Pages 21-40

  Donmez, Birsen ^a   Pina, Patricia E ^a   Cummings, M L ^a  

^a Massachusetts Institute of Technology   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

BENCHMARKING; EFFICIENCY; INTELLIGENT SYSTEMS; QUALITY CONTROL;

CONSTRUCT VALIDITY; EVALUATION CRITERIA; EVALUATION EXPERIMENTS; MEASUREMENT TECHNIQUES; METRIC SELECTIONS; PERFORMANCE METRICS; STATISTICAL EFFICIENCY; SYSTEMATIC METHOD;

COST BENEFIT ANALYSIS;

EID: 84892300664     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1007/978-1-4419-0492-8_2     Document Type: Conference Paper

References (42)

1. T. B. Sheridan, Telerobotics, Automation, and Human Supervisory Control. Cambridge, MA: The MIT Press, 1992.
2. J. Scholtz, J. Young, J. L. Drury, and H. A. Yanco, "Evaluation of human-robot interaction awareness in search and rescue," in Proceedings of the IEEE International Conference on Robotics and Automation (ICRA). New Orleans, 2004.
3. N. J. Cooke, E. Salas, P. A. Kiekel, and B. Bell, "Advances in measuring team cognition," in Team Cognition: Understanding the Factors that Drive Process and Performance, E. Salas and S. M. Fiore, Eds. Washington, D. C.: American Psychological Association, 2004, pp. 83-106.
4. R. O. Olsen and M. A. Goodrich, "Metrics for evaluating human-robot interactions," in Proceedings of NIST Performance Metrics for Intelligent Systems Workshop, 2003.
5. P. E. Pina, M. L. Cummings, J. W. Crandall, and M. Della Penna, "Identifying generalizable metric classes to evaluate human-robot teams," in Proceedings of Metrics for Human-Robot Interaction Workshop at the 3rd Annual Conference on Human-Robot Interaction. Amsterdam, The Netherlands, 2008.
6. J. W. Crandall and M. L. Cummings, "Identifying predictive metrics for supervisory control of multiple robots," IEEE Transactions on Robotics-Special Issue on Human-Robot Interaction, vol. 23, pp. 942-951, 2007.
7. A. Steinfeld, T. Fong, D. Kaber, M. Lewis, J. Scholtz, A. Schultz, and M. A. Goodrich, "Common metrics for human-robot interaction," in Proceedings of the 1st Annual IEEE/ACM Conference on Human Robot Interaction (Salt Lake City, Utah). New York, NY: ACM Press, 2006.
8. C. D. Wickens, J. D. Lee, Y. Liu, and S. G. Becker, An Introduction to Human Factors Engineering, 2nd ed. Upper Saddle River, New Jersey: Pearson Education, Inc., 2004.
9. P. E. Pina, B. Donmez, and M. L. Cummings, Selecting Metrics to Evaluate Human Supervisory Control Applications, MIT Humans and Automation Laboratory, Cambridge, MA HAL2008-04, 2008.
10. M. A. Vidulich and E. R. Hughes, "Testing a subjective metric of situation awareness," in Proceedings of the Human Factors Society 35th Annual Meeting. Santa Monica, CA: The Human Factors and Ergonomics Society, 1991, pp. 1307-1311.
11. A. Chapanis, Research Techniques in Human Engineering. Baltimore: The Johns Hopkins Press, 1965.
12. M. S. Sanders and E. J. McCormick, Human Factors in Engineering and Design. New York: McGraw-Hill, 1993.
13. B. Donmez, L. Boyle, and J. D. Lee, "The impact of distractionmitigation strategies on driving performance," Human Factors, vol. 48, pp. 785-804, 2006.
14. R. M. Taylor, "Situational awareness rating technique (SART): the development of a tool for aircrew systems design," in Proceedings of the NATO Advisory Group for Aerospace Research and Development (AGARD) Situational Awareness in Aerospace Operations Symposium (AGARD-CP-478), 1989, p. 17.
15. F. T. Eggemeier, C. A. Shingledecker, and M. S. Crabtree, "Workload measurement in system design and evalution," in Proceeding of the Human Factors Society 29th Annual Meeting. Baltimore, MD, 1985, pp. 215-219.
16. F. T. Eggemeier, M. S. Crabtree, and P. A. LaPoint, "The effect of delayed report on subjective ratings of mental workload," in Proceedings of the Human Factors Society 27th Annual Meeting. Norfolk, VA, 1983, pp. 139-143.
17. S. Levin, D. J. France, R. Hemphill, I. Jones, K. Y. Chen, D. Ricard, R. Makowski, and D. Aronsky, "Tracking workload in the emergency department," Human Factors, vol. 48, pp. 526-539, 2006.
18. W. W. Wierwille and J. G. Casali, "A validated rating scale for global mental workload measurement applications," in Proceedings of the Human Factors Society 27th Annual Meeting. Santa Monica, CA, 1983, pp. 129-133.
19. S. G. Hart and L. E. Staveland, "Development of NASA-TLX (Task Load Index): results of empirical and theoretical research," in Human MentalWorkload, P. Hancock and N. Meshkati, Eds. Amsterdam, The Netherlands: North Holland B. V., 1988, pp. 139-183.
20. R. D. O'Donnell and F. T. Eggemeier, "Workload assessment methodology," in Handbook of Perception and Human Performance: Vol. II. Cognitive Processes and Performance, K. R. Boff, L. Kaufmann, and J. P. Thomas, Eds. New York: Wiley Interscience, 1986, pp. 42-1-42-49.
21. R. A. Johnson and D. W. Wichern, Applied Multivariate Statistical Analysis, 5th ed. NJ: Pearson Education, 2002.
22. T. B. Sheridan, Humans and Automation: System Design and Research Issues. NewYork, NY: John Wiley & Sons Inc., 2002.
23. M. E. Janzen and K. J. Vicente, "Attention allocation within the abstraction hierarchy," International Journal of Human-Computer Studies, vol. 48, pp. 521-545, 1998.
24. B. Donmez, L. Boyle, and J. D. Lee, "Safety implications of providing real-time feedback to distracted drivers," Accident Analysis & Prevention, vol. 39, pp. 581-590, 2007.
25. M. R. Endsley, B. Bolte, and D. G. Jones, Designing for Situation Awareness: An Approach to User-Centered Design. Boca Raton, FL: CRC Press, Taylor & Francis Group, 2003.
26. M. Buchin, "Assessing the impact of automated path planning aids in the maritime community," in Electrical Engineering and Computer Science M. Eng. Cambridge, MA: Massachusetts Institute of Technology, 2009.
27. C. D. Wickens and J. G. Hollands, Engineering Psychology and Human Performance, 3rd ed. New Jersey: Prentice Hall, 1999.
28. G. D. Ogden, J. M. Levine, and E. J. Eisner, "Measurement of workload by secondary tasks," Human Factors, vol. 21, pp. 529-548, 1979.
29. M. L. Cummings and S. Guerlain, "Using a chat interface as an embedded secondary tasking tool," in Proceedings of the 2nd Annual Human Performance, Situation Awareness, and Automation Technology Conference. Daytona Beach, FL, 2004.
30. A. H. Roscoe and G. A. Ellis, A Subjective Rating Scale for Assessing Pilot Workload in Flight: A Decade of Practical Use, Royal Aeronautical Establishment, Farnborough, England TR90019, 1990.
31. G. B. Reid and T. E. Nygren, "The subjective workload assessment technique: a scaling procedure for measuring mental workload," in Human Mental Workload, P. Hancock and N. Meshkati, Eds. Amsterdam, The Netherlands: North Holland, 1988, pp. 185-218.
32. U. Ahlstrom and F. Friedman-Berg, Subjective Workload Ratings and Eye Movement Activity Measures, US Department of Transportation, Federal Aviation Administration DOT/FAA/ACT-05/32, 2005.
33. A. J. Tattersall and G. R. J. Hockey, "Level of operator control and changes in heart rate variability during simulated flight maintenance," Human Factors, vol. 37, pp. 682-698, 1995.
34. C. Berka, D. J. Levendowski, M. Cventovic, M. M. Petrovic, G. F. Davis, M. N. Lumicao, M. V. Popovic, V. T. Zivkovic, R. E. Olmstead, and P. Westbrook, "Real-time analysis of EEG indices of alertness, cognition, and memory acquired with a wireless EEG headset," International Journal of Human Computer Interaction, vol. 17, pp. 151-170, 2004.
35. J. B. Brookings, G. F. Wilson, and C. R. Swain, "Psychophysiological responses to changes in workload during simulated air-traffic control," Biological Psychology, vol. 42, pp. 361-377, 1996.
36. K. A. Brookhuis and D. De Waard, "The use of psychophysiology to assess driver status," Ergonomics, vol. 36, 1993.
37. S. T. Iqbal, P. D. Adamczyk, S. Zheng, and B. P. Bailey, "Towards an index of opportunity: understanding changes in mental workload during task execution," in Proceedings of the ACM Conference on Human Factors in Computing Systems. Portland, Oregon, 2005, pp. 311-320.
38. R. Parasuraman and P. A. Hancock, "Adaptive control of mental workload," in Stress, Workload, and Fatigue, P. A. Hancock and P. A. Desmond, Eds. Mahwah, New Jersey: Lawrence Erlbaum Associates, Publishers, 2001, pp. 305-320.
39. D. A. Talluer and C. D. Wickens, "The effect of pilot visual scanning strategies on traffic detection accuracy and aircraft control," in Proceedings of the 12th International Symposium on Aviation Psychology. Dayton, OH, 2003.
40. M. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming. New Jersey: Wiley, 2005.
41. C. D. Wickens, J. Helleberg, J. Goh, Xu, X., andW. J. Horrey, Pilot TaskManagement: Testing and Attentional Expected Value Model of Visual Scanning, NASA Ames Research Center, Moffett Field, CA ARL-01-14/NASA-01-7, 2001.
42. S. Bruni, J. Marquez, A. S. Brzezinski, and M. L. Cummings, "Visualizing operators' cognitive strategies in multivariate optimization," in Proceedings of the Human Factors and Ergonomics Society's 50th Annual Meeting. San Francisco, CA, 2006.