Evaluation of an integrated multi-task machine learning system with humans in the loop

Performance Metrics for Intelligent Systems (PerMIS) Workshop

Volumn , Issue , 2007, Pages 182-188

  Steinfeld, Aaron ^a   Bennett, S Rachael ^a   Cunningham, Kyle ^a   Lahut, Matt ^a   Quinones, Pablo Alejandro ^a   Wexler, Django ^a   Siewiorek, Dan ^a   Hayes, Jordan ^b   Cohen, Paul ^c   Fitzgerald, Julie ^d   Hansson, Othar ^b   Pool, Mike ^e   Drummond, Mark ^f  

^a Carnegie Mellon University   (United States)

^b Bitway Inc

^c UNIVERSITY OF SOUTHERN CALIFORNIA   (United States)

^d JSF Consulting

^e IET ^*

^f SRI INTERNATIONAL   (United States)

Author keywords

Evaluation; Intelligent systems; Machine learning; Mixed initiative assistants

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPUTATIONAL LINGUISTICS; DETECTORS; EDUCATION; INTELLIGENT CONTROL; INTELLIGENT SYSTEMS; MACHINE COMPONENTS; NATURAL LANGUAGE PROCESSING SYSTEMS; RADAR; RADIO; ROBOT LEARNING;

(I ,J) CONDITIONS; HUMAN USERS; MACHINE-LEARNING; MULTI-TASK; NATURAL LANGUAGE PROCESSING (NLP); OVERALL PERFORMANCE; PERFORMANCE METRICS; REAL WORLD SETTING; SYSTEM PERFORMANCES;

LEARNING SYSTEMS;

EID: 45349083234     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (11)

1. Steinfeld, A., Bennett, R., Cunningham, K., Lahut, M., Quinones, P.-A., Wexler, D., Siewiorek, D., Cohen, P., Fitzgerald, J., Hansson, O., Hayes, J., Pool, M., and Drummond, M., The RADAR Test Methodology: Evaluating a Multi-Task Machine Learning System with Humans in the Loop. 2006, Carnegie Mellon University, School of Computer Science: Pittsburgh, PA. http://reports-archive. adm.cs.cmu.edu/anon/2006/abstracts/06-125.html
2. Clymer, J. R. Simulation of a vehicle traffic control network using a fuzzy classifier system. In Proc. of the IEEE Simulation Symposium. 2002.
3. Clymer, J. R. and Harrsion, V. Simulation of air traffic control at a VFR airport using OpEMCSS. In Proc. IEEE Digital Avionics Systems Conference. 2002.
4. Zhang, L., Samaras, D., Tomasi, D., Volkow, N., and Goldstein, R. Machine learning for clinical diagnosis from functional magnetic resonance imaging. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2005.
5. Hu, Y., Li, H., Cao, Y., Meyerzon, D., and Zheng, Q. Automatic extraction of titles from general documents using machine learning. In Proc. of ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL). 2005.
6. Allen, J., Chambers, N., Ferguson, G., Galescu, L., Jung, H., Swift, M., and Taysom, W. PLOW: A Collaborative Task Learning Agent. In Proc. Conference on Artificial Intelligence (AAAI). 2007. Vancouver, Canada.
7. Schrag, R., Pool, M., Chaudhri, V., Kahlert, R., Powers, J., Cohen, P., Fitzgerald, J., and Mishra, S. Experimental evaluation of subject matter expert-oriented knowledge base authoring tools. In Proc. NIST Performance Metrics for Intelligent Systems Workshop. 2002. http://www.iet.com/Projects/RKF/ PerMIS02.doc
8. Shen, J., Li, L., Dietterich, T. G., and Herlocker, J. L. A hybrid learning system for recognizing user tasks from desktop activities and email messages. In Proc. International Conference on Intelligent User Interfaces (IUI). 2006.
9. Yoo, J., Gervasio, M., and Langley, P. An adaptive stock tracker for personalized trading advice. In Proc. International Conference on Intelligent User Interfaces (IUI). 2003.
10. Airspace: Tools for evaluating complex systems, machine language, and complex tasks. http://www.cs.cmu.edu/~airspace
11. Steinfeld, A., Quinones, P.-A., Zimmerman, J., Bennett, S. R., and Siewiorek, D. Survey measures for evaluation of cognitive assistants. In Proc. NIST Performance Metrics for Intelligent Systems Workshop (PerMIS). 2007.