Power to the people: The role of humans in interactive machine learning

AI Magazine

Volumn 35, Issue 4, 2014, Pages 105-120

  Amershi, Saleema ^a   Cakmak, Maya ^b   Knox, W Bradley ^c   Kulesza, Todd ^d  

^a Computer Human Interactive Learning ^*  (United States)

^b UNIVERSITY OF WASHINGTON   (United States)

^c MIT MEDIA LAB   (United States)

^d OREGON STATE UNIVERSITY   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COUPLINGS;

CASE-STUDIES; EFFECTIVE LEARNING; END USERS; EXISTING SYSTEMS; INTERACTIVE MACHINE LEARNING; INTERACTIVITY; TIGHT COUPLING; USER EXPERIENCE;

MACHINE LEARNING;

EID: 84920541067     PISSN: 07384602     EISSN: None     Source Type: Journal    
DOI: 10.1609/aimag.v35i4.2513     Document Type: Article

References (42)

1. Amershi, S. 2012. Designing for Effective End-User Interaction with Machine Learning. Ph.D. Dissertation, Department of Computer Science. University of Washington, Seattle, WA. (hdl.handle.net/1773/22006)
2. Amershi, S.; Cakmak, M.; Knox, W. B.; Kulesza, T.; and Lau, T. 2013. IUI Workshop on Interactive Machine Learning. In Proceedings of the Companion Publication of the 2013 International Conference on Intelligent User Interfaces, 121-124. New York: Association for Computing Machinery. dx.doi.org/10.1145/2451176.2451230
3. Amershi, S.; Fogarty, J.; Kapoor, A.; and Tan, D. 2010. Examining Multiple Potential Models in End-User Interactive Concept Learning. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2010 (CHI 2010), 1357-1360. New York: Association for Computing Machinery.
4. Amershi, S.; Fogarty, J.; Kapoor, A.; and Tan, D. 2009. Overview-Based Example Selection in Mixed-Initiative Concept Learning. In Proceedings of the ACM Symposium on User Interface Software and Technology, 2009 (UIST 2009), 47-256. New York: Association for Computing Machinery.
5. Blackwell, A. F. 2002. First Steps in Programming: A Rationale for Attention Investment Models. In Proceedings of the 2002 IEEE Symposium on Human Centric Computing Languages and Environments, 2-10. Piscataway, NJ: Institute for Electrical and Electronics Engineers.
6. Cakmak, M., and Thomaz, A. L. 2010. Optimality of Human Teachers for Robot Learners. In Proceedings of the 2010 IEEE 9th International Conference on Development and Learning (ICDL), 64-69. Piscataway, NJ: Institute for Electrical and Electronics Engineers.
7. Cakmak, M.; Chao, C.; and Thomaz, A. L. 2010. Designing Interactions for Robot Active Learners. Autonomous Mental Development 2(2): 108-118. dx.doi.org/10.1109/ TAMD.2010.2051030
8. Caruana, R.; Elhaway, M.; Nguyen, N.; and Smith, C. 2006. Meta Clustering. In Proceedings of the Sixth IEEE International Conference on Data Mining, 2006. (ICDM'06), 107-118. Piscataway, NJ: Institute for Electrical and Electronics Engineers.
9. Cohn, D.; Caruana, R.; and McCallum, A. 2003. Semi-Supervised Clustering with User Feedback. Constrained Clustering: Advances in Algorithms, Theory, and Applications 4(1): 17-32.
10. Dix, A.; Finlay, J.; Abowd, G. D.; and Beal, R. 2004. Interaction Design Basics. In Human Computer Interaction, 3rd edition, chapter 5, 189-224. Harlow, England: Pearson Education Ltd.
11. th International Conference on Intelligent User Interfaces, 39-45. New York: Association for Computing Machinery.
12. Fiebrink, R.; Cook, P. R.; and Trueman, D. 2011. Human Model Evaluation in Interactive Supervised Learning. In Proceedings of the Conference on Human Factors in Computing Systems (CHI 2011), 147-156. New York: Association for Computing Machinery. dx.doi.org/10.1145/1978942.1978965
13. Fogarty, J.; Tan, D.; Kapoor, A.; and Winder, S. 2008. CueFlik: Interactive Concept Learning in Image Search. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 29-38. New York: Association for Computing Machinery.
14. Freund, Y., and Schapire, R. E. (1995). A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting. In Proceedings of the Second European Conference on Computational Learning Theory, 23-37. Berlin, Heidelberg: Springer. dx.doi.org/10.1007/3-540-59119-2-166
15. th International Conference on Machine Learning (ICML-11), 369-376). Princeton, NJ: International Machine Learning Society, Inc.
16. Hoffman R.; Amershi, S.; Patel, K.; Wu, F.; Fogarty, J.; and Weld, D. S. 2009. Amplifying Community Content Creation with Mixed-Initiative Information Extraction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2009), 1849-1858. New York: Association for Computing Machinery.
17. Höök, K. 2000. Steps to Take Before Intelligent User Interfaces Become Real. Interacting with Computers 12(4): 409-426. dx.doi.org/ 10.1016/S0953-5438(99)00006-5
18. Horvitz, E. 1999. Principles of Mixed-Initiative User Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 159-166. New York: Association for Computing Machinery.
19. Isbell Jr., C. L.; Kearns, M.; Singh, S.; Shelton, C. R.; Stone, P.; and Kormann, D. 2006. Cobot in LambdaMOO: An Adaptive Social Statistics Agent. Autonomous Agents and Multi-Agent Systems 13(3): 327-354. dx.doi.org/10.1007/s10458-006-0005-z
20. Jameson, A. 2009. Adaptive Interfaces and Agents. Human-Computer Interaction: Design Issues, Solutions, and Applications, 105. Boca Raton, FL: CRC Press.
21. Kamar, E.; Hacker, S.; and Horvitz, E. 2012. Combining Human and Machine Intelligence in Large-Scale Crowdsourcing. In Proceedings of the International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS 2012). Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems.
22. Kaochar, T.; Peralta, R. T.; Morrison, C. T.; Fasel, I. R.; Walsh, T. J.; and Cohen, P. R. 2011. Towards Understanding How Humans Teach Robots. In User Modeling, Adaption and Personalization, Lecture Notes in Computer Science Volume 6787, 347-352. Berlin: Springer. dx.doi.org/10.1007/ 978-3-642-22362-4-31
23. Kapoor, A.; Lee, B.; Tan, D.; and Horvitz, E. 2010. Interactive Optimization for Steering Machine Classiication. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1343-1352. New York: Association for Computing Machinery.
24. Knox, W. B., and Stone, P. 2013. Learning Non-Myopically from Human-Generated Reward. In Proceedings of the 2013 International Conference on Intelligent User Interfaces, 191-202. New York: Association for Computing Machinery. dx.doi.org/10.1145/ 2449396.2449422
25. Knox, W. B., and Stone, P. 2012. Reinforcement Learning from Human Reward: Discounting in Episodic Tasks. In Proceedings of the 2012 IEEE International Workshop on Robots and Human Interactive Communications (RO-MAN), 878-885. Piscataway, NJ: Institute for Electrical and Electronics Engineers.
26. Kulesza, T.; Stumpf, S.; Burnett, M.; and Kwan, I. 2012. Tell Me More?: The Effects of Mental Model Soundness on Personalizing an Intelligent Agent. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 1-10. New York: Association for Computing Machinery.
27. Kulesza, T.; Stumpf, S.; Wong, W. K.; Burnett, M. M.; Perona, S.; Ko, A.; and Oberst, I. 2011. Why-Oriented End-User Debugging of Naive Bayes Text Classiication. ACM Transactions on Interactive Intelligent Systems 1(1): 2. dx.doi.org/10.1145/2030365. 2030367
28. Law, E., and von Ahn, R. 2009. Input-Agreement: A New Mechanism for Data Collection Using Human Computation Games. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2009). New York: Association for Computing Machinery.
29. Moggridge, B., and Smith, G. C.2007. Designing Interactions, Volume 17. Cambridge, MA: The MIT Press.
30. Norman, D. A. 1994. How Might People Interact with Agents. Communications of the ACM 37(7): 68-71. dx.doi.org/10.1145/ 176789.176796
31. Norman, D. A. 1988. The Design of Everyday Things. New York: Basic Books.
32. Porter, R.; Theiler, J.; and Hush, D. 2013. Interactive Machine Learning in Data Exploitation. Technical Report. Los Alamos National Laboratories, Los Alamos, NM. dx.doi.org/10.2172/1060903
33. Pu, P., and Chen, L. 2009. User-Involved Preference Elicitation for Product Search and Recommender Systems. AI Magazine 29(4): 93.
34. Rashid, A. M.; Ling, K.; Tassone, R. D.; Resnick, P.; Kraut, R.; and Riedl, J. 2006. Motivating Participation by Displaying the Value of Contribution. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 955-958. New York: Association for Computing Machinery.
35. Rosenthal, S. L., and Dey, A. K. 2010. Towards Maximizing the Accuracy of Human-Labeled Sensor Data. In Proceedings of the 15th International Conference on Intelligent User Interfaces, 259-268. New York: Association for Computing Machinery. dx.doi.org/10.1145/1719970.1720006
36. Settles, B. 2010. Active Learning Literature Survey. Technical Report, Department of Computer Science, University of Wisconsin, Madison, Madison, WI.
37. th edition. Reading, MA: Addi-son-Wesley.
38. Stumpf, S.; Rajaram, V.; Li, L.; Burnett, M.; Dietterich, T.; Sullivan, E.; Drummond, R.; and Herlocker, J. 2007. Toward Harnessing User Feedback for Machine Learning. In Proceedings of the 12th International Conference on Intelligent User Interfaces, 82-91. New York: Association for Computing Machinery. dx.doi.org/10.1145/1216295.1216316
39. Talbot, J.; Lee, B.; Kapoor, A.; and Tan, D. S. 2009. EnsembleMatrix: Interactive Visualization to Support Machine Learning with Multiple Classiiers. In Proceedings of the 27th International Conference on Human Factors in Computing Systems, 1283-1292. New York: Association for Computing Machinery.
40. Thomaz, A. L., and Breazeal, C. 2008. Teachable Robots: Understanding Human Teaching Behavior to Build More Effective Robot Learners. Artiicial Intelligence 172(6): 716-737.
41. th International Conference on Intelligent User Interfaces, 93-102. New York: Association for Computing Machinery. dx.doi.org/10.1145/1943403.1943418
42. Winograd, T. 1996. Bringing Design to Software. New York: Association for Computing Machinery.