Principles of Explanatory Debugging to personalize interactive machine learning

International Conference on Intelligent User Interfaces, Proceedings IUI

Volumn 2015-January, Issue , 2015, Pages 126-137

  Kulesza, Todd ^a   Burnett, Margaret ^a   Wong, Weng Keen ^a   Stumpf, Simone ^b  

^a 1148 Kelley Engineering Center Corvallis   (United States)

^b CITY UNIVERSITY   (United Kingdom)

Author keywords

End user programming; Interactive machine learning

Indexed keywords

MACHINE LEARNING; USER INTERFACES;

EMPIRICAL EVALUATIONS; END USER PROGRAMMING; END USERS; INTERACTIVE MACHINE LEARNING; TRADITIONAL LEARNING;

PROGRAM DEBUGGING;

EID: 84939637881     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2678025.2701399     Document Type: Conference Paper

References (47)

1. Amershi, S., Cakmak, M., Knox, W. B., and Kulesza, T. Power to the people: The role of humans in interactive machine learning. AI Magazine (in press).
2. Amershi, S., Fogarty, J., Kapoor, A., and Tan, D. Examining multiple potential models in end-user interactive concept learning. In Proceedings of the ACM Conference on Human Factors in Computing Systems (2010), 1357-1360.
3. Blackwell, A. F. First steps in programming: A rationale for attention investment models. In Proceedings of the IEEE 2002 Symposia on Human Centric Computing Languages and Environments, IEEE (2002), 2-10.
4. Bostandjiev, S., O'Donovan, J., and Höllerer, T. TasteWeights: A visual interactive hybrid recommender system. In Proceedings of the 6th ACM Conference on Recommender Systems (2012), 35-42.
5. Bryan, N. J., Mysore, G. J., and Wang, G. ISSE: An interactive source separation editor. In Proceedings of the ACM Conference on Human Factors in Computing Systems (2014), 257-266.
6. Bunt, A., Lount, M., and Lauzon, C. Are explanations always important? A study of deployed, low-cost intelligent interactive systems. In Proceedings of the 2012 ACM International Conference on Intelligent User Interfaces (2012), 169-178.
7. Cakmak, M., Chao, C., and Thomaz, A. L. Designing interactions for robot active learners. IEEE Transactions on Autonomous Mental Development 2, 2(2010), 108-118.
8. Cramer, H., Evers, V., Ramlal, S., van Someren, M., Rutledge, L., Stash, N., Aroyo, L., and Wielinga, B. The effects of transparency on trust in and acceptance of a content-based art recommender. User Modeling and User-Adapted Interaction 18, 5(2008), 455-496.
9. Craven, M. W., and Shavlik, J. W. Using neural networks for data mining. Future generation computer systems 13(1997), 211-229.
10. Dzindolet, M. T., Peterson, S. A., Pomranky, R. A., Pierce, L. G., and Beck, H. P. The role of trust in automation reliance. International Journal of Human-Computer Studies 58, 6(2003), 697-718.
11. Fails, J. A., Olsen, D. R., and Jr. Interactive machine learning. In Proceedings of the 8th International Conference on Intelligent User Interfaces (2003), 39-45.
12. Fiebrink, R., Cook, P. R., and Trueman, D. Human model evaluation in interactive supervised learning. In Proceedings of the ACM Conference on Human Factors in Computing Systems (2011), 147-156.
13. Fogarty, J., Tan, D., Kapoor, A., and Winder, S. CueFlik: Interactive concept learning in image search. In Proceedings of the ACM Conference on Human Factors in Computing Systems (2008), 29-38.
14. Groce, A., Kulesza, T., Zhang, C., Shamasunder, S., Burnett, M. M., Wong, W.-K., Stumpf, S., Das, S., Shinsel, A., Bice, F., and McIntosh, K. You are the only possible oracle: Effective test selection for end users of interactive machine learning systems. IEEE Transactions on Software Engineering 40, 3(2014), 307-323.
15. Hart, S. G., and Staveland, L. E. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. Advances in Psychology 52(1988), 139-183.
16. Herlocker, J. L., Konstan, J. A., and Riedl, J. Explaining collaborative filtering recommendations. In Proceedings of the 2000 ACM Conference on Computer Supported Cooperative Work (2000), 241-250.
17. Johnson-Laird, P. N. Mental models: Towards a cognitive science of language, inference, and consciousness. Harvard University Press, 1983.
18. Kapoor, A., Lee, B., Tan, D., and Horvitz, E. Interactive optimization for steering machine classification. In Proceedings of the ACM Conference on Human Factors in Computing Systems (2010), 1343-1352.
19. Kempton, W. Two theories of home heat control. Cognitive Science 10(1986), 75-90.
20. Kibriya, A. M., Frank, E., Pfahringer, B., and Holmes, G. Multinomial naive Bayes for text categorization revisited. In AI 2004: Advances in Artificial Intelligence. Springer Berlin Heidelberg, Berlin, Heidelberg, 2004, 488-499.
21. Knox, W. B., and Stone, P. Reinforcement learning from human reward: Discounting in episodic tasks. In Proceedings of the 21st IEEE International Symposium on Robot and Human Interactive Communication (2012), 878-885.
22. Ko, A. J., and Myers, B. A. Debugging reinvented. the 13th international conference (2008), 301-310.
23. Kulesza, T., Stumpf, S., Burnett, M. M., and Kwan, I. Tell me more? The effects of mental model soundness on personalizing an intelligent agent. In Proceedings of the ACM Conference on Human Factors in Computing Systems (2012), 1-10.
24. Kulesza, T., Stumpf, S., Burnett, M. M., Wong, W.-K., Riche, Y., Moore, T., Oberst, I., Shinsel, A., and McIntosh, K. Explanatory debugging: Supporting end-user debugging of machine-learned programs. In Proceedings of the 2010 IEEE Symposium on Visual Languages and Human-Centric Computing (2010), 41-48.
25. Kulesza, T., Stumpf, S., Burnett, M. M., and Yang, S. Too much, too little, or just right? Ways explanations impact end users' mental models. Proceedings of the 2013 IEEE Symposium on Visual Languages and Human-Centric Computing (2013), 3-10.
26. Kulesza, T., Stumpf, S., Wong, W.-K., Burnett, M. M., Perona, S., Ko, A. J., and Oberst, I. Why-oriented end-user debugging of naive Bayes text classification. ACM Transactions on Interactive Intelligent Systems 1, 1 (2011).
27. Lacave, C., and Díez, F. J. A review of explanation methods for Bayesian networks. The Knowledge Engineering Review 17, 2(2002), 107-127.
28. Lim, B. Y. Improving understanding and trust with intelligibility in context-aware applications. PhD thesis, Carnegie Mellon University, 2012.
29. Lim, B. Y., and Dey, A. K. Assessing demand for intelligibility in context-aware applications. In Proceedings of the 11th International Conference on Ubiquitous Computing (2009), 195-204.
30. Lim, B. Y., Dey, A. K., and Avrahami, D. Why and why not explanations improve the intelligibility of context-aware intelligent systems. In Proceedings of the ACM Conference on Human Factors in Computing Systems (2009), 2119-2128.
31. McCrum-Gardner, E. Which is the correct statistical test to use? British Journal of Oral and Maxillofacial Surgery 46(2008), 38-41.
32. Norman, D. A. Some observations on mental models. In Human-Computer Interaction, R. M. Baecker and W. A. S. Buxton, Eds. San Francisco, CA, USA, 1987, 241-244.
33. Norman, D. A. The Design of Everyday Things. Revised and Expanded Edition. Basic Books, 2002.
34. Prabhakararao, S., Cook, C., Ruthruff, J., Creswick, E., Main, M., Durham, M., and Burnett, M. M. Strategies and behaviors of end-user programmers with interactive fault localization. In Proceedings of the IEEE Symposium on Human Centric Computing Languages and Environments (2003), 15-22.
35. Rowe, M. B. Teaching science as continuous inquiry. McGraw-Hill, 1973.
36. Rugaber, S., Goel, A. K., and Martie, L. GAIA: A CAD environment for model-Based Adaptation of game-playing Software Agents. Procedia Computer Science 16(2013), 29-38.
37. Settles, B. Active learning literature survey. Tech. Rep. 1648, University of Wisconsin-Madison, 2010.
38. Stumpf, S., Rajaram, V., Li, L., Wong, W.-K., Burnett, M. M., Dietterich, T., Sullivan, E., and Herlocker, J. Interacting meaningfully with machine learning systems: Three experiments. International Journal of Human-Computer Studies 67, 8(2009), 639-662.
39. Szafron, D., Greiner, R., Lu, P., and Wishart, D. Explaining naïve Bayes classifications. Tech. Rep. TR03-09, University of Alberta, 2003.
40. Talbot, J., Lee, B., Kapoor, A., and Tan, D. S. EnsembleMatrix: Interactive visualization to support machine learning with multiple classifiers. In Proceedings of the ACM Conference on Human Factors in Computing Systems (2009), 1283-1292.
41. Thomaz, A. L., and Breazeal, C. Transparency and socially guided machine learning. In Proceedings of the 5th International Conference on Development and Learning (2006).
42. Tintarev, N., and Masthoff, J. Evaluating the effectiveness of explanations for recommender systems. User Modeling and User-Adapted Interaction 22, 4-5(2012), 399-439.
43. Tullio, J., Dey, A. K., Chalecki, J., and Fogarty, J. How it works: A field study of non-technical users interacting with an intelligent system. In Proceedings of the ACM Conference on Human Factors in Computing Systems (2007), 31-40.
44. van der Meij, H., and Carroll, J. M. Principles and heuristics for designing minimalist instruction. In Minimalism beyond the Nurnberg funnel, J. M. Carroll, Ed. MIT Press, Cambridge, MA, 1998, 19-53.
45. Vig, J., Sen, S., and Riedl, J. Navigating the tag genome. In Proceedings of the 16th International Conference on Intelligent User Interfaces (2011), 93-102.
46. Yang, R., and Newman, M. W. Learning from a learning thermostat: Lessons for intelligent systems for the home. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (2013), 93-102.
47. Yang, Y., and Pedersen, J. O. A comparative study on feature selection in text categorization. In Proceedings of the Twentieth International Conference on Machine Learning (1997), 412-420.