You are the only possible oracle: Effective test selection for end users of interactive machine learning systems

IEEE Transactions on Software Engineering

Volumn 40, Issue 3, 2014, Pages 307-323

  Groce, Alex ^a   Kulesza, Todd ^a   Zhang, Chaoqiang ^a   Shamasunder, Shalini ^a   Burnett, Margaret ^a   Wong, Weng Keen ^a   Stumpf, Simone ^b   Das, Shubhomoy ^a   Shinsel, Amber ^a   Bice, Forrest ^a   McIntosh, Kevin ^a  

^a Oregon State University   (United States)

^b CITY UNIVERSITY   (United Kingdom)

Author keywords

end user testing; Machine learning; test suite size

Indexed keywords

LEARNING ALGORITHMS; LEARNING SYSTEMS; TESTING;

END USER TESTING; FAILURE RATE; GOLD STANDARDS; HIGH CONFIDENCE; INTERACTIVE MACHINE LEARNING; RANDOM EXPERIMENTS; SINGLE USERS; TEST SELECTION;

SOFTWARE TESTING;

EID: 84898034150     PISSN: 00985589     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSE.2013.59     Document Type: Article

References (63)

1. IEEE Standard Engineering Glossary Software Eng. Terminology. IEEE Press, 1990.
2. S. Amershi, J. Fogarty, and D. Weld, "Regroup: Interactive Machine Learning for On-Demand Group Creation in Social Networks," Proc. SIGCHI Conf. Human Factors in Computing Systems (CHI '12), pp. 21-30, May 2012.
3. A. Arcuri, M. Iqbal, and L. Briand, "Formal Analysis of the Effectiveness and Predictability of Random Testing," Proc. Int'l Symp. Software Testing and Analysis, pp. 219-230, 2010.
4. A. Asuncion, D. Newman, UCI Machine Learning Repository, 2007.
5. R. Baeza-Yates and B. Ribeiro-Neto, Modern Information Retrieval. Pearson Education India, 1999.
6. A. Blackwell, "First Steps in Programming: A Rationale for Attention Investment Models," Proc. IEEE Conf. Human-Centric Computing, pp. 2-10, 2002.
7. D. Brain and G. Webb, "On the Effect of Data Set Size on Bias and Variance in Classification Learning," Proc. Fourth Australian Knowledge Acquisition Workshop, pp. 117-128, 1999.
8. C.-C. Chang, and C.-J. Lin, LIBSVM: A Library for Support Vector Machines, http://www.csie.ntu.edu.tw/cjlin/libsvm, 2001.
9. T. Chen, T. Tse, and Z.Quan Zhou, "Fault-Based Testing without the Need of Oracles," Information and Software Technology, vol. 45, no. 1, pp. 1-9, 2003.
10. T.Y. Chen, S.C. Cheung, and S. Yiu, "Metamorphic Testing: A New Approach for Generating Next Test Cases," Technical Report HKUST-CS98-01, Hong Kong Univ. of Science and Technology, 1998.
11. Y. Chen, A. Groce, C. Zhang, W.-K. Wong, X. Fern, E. Eide, and J. Regehr, "Taming Compiler Fuzzers," Proc. ACM SIGPLAN Conf. Programming Language Design and Implementation, pp. 197-208, 2013.
12. C. Cortes and V. Vapnik, "Support Vector Networks," Machine Learning, vol. 20, pp. 273-297, 1995.
13. W. Dickinson, D. Leon, and A. Podgurski, "Pursuing Failure: The Distribution of Program Failures in a Profile Space," Proc. European Software Eng. Conf., pp. 246-255, 2001. (Pubitemid 35466929)
14. S. Elbaum, A. Malishevsky, and G. Rothermel, "Test Case Prioritization: A Family of Empirical Studies," IEEE Trans. Software Eng., vol. 28, no. 2, pp. 159-182, Feb. 2002. (Pubitemid 34198321)
15. J. Fogarty, D. Tan, A. Kapoor, and S. Winder, "CueFlik: Interactive Concept Learning in Image Search," Proc. SIGCHI Conf. Human Factors in Computing Systems (CHI '08), pp. 29-38, Apr. 2008.
16. P. Francis, D. Leon, M. Minch, and A. Podgurski, "Tree-Based Methods for Classifying Software Failures," Proc. Int'l Symp. Software Reliability Eng., pp. 451-462, 2004. (Pubitemid 40455715)
17. P. Frankl and S. Weiss, "An Experimental Comparison of the Effectiveness of Branch Testing and Data Flow Testing," IEEE Trans. Software Eng., vol. 19, no. 3, pp. 202-213, Aug. 1993.
18. P. Frankl, S. Weiss, and C. Hu, "All-Uses vs Mutation Testing: An Experimental Comparison of Effectiveness," J. Systems and Software, vol. 38, no. 3, pp. 235-253, 1997. (Pubitemid 127384432)
19. A. Glass, D. McGuinness, and M. Wolverton, "Toward Establishing Trust in Adaptive Agents," Proc. 13th Int'l Conf. Intelligent User Interfaces (IUI), pp. 227-236, 2008.
20. M. Gligoric, A. Groce, C. Zhang, R. Sharma, A. Alipour, and D. Marinov, "Comparing Non-Adequate Test Suites Using Coverage Criteria," Proc. Int'l Symp. Software Testing and Analysis, pp. 302-313, 2013.
21. V. Grigoreanu, J. Cao, T. Kulesza, C. Bogart, K. Rector, M. Burnett, and S. Wiedenbeck, "Can Feature Design Reduce the Gender Gap in End-User Software Development Environments?" Proc. IEEE Symp. Visual Languages and Human-Centric Computing (VL/HCC), pp. 149-156, 2008.
22. A. Groce, A. Fern, J. Pinto, T. Bauer, A. Alipour, M. Erwig, and C. Lopez, "Lightweight Automated Testing with Adaptation-Based Programming," Proc. IEEE Int'l Symp. Software Reliability Eng., pp. 161-170, 2012.
23. M. Harman, "The Role of Artificial Intelligence in Software Engineering," Proc. First Int'l Workshop Realizing Artificial Intelligence Synergies in Software Eng., pp. 1-6, 2012.
24. M. Harman, E. Burke, J. Clark, and X. Yao, "Dynamic Adaptive Search Based Software Engineering," Proc. ACM/IEEE Int'l Symp. Empirical Software Eng. and Measurement, pp. 1-8, 2012.
25. M. Harman and P. McMinn, "A Theoretical and Empirical Study of Search-Based Testing: Local, Global, and Hybrid Search," IEEE Trans. Software Eng., vol. 36, no. 2, pp. 226-247, Mar./Apr. 2010.
26. S. Hart and L. Staveland, "Development of a NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research," Human Mental Workload, P. Hancock and N. Meshkati, eds., pp. 139-183, North-Holland, 1988.
27. T. Hastie, R. Tibshirani, and J.H. Friedman, The Elements of Statistical Learning. Springer, 2003.
28. D. Isa, L. Lee, V. Kallimani, and R. RajKumar, "Text Document Preprocessing with the Bayes Formula for Classification Using the Support Vector Machine," IEEE Trans. Knowledge and Data Eng., vol. 20, no. 9, pp. 1264-1272, Sept. 2008.
29. T. Jiang, L. Tan, and S. Kim, "Personalized Defect Prediction," Proc. ACM Conf. Automated Software Eng., pp. 279-289, 2013.
30. A. Kapoor, B. Lee, D. Tan, and E. Horvitz, "Interactive Optimization for Steering Machine Classification," Proc. SIGCHI Conf. Human Factors in Computing Systems (CHI), pp. 1343-1352, 2010.
31. G. Kniesel, and T. Rho, Newsgroup Data Set, http://www.ai.mit. edu/jrennie/20newsgroups, 2005.
32. T. Kulesza, M. Burnett, S. Stumpf, W.-K. Wong, S. Das, A. Groce, A. Shinsel, F. Bice, and K. McIntosh, "Where Are My Intelligent Assistant's Mistakes? A Systematic Testing Approach," Proc. Int'l Symp. End-User Development, pp. 171-186, 2011.
33. T. Kulesza, S. Stumpf, M. Burnett, W.-K. Wong, Y. Riche, T. Moore, I. Oberst, A. Shinsel, and K. McIntosh, "Explanatory Debugging: Supporting End-User Debugging of Machine-Learned Programs," Proc. IEEE Symp. Visual Languages and Human-Centric Computing, pp. 41-48, 2010.
34. T. Kulesza, S. Stumpf, W.-K. Wong, M. Burnett, S. Perona, A. Ko, and I. Obsert, "Why-Oriented End-User Debugging of Naive Bayes Text Classification," ACM Trans. Interactive Intelligent Systems, vol. 1, no. 1, pp. 1-31, Oct. 2011.
35. T. Kulesza, W.-K. Wong, S. Stumpf, S. Perona, R. White, M. Burnett, I. Oberst, and A. Ko, "Fixing the Program My Computer Learned: Barriers for End Users, Challenges for the Machine," Proc. ACM Int'l Conf. Intelligent User Interfaces, pp. 187-196, 2009.
36. K. Lang, "Newsweeder: Learning to Filter Netnews," Proc. Int'l Conf. Machine Learning, pp. 331-339, 1995.
37. D. Lewis and W. Gale, "A Sequential Algorithm for Training Text Classifiers," Proc. ACM Conf. Research and Development in Information Retrieval, pp. 3-12, 1994.
38. B. Lim and A. Dey, "Toolkit to Support Intelligibility in Context-Aware Applications," Proc. Int'l Conf. Ubiquitous Computing, pp. 13-22, 2010.
39. B. Lim, A. Dey, and D. Avrahami, "Why and Why Not Explanations Improve the Intelligibility of Context-Aware Intelligent Systems," Proc. ACM Conf. Human Factors in Computing Systems, pp. 2119-2128, 2009.
40. M.E. Maron, "Automatic Indexing: An Experimental Inquiry," J. ACM, vol. 8, no. 3, pp. 404-417, 1961.
41. A. McCallum, Mallet: A Machine Learning for Language Toolkit, http://mallet.cs.umass.edu, 2002.
42. R. Miller and B. Myers, "Outlier Finding: Focusing User Attention on Possible Errors," Proc. 14th Ann. ACM Symp. User Interface Software and Technology (UIST), pp. 81-90, 2001. (Pubitemid 34991265)
43. C. Murphy, G. Kaiser, and M. Arias, "An Approach to Software Testing of Machine Learning Applications," Proc. Int'l Conf. Software Eng. and Knowledge Eng., pp. 167-172, 2007.
44. C. Murphy, K. Shen, and G. Kaiser, "Automatic System Testing of Programs without Test Oracles," Proc. Int'l Symp. Software Testing and Analysis, pp. 189-200, 2009.
45. R. Panko, "What We Know about Spreadsheet Errors" http://reference. kfupm.edu.sa/content/w/h/what-we-know-about-spreadsheet- errors-72956.pdf, Retreived Aug. 2010. expanded version of article in J. End User Computing, vol. 19, no. 2, pp. 15-21, Spring 1998.
46. H. Raghavan, O. Madani, and R. Jones, "Active Learning with Feedback on Both Features and Instances," J. Machine Learning Research, vol. 7, pp. 1655-1686, 2006. (Pubitemid 44223089)
47. O. Raz, P. Koopman, and M. Shaw, "Semantic Anomaly Detection in Online Data Sources," Proc. 24th Int'l Conf. Software Eng. (ICSE), pp. 302-312, 2002. (Pubitemid 35009119)
48. G. Rothermel, M. Burnett, L. Li, C. DuPois, and A. Sheretov, "A Methodology for Testing Spreadsheets," ACM Trans. Software Eng. and Methodology, vol. 10, no. 1, pp. 110-147, 2001. (Pubitemid 33631327)
49. G. Rothermel, M.J. Harrold, J. Ostrin, and C. Hong, "An Empirical Study of the Effects of Minimization on the Fault Detection Capabilities of Test Suites," Proc. Int'l Conf. Software Maintenance, 1998.
50. R.K. Saha, M. Lease, S. Khurshid, and D.E. Perry, "Improving Bug Localization Using Structured Information Retrieval," Proc. ACM Conf. Automated Software Eng., pp. 345-355, 2013.
51. C. Scaffidi, "Unsupervised Inference of Data Formats in Human-Readable Notation," Proc. Int'l Conf. Enterprise Integration Systems, pp. 236-241, 2007.
52. J. Segal, "Some Problems of Professional End User Developers," Proc. IEEE Symp. Visual Languages and Human-Centric Computing, 2007.
53. B. Settles, "Active Learning Literature Survey," Technical Report 1648, Univ. of Wisc., http://pages.cs.wisc.edu/~bsettles/pub/settles. activelearning.pdf, Jan. 2010.
54. J. Shen and T. Dietterich, "Active EM to Reduce Noise in Activity Recognition," Proc. 12th Int'l Conf. Intelligent User Interfaces (IUI), pp. 132-140, 2007. (Pubitemid 47463998)
55. J. Shetty and J. Adibi, "The Enron Email Dataset Database Schema and Brief Statistical Report," technical report, Univ. of Southern California, 2004.
56. A. Shinsel, T. Kulesza, M.M. Burnett, W. Curan, A. Groce, S. Stumpf, and W.-K. Wong, "Mini-Crowdsourcing End-User Assessment of Intelligent Assistants: A Cost-Benefit Study," Proc. IEEE Symp. Visual Languages and Human-Centric Computing, pp. 47-54, 2011.
57. J. Talbot, B. Lee, A. Kapoor, and D. Tan, "Ensemblematrix: Interactive Visualization to Support Machine Learning with Multiple Classifiers," Proc. SIGCHI Conf. Human Factors in Computing Systems (CHI), pp. 1283-1292, 2009.
58. J. Tullio, A. Dey, J. Chalecki, and J. Fogarty, "How It Works: A Field Study of Non-Technical Users Interacting with an Intelligent System," Proc. ACM Conf. Human Factors in Computing Systems, pp. 31-40, 2007. (Pubitemid 47581582)
59. L. Wasserman, All of Statistics. Springer, 2004.
60. W.-K. Wong, I. Oberst, S. Das, T. Moore, S. Stumpf, K. McIntosh, and M. Burnett, "End-User Feature Labeling: A Locally-Weighted Regression Approach," Proc. Int'l Conf. Intelligent User Interfaces, pp. 115-124, 2011.
61. T.-F. Wu, C.-J. Lin, and R.C. Weng, "Probability Estimates for Multi-Class Classification by Pairwise Coupling," J. Machine Learning Research, vol. 5, pp. 975-1005, 2004.
62. X. Xie, J. Ho, C. Murphy, B. Xu, and T.Y. Chen, "Application of Metamorphic Testing to Supervised Classifiers," Proc. Int'l Conf. Quality Software, pp. 135-144, 2009.
63. B. Zadrozny and C. Elkan, "Obtaining Calibrated Probability Estimates from Decision Trees and Naive Bayesian Classifiers," Proc. 18th Int'l Conf. Machine Learning, pp. 609-616, 2001.