On human intellect and machine failures: Troubleshooting integrative machine learning systems

31st AAAI Conference on Artificial Intelligence, AAAI 2017

Volumn , Issue , 2017, Pages 1017-1025

  Nushi, Besmira ^a   Kamar, Ece ^b   Horvitz, Eric ^b   Kossmann, Donald ^a,b  

^a ETH ZURICH   (Switzerland)

^b MICROSOFT RESEARCH   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; MACHINE COMPONENTS; SYSTEMS ENGINEERING;

BLAME ASSIGNMENTS; EXPECTED IMPROVEMENTS; HUMAN COMPUTATION; HUMAN-IN-THE-LOOP; IMAGE CAPTIONING; MACHINE FAILURE; MULTIPLE POINTS; TROUBLESHOOTING SYSTEMS;

LEARNING SYSTEMS;

EID: 85030469300     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (21)

1. Attenberg, J.; Ipeirotis, P. G.; and Provost, F. J. 2011. Beat the machine: Challenging workers to find the unknown unknowns. Human Computation 11: 11.
2. Berger, A. L.; Pietra, V. J. D.; and Pietra, S. A. D. 1996. A maximum entropy approach to natural language processing. Computational linguistics 22 (1): 39-71.
3. Breese, J. S., and Heckerman, D. 1996. Decision-theoretic troubleshooting: A framework for repair and experiment. In UAI.
4. Chakarov, A.; Nori, A.; Rajamani, S.; Sen, S.; and Vijaykeerthy, D. 2016. Debugging machine learning tasks. arXiv preprint arXiv: 1603. 07292.
5. Chang, J. C.; Kittur, A.; and Hahn, N. 2016. Alloy: Clustering with crowds and computation. In CHI.
6. Cheng, J., and Bernstein, M. S. 2015. Flock: Hybrid crowdmachine learning classifiers. In CHI, 600-611. ACM.
7. Darwiche, A. 2000. Model-based diagnosis under realworld constraints. AI magazine 21 (2): 57.
8. Fang, H.; Gupta, S.; Iandola, F.; Srivastava, R. K.; Deng, L.; Dollár, P.; Gao, J.; He, X.; Mitchell, M.; Platt, J. C.; et al. 2015. From captions to visual concepts and back. In CVPR, 1473-1482.
9. Kulesza, T.; Stumpf, S.; Burnett, M.; Wong, W.-K.; Riche, Y.; Moore, T.; Oberst, I.; Shinsel, A.; and McIntosh, K. 2010. Explanatory debugging: Supporting end-user debugging of machine-learned programs. In VL/HCC, 41-48. IEEE.
10. Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014. Microsoft coco: Common objects in context. In ECCV 2014. Springer. 740-755.
11. Nushi, B.; Kamar, E.; Horvitz, E.; and Kossmann, D. 2016. On human intellect and machine failures: Troubleshooting integrative machine learning systems. arXiv preprint arXiv: 1611. 08309.
12. Parikh, D., and Zitnick, C. 2011a. Human-debugging of machines. NIPS WCSSWC 2: 7.
13. Parikh, D., and Zitnick, C. L. 2011b. Finding the weakest link in person detectors. In CVPR, 1425-1432. IEEE.
14. Patel, K.; Bancroft, N.; Drucker, S. M.; Fogarty, J.; Ko, A. J.; and Landay, J. 2010. Gestalt: integrated support for implementation and analysis in machine learning. In UIST, 37-46. ACM.
15. Sculley, D.; Holt, G.; Golovin, D.; Davydov, E.; Phillips, T.; Ebner, D.; Chaudhary, V.; Young, M.; Crespo, J.-F.; and Dennison, D. 2015. Hidden technical debt in machine learning systems. In NIPS.
16. Settles, B. 2010. Active learning literature survey. University of Wisconsin, Madison 52 (55-66): 11.
17. Sheng, V. S.; Provost, F.; and Ipeirotis, P. G. 2008. Get another label? improving data quality and data mining using multiple, noisy labelers. In KDD, 614-622. ACM.
18. Yao, L.; Ballas, N.; Cho, K.; Smith, J. R.; and Bengio, Y. 2015. Trainable performance upper bounds for image and video captioning. arXiv preprint arXiv: 1511. 04590.
19. Yih, W.-t.; Chang, M.-W.; He, X.; and Gao, J. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In ACL.
20. Zhang, C.; Platt, J. C.; and Viola, P. A. 2005. Multiple instance boosting for object detection. In NIPS, 1417-1424.
21. Zou, J. Y.; Chaudhuri, K.; and Kalai, A. T. 2015. Crowdsourcing feature discovery via adaptively chosen comparisons. HCOMP.