Beat the machine: Challenging humans to find a predictive model's "unknown unknowns"

Journal of Data and Information Quality

Volumn 6, Issue 1, 2015, Pages

  Attenberg, Joshua ^a   Ipeirotis, Panos ^b   Provost, Foster ^b  

^a ETSY INC   (United States)

^b NEW YORK UNIVERSITY   (United States)

Author keywords

Crowdsourcing; Incentives; Machine learning evaluation; Model assessment; Risk identification; System design

Indexed keywords

CROWDSOURCING; ERRORS; LEARNING SYSTEMS; RISK ASSESSMENT; SYSTEMS ANALYSIS;

AUTOMATIC MODELING; INCENTIVES; MODEL ASSESSMENT; NORMAL OPERATIONS; PREDICTIVE MODELING; PREDICTIVE MODELS; RISK IDENTIFICATION; TRADITIONAL TECHNIQUES;

PREDICTIVE ANALYTICS;

EID: 84924757988     PISSN: 19361955     EISSN: 19361963     Source Type: Journal    
DOI: 10.1145/2700832     Document Type: Article

References (17)

1. Josh Attenberg, Panagiotis G. Ipeirotis, and Foster J. Provost. 2011. Beat the machine: Challenging workers to find the unknown unknowns. In Proceedings of the 3rd Human Computation Workshop (HCOMP'11).
2. Josh Attenberg and Foster Provost. 2010. Inactive learning? Difficulties employing active learning in practice. SIGKDD Explorations 12, 2(2010), 36-41.
3. C. Chow. 1970. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory 16, 1(Jan. 1970), 41-46.
4. C. K. Chow. 1957. An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers EC-6, 4(Dec. 1957), 247-254.
5. Pedro Domingos. 1999. Metacost: A general method for making classifiers cost-sensitive. In Proceedings of the 5th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 155-164.
6. Charles Elkan. 2001. The foundations of cost-sensitive learning. In Proceedings of the 17th International Joint Conference on Artificial Intelligence-Volume 2. 973-978.
7. David D. Lewis and William A. Gale. 1994. A sequential algorithm for training text classifiers. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. 3-12.
8. C. Perlich, B. Dalessandro, T. Raeder, O. Stitelman, and F. Provost. 2014. Machine learning for targeted display advertising: Transfer learning in action. Machine Learning 95, 1(2014), 103-127.
9. Foster Provost and Tom Fawcett. 2013. Data Science for Business: What You Need to Know about Data Mining and Data-Analytic Thinking.
10. O'Reilly Media. Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Anna Jerebko, Charles Florin, Gerardo Hermosillo Valadez, Luca Bogoni, and Linda Moy. 2009. Supervised learning from multiple experts: Whom to trust when everyone lies a bit. In Proceedings of the 26th Annual International Conference on Machine Learning. 889-896.
11. R. Reiter. 1977. On Closed World Data Bases. Technical Report. University of British Columbia, Vancouver, BC, Canada.
12. Maytal Saar-Tsechansky and Foster Provost. 2004. Active sampling for class probability estimation and ranking. Machine Learning 54, 2(2004), 153-178.
13. Robert E. Schapire. 1999. A brief introduction to boosting. In Proceedings of the 16th International Joint Conference on Artificial Intelligence - Volume 2. Burr Settles. 2012. Active Learning. Vol. 6. Morgan & Claypool Publishers.
14. Victor S. Sheng, Foster Provost, and Panagiotis G. Ipeirotis. 2008. Get another label? Improving data quality and data mining using multiple, noisy labelers. In Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 614-622.
15. Kurt VanLehn. 1998. Analogy events: How examples are used during problem solving. Cognitive Science 22, 3(1998), 347-388.
16. Gary M. Weiss. 2010. The impact of small disjuncts on classifier learning. In Data Mining, Robert Stahlbock, Sven F. Crone, and Stefan Lessmann (Eds.). Annals of Information Systems, Vol. 8. Springer, 193-226.
17. Patrick H. Winston. 1970. Learning Structural Descriptions From Examples. Technical Report. Massachusetts Institute of Technology.