Flock: Hybrid crowd-machine learning classifiers

CSCW 2015 - Proceedings of the 2015 ACM International Conference on Computer-Supported Cooperative Work and Social Computing

Volumn , Issue , 2015, Pages 600-611

  Cheng, Justin ^a   Bernstein, Michael S ^a  

^a STANFORD UNIVERSITY   (United States)

Author keywords

Crowdsourcing; Interactive Machine Learning

Indexed keywords

ALGORITHMS; ARTIFICIAL INTELLIGENCE; COMPUTER SUPPORTED COOPERATIVE WORK; CROWDSOURCING; FEATURE EXTRACTION; GROUPWARE; HYBRID SYSTEMS; INTERACTIVE COMPUTER SYSTEMS;

ALGORITHM PERFORMANCE; AUTOMATED FEATURE EXTRACTION; CLASSIFICATION MODELS; HYBRID CLASSIFIER; INTERACTIVE MACHINE LEARNING; MACHINE LEARNING MODELS; OFF-THE-SHELF MACHINE; WRITTEN DESCRIPTION;

LEARNING SYSTEMS;

EID: 84968903161     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1145/2675133.2675214     Document Type: Conference Paper

References (49)

1. Crowdflower. http://www.crowdflower.com.
2. Anderson, A., Huttenlocher, D., Kleinberg, J., and Leskovec, J. Discovering value from community activity on focused question answering sites: A case study of stack overflow. In Proc. KDD (2012).
3. Andŕe, P., Kittur, A., and Dow, S. P. Crowd synthesis: Extracting categories and clusters from complex data. In Proc. CSCW (2014).
4. Attenberg, J., Ipeirotis, P. G., and Provost, F. J. Beat the machine: Challenging workers to find the unknown unknowns. In Proc. AAAIW (2011).
5. Blumenstock, J. E. Automatically assessing the quality of wikipedia articles. UCB School of Information (2008).
6. Branson, S., Wah, C., Schroff, F., Babenko, B., Welinder, P., Perona, P., and Belongie, S. Visual recognition with humans in the loop. In Proc. ECCV. 2010.
7. Chilton, L. B., Little, G., Edge, D., Weld, D. S., and Landay, J. A. Cascade: Crowdsourcing taxonomy creation. In Proc. CHI (2013).
8. Dai, P., Weld, D. S., et al. Decision-Theoretic control of crowd-sourced workflows. In Proc. AAAI (2010).
9. Deng, J., Krause, J., and Fei-Fei, L. Fine-grained crowdsourcing for fine-grained recognition. In Proc. CVPR (2013).
10. Domingos, P. A few useful things to know about machine learning. Communications of the ACM (2012).
11. Fails, J. A., and Olsen Jr, D. R. Interactive machine learning. In Proc. IUI (2003).
12. Fogarty, J., Tan, D., Kapoor, A., and Winder, S. Cueflik: interactive concept learning in image search. In Proc. CHI (2008).
13. Gazzaniga, M. S., LeDoux, J. E., Gazzaniga, M., and LeDoux, J. The integrated mind. 1978.
14. Gentner, D., Loewenstein, J., and Thompson, L. Learning and transfer: A general role for analogical encoding. Journal of Educational Psychology (2003).
15. Gentner, D., and Markman, A. B. Structure mapping in analogy and similarity. American Psychologist (1997).
16. Gomes, R. G., Welinder, P., Krause, A., and Perona, P. Crowdclustering. In Advances in Neural Information Processing Systems (2011), 558-566.
17. Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., and Witten, I. H. The weka data mining software: An update. ACM SIGKDD Explorations (2009).
18. Hammond, K. R., Hursch, C. J., and Todd, F. J. Analyzing the components of clinical inference. Psychological review (1964).
19. Hartmann, B., Abdulla, L., Mittal, M., and Klemmer, S. R. Authoring sensor-based interactions by demonstration with direct manipulation and pattern recognition. In Proc. CHI (2007).
20. Kahneman, D. A perspective on judgment and choice: mapping bounded rationality. American psychologist (2003).
21. Kamar, E., Kapoor, A., and Horvitz, E. Lifelong learning for acquiring the wisdom of the crowd. In Proc. JCAI (2013).
22. Kim, J., Cheng, J., and Bernstein, M. S. Ensemble: Exploring complementary strengths of leaders and crowds in creative collaboration. In Proc. CSCW (2014).
23. Kittur, A., Peters, A. M., Diriye, A., and Bove, M. Standing on the schemas of giants: socially augmented information foraging. In Proc. CSCW (2014).
24. Lasecki, W. S., Song, Y. C., Kautz, H., and Bigham, J. P. Real-Time crowd labeling for deployable activity recognition. In Proc. CSCW (2013).
25. Law, E. Attribute Learning using Joint Human and Machine Computation. PhD thesis, 2012.
26. Law, E., Settles, B., Snook, A., Surana, H., Von Ahn, L., and Mitchell, T. Human computation for attribute and attribute value acquisition. In FPGV (2011).
27. Law, E., and Von Ahn, L. Input-Agreement: A new mechanism for collecting data using human computation games. In Proc. CHI, ACM (2009).
28. Le, J., Edmonds, A., Hester, V., and Biewald, L. Ensuring quality in crowdsourced search relevance evaluation: The effects of training question distribution. In Proc. SIGIR (2010).
29. Li, H., Zhao, B., and Fuxman, A. The wisdom of minority: Discovering and targeting the right group of workers for crowdsourcing. In Proc. WWW (2014).
30. Lowe, D. G. Object recognition from local scale-invariant features. In ICCV (1999).
31. Ott, M., Choi, Y., Cardie, C., and Hancock, J. T. Finding deceptive opinion spam by any stretch of the imagination. In Proc. ACL (2011).
32. Parameswaran, A., Sarma, A. D., Garcia-Molina, H., Polyzotis, N., and Widom, J. Human-Assisted graph search: its okay to ask questions. Proc. VLDB (2011).
33. Parikh, D., and Grauman, K. Interactively building a discriminative vocabulary of nameable attributes. In CVPR (2011).
34. Parkash, A., and Parikh, D. Attributes for classifier feedback. In Proc. ECCV. 2012.
35. Patel, K., Bancroft, N., Drucker, S. M., Fogarty, J., Ko, A. J., and Landay, J. Gestalt: integrated support for implementation and analysis in machine learning. In Proc. UIST (2010).
36. Patel, K., Fogarty, J., Landay, J. A., and Harrison, B. Investigating statistical machine learning as a tool for software development. In Proc. CHI (2008).
37. Raykar, V. C., Yu, S., Zhao, L. H., Valadez, G. H., Florin, C., Bogoni, L., and Moy, L. Learning from crowds. JMLR (2010).
38. Rzeszotarski, J., and Kittur, A. Crowdscape: interactively visualizing user behavior and output. In Proc. UIST (2012).
39. Rzeszotarski, J. M., and Kittur, A. Instrumenting the crowd: using implicit behavioral measures to predict task performance. In Proc. UIST (2011).
40. Settles, B. Closing the loop: Fast, interactive semi-supervised annotation with queries on features and instances. In Proc. EMNLP (2011).
41. Sheng, V. S., Provost, F., and Ipeirotis, P. G. Get another label? improving data quality and data mining using multiple, noisy labelers. In Proc. KDD (2008).
42. Skinner, B. F. Reinforcement today. American Psychologist (1958).
43. Snow, R., OConnor, B., Jurafsky, D., and Ng, A. Y. Cheap and fast-but is it good?: evaluating non-expert annotations for natural language tasks. In Proc. EMNLP (2008).
44. Talbot, J., Lee, B., Kapoor, A., and Tan, D. S. Ensemblematrix: interactive visualization to support machine learning with multiple classifiers. In Proc. CHI (2009).
45. Vijayanarasimhan, S., and Grauman, K. Large-scale live active learning: Training object detectors with crawled data and crowds. In CVPR (2011).
46. Wais, P., Lingamneni, S., Cook, D., Fennell, J., Goldenberg, B., Lubarov, D., Marin, D., and Simons, H. Towards building a high-quality workforce with mechanical turk. In Proc. NIPSCSS (2010).
47. Whitehill, J., Ruvolo, P., Wu, T., Bergsma, J., and Movellan, J. R. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Proc. NIPS (2009).
48. Yu, L., Kittur, A., and Kraut, R. E. Distributed analogical idea generation: Inventing with crowds. In Proc. CHI (2014).
49. Yu, L., and Nickerson, J. V. Cooks or cobblers?: crowd creativity through combination. In Proc. CHI (2011).