Crowdsourcing in computer vision

Foundations and Trends in Computer Graphics and Vision

Volumn 10, Issue 3, 2016, Pages 177-243

  Kovashka, Adriana ^a   Russakovsky, Olga ^b   Fei Fei, Li ^c   Grauman, Kristen ^d  

^a UNIVERSITY OF PITTSBURGH   (United States)

^b CARNEGIE MELLON UNIVERSITY   (United States)

^c STANFORD UNIVERSITY   (United States)

^d UNIVERSITY OF TEXAS AT AUSTIN   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

CROWDSOURCING; DATA ACQUISITION; OBJECT RECOGNITION;

COMPUTER VISION SYSTEM; CROWDSOURCING PLATFORMS; DATA COLLECTION; DESIGN DECISIONS; HUMAN KNOWLEDGE; STORY TELLING; VISUAL CONCEPT; VISUAL PERCEPTION;

COMPUTER VISION;

EID: 85017143359     PISSN: 15722740     EISSN: 15722759     Source Type: Journal    
DOI: 10.1561/0600000071     Document Type: Review

References (194)

1. E. Ahmed, S. Maji, G. Shakhnarovich, and L. S. Davis. Using human knowledge to judge part goodness: Interactive part selection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop: Computer Vision and Human Computation, 2014.
2. M. Andriluka, L. Pishchulin, P. Gehler, and B. Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
3. R. Anirudh and P. Turaga. Interactively test driving an object detector: Estimating performance on unlabeled data. In IEEE Winter Conference on Applications of Computer Vision (WACV), 2014.
4. S. Antol, C. L. Zitnick, and D. Parikh. Zero-shot learning via visual abstraction. In European Conference on Computer Vision (ECCV). 2014.
5. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh. VQA: Visual Question Answering. In IEEE International Conference on Computer Vision (ICCV), 2015.
6. P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. Contour detection and hierarchical image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 33(5), 2011.
7. H. Azizpour and I. Laptev. Object detection using strongly-supervised deformable part models. In European Conference on Computer Vision (ECCV), 2012.
8. S. Bandla and K. Grauman. Active learning of an action detector from untrimmed videos. In IEEE International Conference on Computer Vision (ICCV), 2013.
9. D. Batra, A. Kowdle, D. Parikh, J. Luo, and T. Chen. iCoseg: Interactive co-segmentation with intelligent scribble guidance. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010.
10. A. Bearman, O. Russakovsky, V. Ferrari, and L. Fei-Fei. What's the point: Semantic segmentation with point supervision. In European Conference on Computer Vision (ECCV), 2016.
11. S. Bell, P. Upchurch, N. Snavely, and K. Bala. OpenSurfaces: A richly annotated catalog of surface appearance. ACM Transactions on Graphics (SIGGRAPH), 32(4), 2013.
12. S. Bell, K. Bala, and N. Snavely. Intrinsic images in the wild. ACM Transaction on Graphics (SIGGRAPH), 33(4), 2014.
13. S. Bell, P. Upchurch, N. Snavely, and K. Bala. Material recognition in the wild with the materials in context database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
14. A. Biswas and D. Jacobs. Active image clustering: Seeking constraints from humans to complement algorithms. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
15. A. Biswas and D. Parikh. Simultaneous active learning of classifiers and attributes via relative feedback. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.
16. E. Borenstein and S. Ullman. Class-specific, top-down segmentation. In European Conference on Computer Vision (ECCV), 2002.
17. L. Bourdev and J. Malik. Poselets: Body part detectors trained using 3d human pose annotations. In IEEE International Conference on Computer Vision (ICCV), 2009.
18. A. Boyko and T. Funkhouser. Cheaper by the dozen: Group annotation of 3D data. In ACM Symposium on User Interface Software and Technology (UIST), 2014.
19. S. Branson, C. Wah, F. Schroff, B. Babenko, P. Welinder, P. Perona, and S. Belongie. Visual Recognition with Humans in the Loop. In European Conference on Computer Vision (ECCV), 2010.
20. S. Branson, P. Perona, and S. Belongie. Strong supervision from weak annotation: Interactive training of deformable part models. In IEEE International Conference on Computer Vision (ICCV), 2011.
21. S. Branson, K. Eldjarn Hjorleifsson, and P. Perona. Active annotation translation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
22. A. Chandrasekaran, A. Kalyan, S. Antol, M. Bansal, D. Batra, C. L. Zitnick, and D. Parikh. We are humor beings: Understanding and predicting visual humor. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
23. Y.-W. Chao, Z. Wang, Y. He, J. Wang, and J. Deng. Hico: A benchmark for recognizing human-object interactions in images. In IEEE International Conference on Computer Vision (ICCV), 2015.
24. C.-Y. Chen and K. Grauman. Predicting the location of "interactees" in novel human-object interactions. In Asian Conference on Computer Vision (ACCV), 2014.
25. C.-Y. Chen and K. Grauman. Subjects and their objects: Localizing interactees for a person-centric view of importance. Computing Research Repository (CoRR), abs/1604.04842, 2016.
26. X. Chen, R. Mottaghi, X. Liu, S. Fidler, R. Urtasun, and A. Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
27. L. B. Chilton, G. Little, D. Edge, D. S. Weld, and J. A. Landay. Cascade: Crowdsourcing taxonomy creation. In SIGCHI Conference on Human Factors in Computing Systems, 2013.
28. G. Christie, A. Parkash, U. Krothapalli, and D. Parikh. Predicting user annoyance using visual attributes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
29. A. Criminisi. Microsoft Research Cambridge (MSRC) object recognition image database (version 2.0). http://research.microsoft.com/vision/cambridge/recognition, 2004.
30. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
31. J. Deng, J. Krause, and L. Fei-Fei. Fine-grained crowdsourcing for fine-grained recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.
32. J. Deng, O. Russakovsky, J. Krause, M. Bernstein, A. C. Berg, and L. Fei-Fei. Scalable multi-label annotation. In SIGCHI Conference on Human Factors in Computing Systems, 2014.
33. J. Deng, J. Krause, M. Stark, and L. Fei-Fei. Leveraging the wisdom of the crowd for fine-grained recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 38(4), April 2016.
34. A. Deza and D. Parikh. Understanding image virality. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
35. J. Donahue and K. Grauman. Annotator rationales for visual recognition. In IEEE International Conference on Computer Vision (ICCV), 2011.
36. I. Endres, A. Farhadi, D. Hoiem, and D. A. Forsyth. The benefits and challenges of collecting richer object annotations. In IEEE Computer Vision and Pattern Recognition (CVPR) Workshops, 2010.
37. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The Pascal Visual Object Classes (VOC) challenge. International Journal of Computer Vision (IJCV), 88(2), June 2010.
38. M. Everingham,, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The Pascal Visual Object Classes (VOC) challenge -A Retrospective. International Journal of Computer Vision (IJCV), 2014.
39. S. Fan, T.-T. Ng, J. S. Herberg, B. L. Koenig, C. Y.-C. Tan, and R. Wang. An automated estimator of image visual realism based on human cognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
40. A. Farhadi, I. Endres, D. Hoiem, and D. A. Forsyth. Describing Objects by Their Attributes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
41. A. Fathi, M. F. Balcan, X. Ren, and J. M. Rehg. Combining self training and active learning for video segmentation. In British Machine Vision Conference (BMVC), 2011.
42. L. Fei-Fei and P. Perona. A bayesian hierarchical model for learning natural scene categories. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2005.
43. L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few examples: An incremental bayesian approach tested on 101 object categories. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2004.
44. V. Ferrari and A. Zisserman. Learning visual attributes. In Advances in Neural Information Processing Systems (NIPS), 2007.
45. A. Freytag, E. Rodner, and J. Denzler. Selecting influential examples: Active learning with expected model output changes. In European Conference on Computer Vision (ECCV). 2014.
46. E. Gavves, T. Mensink, T. Tommasi, C. G. M. Snoek, and T. Tuytelaars. Active transfer learning with zero-shot priors: Reusing past datasets for future tasks. In IEEE International Conference on Computer Vision (ICCV), 2015.
47. D. Geman, S. Geman, N. Hallonquist, and L. Younes. Visual turing test for computer vision systems. Proceedings of the National Academy of Sciences, 112(12), 2015.
48. A. Gilbert and R. Bowden. igroup:Weakly supervised image and video grouping. In IEEE International Conference on Computer Vision (ICCV), 2011.
49. R. G. Gomes, P. Welinder, A. Krause, and P. Perona. Crowdclustering. In Advances in Neural Information Processing Systems (NIPS), 2011.
50. A. Gorban, H. Idrees, Y.-G. Jiang, A. Roshan Zamir, I. Laptev, M. Shah, and R. Sukthankar. THUMOS challenge: Action recognition with a large number of classes. http://www.thumos.info/2015.
51. S. Gould, R. Fulton, and D. Koller. Decomposing a scene into geometric and semantically consistent regions. In IEEE International Conference on Computer Vision (ICCV), 2009.
52. G. Griffin, A. Holub, and P. Perona. Caltech-256 object category dataset. Technical Report 7694, Caltech, 2007.
53. Y. Guo and R. Greiner. Optimistic active-learning using mutual information. In International Joint Conference on Artificial Intelligence (IJCAI), 2007.
54. A. Gupta, A. Kembhavi, and L. S. Davis. Observing human-object interactions: Using spatial and functional compatibility for recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 31 (10), Oct 2009.
55. D. Gurari, D. H. Theriault, M. Sameki, B. Isenberg, T. A. Pham, A. Purwada, P. Solski, M. L. Walker, C. Zhang, J. Y. Wong, and M. Betke. How to collect segmentations for biomedical images? a benchmark evaluating the performance of experts, crowdsourced non-experts, and algorithms. In IEEE Winter Conference on Applications of Computer Vision (WACV), 2015.
56. D. Gurari, S. D. Jain, M. Betke, and K. Grauman. Pull the plug? predicting if computers or humans should segment images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
57. T. S. Haines and T. Xiang. Active learning using dirichlet processes for rare class discovery and classification. In British Machine Vision Conference (BMVC), 2011.
58. B. Hariharan, P. Arbelaez, L. Bourdev, S. Maji, and J. Malik. Semantic contours from inverse detectors. In IEEE International Conference on Computer Vision (ICCV), 2011.
59. F. C. Heilbron, V. Escorcia, B. Ghanem, and J. C. Niebles. ActivityNet: A Large-Scale Video Benchmark for Human Activity Understanding. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
60. G. Hua, C. Long, M. Yang, and Y. Gao. Collaborative active learning of a kernel machine ensemble for recognition. In IEEE International Conference on Computer Vision (ICCV), 2013.
61. P. Jain and A. Kapoor. Active learning for large multi-class problems. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
62. P. Jain, S. Vijayanarasimhan, and K. Grauman. Hashing hyperplane queries to near points with applications to large-scale active learning. In Advances in Neural Information Processing Systems (NIPS), 2010.
63. S. D. Jain and K. Grauman. Predicting sufficient annotation strength for interactive foreground segmentation. In IEEE International Conference on Computer Vision (ICCV), 2013.
64. S. D. Jain and K. Grauman. Active image segmentation propagation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
65. K. G. Jamieson, L. Jain, C. Fernandez, N. J. Glattard, and R. Nowak. Next: A system for real-world development, evaluation, and application of active learning. In Advances in Neural Information Processing Systems (NIPS). 2015.
66. J. H. Janssens. Ranking images on semantic attributes using human computation. In Advances in Neural Information Processing Systems (NIPS) Workshops, 2010.
67. E. Johns, O. Mac Aodha, and G. J. Brostow. Becoming the expert - interactive multi-class machine teaching. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
68. J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. A. Shamma, M. Bernstein, and L. Fei-Fei. Image retrieval using scene graphs. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
69. J. Joo, W. Li, F. F. Steen, and S.-C. Zhu. Visual persuasion: Inferring communicative intents of images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
70. A. J. Joshi, F. Porikli, and N. Papanikolopoulos. Breaking the interactive bottleneck in multi-class classification with active selection and binary feedback. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010.
71. M. Kabra, A. A. Robie, M. Rivera-Alba, S. Branson, and K. Branson. JAABA: interactive machine learning for automatic annotation of animal behavior. Nature Methods, 10(1), 2013.
72. C. Kading, A. Freytag, E. Rodner, P. Bodesheim, and J. Denzler. Active learning and discovery of object categories in the presence of unnameable instances. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
73. A. Kapoor, K. Grauman, R. Urtasun, and T. Darrell. Gaussian processes for object categorization. International Journal of Computer Vision (IJCV), 88(2), 2010.
74. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar, and L. Fei- Fei. Large-scale video classification with convolutional neural networks. In CVPR, 2014.
75. S. Kazemzadeh, V. Ordonez, M. Matten, and T. L. Berg. Referit game: Referring to objects in photographs of natural scenes. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
76. K. Konyushkova, R. Sznitman, and P. Fua. Introducing geometry in active learning for image segmentation. In IEEE International Conference on Computer Vision (ICCV), 2015.
77. A. Kovashka and K. Grauman. Attribute adaptation for personalized image search. In IEEE International Conference on Computer Vision (ICCV), 2013a.
78. A. Kovashka and K. Grauman. Attribute pivots for guiding relevance feedback in image search. In IEEE International Conference on Computer Vision (ICCV), 2013b.
79. A. Kovashka and K. Grauman. Discovering attribute shades of meaning with the crowd. International Journal of Computer Vision (IJCV), 114(1), 2015.
80. A. Kovashka, S. Vijayanarasimhan, and K. Grauman. Actively selecting annotations among objects and attributes. In IEEE International Conference on Computer Vision (ICCV), 2011.
81. A. Kovashka, D. Parikh, and K. Grauman. Whittlesearch: Image search with relative attribute feedback. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
82. A. Kovashka, D. Parikh, and K. Grauman. Whittlesearch: Interactive image search with relative attribute feedback. International Journal of Computer Vision (IJCV), 115(2), 2015.
83. A. Kowdle, Y.-J. Chang, A. Gallagher, and T. Chen. Active learning for piecewise planar 3d reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011.
84. J. Krause, J. Deng, M. Stark, and L. Fei-Fei. Collecting a large-scale dataset of fine-grained cars. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop: Fine-Grained Visual Categorization, 2013.
85. R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, M. Bernstein, and L. Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision (IJCV), 2016.
86. A. Krizhevsky. Learning multiple layers of features from tiny images. https://www.cs.toronto.edu/~kriz/cifar.html 2009.
87. A. Kulkarni, M. Can, and B. Hartmann. Collaboratively crowdsourcing workflows with turkomatic. In Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work (CSCW), 2012.
88. S. Lad and D. Parikh. Interactively Guiding Semi-Supervised Clustering via Attribute-based Explanations. In European Conference on Computer Vision (ECCV), 2014.
89. C. Lampert, H. Nickisch, and S. Harmeling. Learning to Detect Unseen Object Classes By Between-Class Attribute Transfer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
90. D. Larlus, F. Perronnin, P. Kompalli, and V. Mishra. Generating gold questions for difficult visual recognition tasks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop: Computer Vision and Human Computation, 2014.
91. W. Lasecki, M. Gordon, D. Koutra, M. Jung, S. Dow, and J. Bigham. Glance: Rapidly coding behavioral video with the crowd. In ACM Symposium on User Interface Software and Technology (UIST), 2014.
92. S. Lazebnik, C. Schmid, and J. Ponce. Beyond bags of features: Spatial Pyramid Matching for recognizing natural scene categories. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2006.
93. D. Le, R. Bernardi, and J. Uijlings. TUHOI: Trento Universal Human Object Interaction Dataset. In Conference on Computational Linguistics (COLING) Workshop: Vision and Language, 2014.
94. D.-T. Le, J. Uijlings, and R. Bernardi. Exploiting language models for visual recognition. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2013.
95. S. Lee and D. Crandall. Learning to identify local flora with human feedback. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop: Computer Vision and Human Computation, 2014.
96. F. Li, T. Kim, A. Humayun, D. Tsai, and J. M. Rehg. Video segmentation by tracking many figure-ground segments. In IEEE International Conference on Computer Vision (ICCV), 2013.
97. X. Li and Y. Guo. Adaptive active learning for image classification. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.
98. X. Li and Y. Guo. Multi-level adaptive active learning for scene classification. In European Conference on Computer Vision (ECCV). 2014.
99. L. Liang and K. Grauman. Beyond comparing image pairs: Setwise active learning for relative attributes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
100. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. DollÃar, and C. L. Zitnick. Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision (ECCV), 2014.
101. G. Little, L. B. Chilton, M. Goldman, and R. C. Miller. Turkit: Human computation algorithms on mechanical turk. In ACM Symposium on User Interface Software and Technology (UIST), 2010.
102. J. Little, A. Abrams, and R. Pless. Tools for richer crowd source image annotations. In IEEE Winter Conference on Applications of Computer Vision (WACV), 2012.
103. C. Liu, J. Yuen, and A. Torralba. Nonparametric scene parsing via label transfer. Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 33(12), 2011.
104. W. Liu, O. Russakovsky, J. Deng, L. Fei-Fei, and A. Berg. ImageNet Large Scale Visual Recognition Challenge - object detection from video track. http://image-net.org/challenges/LSVRC/2015/2015.
105. C. Long and G. Hua. Multi-class multi-Annotator active learning with robust gaussian process for visual recognition. In IEEE International Conference on Computer Vision (ICCV), 2015.
106. C. Long, G. Hua, and A. Kapoor. Active visual recognition with expertise estimation in crowdsourcing. In IEEE International Conference on Computer Vision (ICCV), 2013.
107. C. C. Loy, T. M. Hospedales, T. Xiang, and S. Gong. Stream-based joint exploration-exploitation active learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
108. W. Luo, X. Wang, and X. Tang. Content-based photo quality assessment. In IEEE International Conference on Computer Vision (ICCV), 2011.
109. S. Maji. Discovering a lexicon of parts and attributes. In European Conference on Computer Vision (ECCV) Workshops, 2012.
110. S. Maji and G. Shakhnarovich. Part discovery from partial correspondence. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.
111. S. Maji, L. Bourdev, and J. Malik. Action recognition using a distributed representation of pose and appearance. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011.
112. M. Malinowski and M. Fritz. A multi-world approach to question answering about real-world scenes based on uncertain input. In Advances in Neural Information Processing Systems (NIPS), 2014.
113. A. Mao, E. Kamar, and E. Horvitz. Why stop now? predicting worker engagement in online crowdsourcing. In AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2013.
114. T. Matera, J. Jakes, M. Cheng, and S. Belongie. A user friendly crowdsourcing task manager. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshop: Computer Vision and Human Computation, 2014.
115. T. Mensink, J. Verbeek, and G. Csurka. Learning structured prediction models for interactive image labeling. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011.
116. A. Montagnini, M. Bicego, and M. Cristani. Tell me what you like and I'll tell you what you are: discriminating visual preferences on flickr data. analysis, 10, 2012.
117. R. Mottaghi, X. Chen, X. Liu, N.-G. Cho, S.-W. Lee, S. Fidler, R. Urtasun, and A. Yuille. The role of context for object detection and semantic segmentation in the wild. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
118. N. S. Nagaraja, F. R. Schmidt, and T. Brox. Video segmentation with just a few strokes. In IEEE International Conference on Computer Vision (ICCV), 2015.
119. A. Oliva and A. Torralba. Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision (IJCV), 2001.
120. D. Parikh and K. Grauman. Interactively building a discriminative vocabulary of nameable attributes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011a.
121. D. Parikh and K. Grauman. Relative attributes. In IEEE International Conference on Computer Vision (ICCV), 2011b.
122. A. Parkash and D. Parikh. Attributes for classifier feedback. In European Conference on Computer Vision (ECCV). Springer, 2012.
123. G. Patterson and J. Hays. SUN Attribute Database: Discovering, Annotating, and Recognizing Scene Attributes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
124. G. Patterson, G. V. Horn, S. Belongie, P. Perona, and J. Hays. Tropel: Crowdsourcing detectors with minimal training. In AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2015.
125. K.-C. Peng, T. Chen, A. Sadovnik, and A. C. Gallagher. A mixed bag of emotions: Model, predict, and transfer emotion distributions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
126. A. Prest, C. Leistner, J. Civera, C. Schmid, and V. Ferrari. Learning object class detectors from weakly annotated video. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
127. M. Rubinstein, C. Liu, and W. Freeman. Annotation propagation: Automatic annotation of large image databases via dense image correspondence. In ECCV, 2012.
128. M. Rubinstein, A. Joulin, J. Kopf, and C. Liu. Unsupervised joint object discovery and segmentation in internet images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.
129. O. Russakovsky and L. Fei-Fei. Attribute learning in large-scale datasets. In European Conference of Computer Vision (ECCV) Workshop: Parts and Attributes, 2010.
130. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3), 2015a. .
131. O. Russakovsky, L.-J. Li, and L. Fei-Fei. Best of both worlds: human-machine collaboration for object annotation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015b.
132. B. Russell, A. Torralba, K. Murphy, and W. T. Freeman. LabelMe: A database and web-based tool for image annotation. International Journal of Computer Vision (IJCV), 2007.
133. H. S. Seung, M. Opper, and H. Sompolinsky. Query by committee. In Conference on Learning Theory (COLT) Workshops. ACM, 1992.
134. N. B. Shah and D. Zhou. Double or nothing: Multiplicative incentive mechanisms for crowdsourcing. In Advances in Neural Information Processing Systems (NIPS), 2014.
135. L. Sharan, R. Rosenholtz, and E. Adelson. Material perception: What can you see in a brief glance? Journal of Vision, 9(8), 2009.
136. V. S. Sheng, F. Provost, and P. G. Ipeirotis. Get another label? improving data quality and data mining using multiple, noisy labelers. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2008.
137. B. Siddiquie and A. Gupta. Beyond active noun tagging: Modeling contextual interactions for multi-class active learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010.
138. B. Siddiquie, R. S. Feris, and L. S. Davis. Image ranking and retrieval based on multi-Attribute queries. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011.
139. G. A. Sigurdsson, O. Russakovsky, A. Farhadi, I. Laptev, and A. Gupta. Much ado about time: Exhaustive annotation of temporal data. In AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2016a.
140. G. A. Sigurdsson, G. Varol, X. Wang, A. Farhadi, I. Laptev, and A. Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In European Conference of Computer Vision (ECCV), 2016b.
141. E. Simo-Serra, S. Fidler, F. Moreno-Noguer, and R. Urtasun. Neuroaesthetics in fashion: Modeling the perception of fashionability. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
142. A. Sorokin and D. Forsyth. Utility data annotation with Amazon Mechanical Turk. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2008.
143. H. Su, J. Deng, and L. Fei-Fei. Crowdsourcing annotations for visual object detection. In AAAI Conference on Artificial Intelligence Workshop: Human Computation (HCOMP), 2012.
144. O. Tamuz, C. Liu, S. Belongie, O. Shamir, and A. T. Kalai. Adaptively learning the crowd kernel. In International Machine Learning Conference (ICML), 2011.
145. M. Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urtasun, and S. Fidler. Movieqa: Understanding stories in movies through question-Answering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
146. T. Tian and J. Zhu. Max-margin majority voting for learning from crowds. In Advances in Neural Information Processing Systems (NIPS), 2015.
147. S. Tong and D. Koller. Support vector machine active learning with applications to text classification. The Journal of Machine Learning Research, 2, 2002.
148. D. Tsai, M. Flagg, and J. Rehg. Motion coherent tracking with multi-label MRF optimization. In British Machine Vision Conference (BMVC), 2010.
149. R. Vedantam, C. Lawrence Zitnick, and D. Parikh. Cider: Consensus-based image description evaluation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
150. A. Vezhnevets, J. M. Buhmann, and V. Ferrari. Active learning for semantic segmentation with expected change. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
151. S. Vijayanarasimhan and K. Grauman. What's it going to cost you?: Predicting effort vs. informativeness for multi-label image annotations. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2009.
152. S. Vijayanarasimhan and K. Grauman. Cost-sensitive active visual category learning. International Journal of Computer Vision (IJCV), 91(1), 2011a.
153. S. Vijayanarasimhan and K. Grauman. Large-scale live active learning: Training object detectors with crawled data and crowds. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2011b.
154. S. Vijayanarasimhan and K. Grauman. Active frame selection for label propagation in videos. In European Conference on Computer Vision (ECCV), 2012.
155. S. Vijayanarasimhan and K. Grauman. Large-scale live active learning: Training object detectors with crawled data and crowds. International Journal of Computer Vision (IJCV), 108(1-2), 2014.
156. S. Vijayanarasimhan, P. Jain, and K. Grauman. Far-sighted active learning on a budget for image and video recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010.
157. S. Vijayanarasimhan, P. Jain, and K. Grauman. Hashing hyperplane queries to near points with applications to large-scale active learning. Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 36(2), 2014.
158. S. Vittayakorn and J. Hays. Quality assessment for crowdsourced object annotations. In British Machine Vision Conference (BMVC), 2011.
159. L. von Ahn and L. Dabbish. Esp: Labeling images with a computer game. In AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors, 2005.
160. L. von Ahn, M. Kedia, and M. Blum. Verbosity: A game for collecting common-sense facts. In SIGCHI Conference on Human Factors in Computing Systems, 2006a.
161. L. von Ahn, R. Liu, and M. Blum. Peekaboom: A game for locating objects in images. In SIGCHI Conference on Human Factors in Computing Systems, 2006b.
162. C. Vondrick and D. Ramanan. Video Annotation and Tracking with Active Learning. In Advances in Neural Information Processing Systems (NIPS), 2011.
163. C. Vondrick, D. Patterson, and D. Ramanan. Efficiently scaling up crowdsourced video annotation. International Journal of Computer Vision (IJCV), 101(1), 2013.
164. C. Vondrick, D. Oktay, H. Pirsiavash, and A. Torralba. Predicting the motivations behind actions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
165. C. Wah and S. Belongie. Attribute-Based Detection of Unfamiliar Classes with Humans in the Loop. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.
166. C. Wah, S. Branson, P. Perona, and S. Belongie. Multiclass recognition and part localization with humans in the loop. In IEEE International Conference on Computer Vision (ICCV), 2011a.
167. C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech- UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011b.
168. C. Wah, G. Van Horn, S. Branson, S. Maji, P. Perona, and S. Belongie. Similarity comparisons for interactive fine-grained categorization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
169. C. Wah, S. Maji, and S. Belongie. Learning localized perceptual similarity metrics for interactive categorization. In IEEE Winter Conference on Applications of Computer Vision (WACV), 2015.
170. D. Wang, C. Yan, S. Shan, and X. Chen. Active learning for interactive segmentation with expected confidence change. In Asian Conference on Computer Vision (ACCV). 2012.
171. J. Wang, P. G. Ipeirotis, and F. Provost. Quality-based pricing for crowdsourced workers. NYU-CBA Working Paper CBA-13-06, 2013.
172. Y. Wang and G. Mori. A discriminative latent model of object classes and attributes. In European Conference on Computer Vision (ECCV). Springer, 2010.
173. D. S. Weld, Mausam, and P. Dai. Human intelligence needs artificial intelligence. In AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2011.
174. P. Welinder and P. Perona. Online crowdsourcing: Rating annotators and obtaining cost-effective labels. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2010.
175. P. Welinder, S. Branson, S. Belongie, and P. Perona. The multidimensional wisdom of crowds. In Advances in Neural Information Processing Systems (NIPS), 2010.
176. M. Wigness, B. A. Draper, and J. R. Beveridge. Efficient label collection for unlabeled image datasets. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
177. M. J. Wilber, I. S. Kwak, and S. J. Belongie. Cost-effective hits for relative similarity comparisons. In AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2014.
178. M. J. Wilber, I. S. Kwak, D. Kriegman, and S. Belongie. Learning concept embeddings with combined human-machine expertise. In IEEE International Conference on Computer Vision (ICCV), 2015.
179. A. G. Wilson, C. Dann, C. Lucas, and E. P. Xing. The human kernel. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems (NIPS). 2015.
180. J. Xiao, J. Hays, K. Ehinger, A. Oliva, and A. Torralba. SUN database: Largescale scene recognition from Abbey to Zoo. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010.
181. C. Xiong, D. M. Johnson, and J. J. Corso. Spectral active clustering via purification of the k-nearest neighbor graph. In Proceedings of European Conference on Data Mining, 2012.
182. A. Yao, J. Gall, C. Leistner, and L. V. Gool. Interactive object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
183. B. Yao and L. Fei-Fei. Grouplet: A structured image representation for recognizing human and object interactions. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010.
184. B. Yao, X. Yang, and S.-C. Zhu. Introduction to a large scale general purpose ground truth dataset: methodology, annotation tool, and benchmarks. In International Conference on Energy Minimization Methods in Computer Vision and Pattern Recognition (EMMCVPR), 2007.
185. B. Yao, X. Jiang, A. Khosla, A. Lin, L. Guibas, and L. Fei-Fei. Human action recognition by learning bases of action attributes and parts. In IEEE International Conference on Computer Vision (ICCV), 2011.
186. G. Ye, Y. Li, H. Xu, D. Liu, and S.-F. Chang. Eventnet: A large scale structured concept library for complex event detection in video. In ACM International Conference on Multimedia, 2015.
187. S. Yeung, O. Russakovsky, N. Jin, M. Andriluka, G. Mori, and L. Fei-Fei. Every moment counts: Dense detailed labeling of actions in complex videos. Computing Research Repository (CoRR), abs/1507.05738, 2015.
188. A. Yu and K. Grauman. Fine-grained visual comparisons with local learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
189. L. Yu, E. Park, A. C. Berg, and T. L. Berg. Visual madlibs: Fill in the blank description generation and question answering. In IEEE International Conference on Computer Vision (ICCV), 2015.
190. J. Yuen, B. Russell, C. Liu, and A. Torralba. Labelme video: Building a video database with human annotations. In IEEE International Conference on Computer Vision (ICCV), 2009.
191. C. Zhang and K. Chaudhuri. Active learning from weak and strong labelers. In Advances in Neural Information Processing Systems (NIPS), 2015.
192. B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning deep features for scene recognition using places database. Advances in Neural Information Processing Systems (NIPS), 2014.
193. Y. Zhu, O. Groth, M. Bernstein, and L. Fei-Fei. Visual7w: Grounded question answering in images. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
194. C. L. Zitnick and D. Parikh. Bringing semantics into focus using visual abstraction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2013.