Best of both worlds: Human-machine collaboration for object annotation

Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

Volumn 07-12-June-2015, Issue , 2015, Pages 2121-2131

  Russakovsky, Olga ^a   Li, Li Jia ^b   Fei Fei, Li ^a  

^a STANFORD UNIVERSITY   (United States)

^b SNAP INC   (United States)

Author keywords

[No Author keywords available]

Indexed keywords

COMPUTER VISION; MARKOV PROCESSES; OBJECT RECOGNITION; PATTERN RECOGNITION;

ENGINEERING INNOVATIONS; HUMAN-IN-THE-LOOP; HUMAN-MACHINE COLLABORATION; MARKOV DECISION PROCESSES; MULTIPLE COMPUTERS; MULTIPLE SOURCE; OBJECT DETECTORS; STATE OF THE ART;

OBJECT DETECTION;

EID: 84959184491     PISSN: 10636919     EISSN: None     Source Type: Conference Proceeding    
DOI: 10.1109/CVPR.2015.7298824     Document Type: Conference Paper

References (65)

1. B. Alexe, T. Deselares, and V. Ferrari. Measuring the objectness of image windows. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012
2. A. C. Berg, T. L. Berg, H. Daume, J. Dodge, A. Goyal, X. Han, A. Mensch, M. Mitchell, A. Sood, K. Stratos, et al. Understanding and predicting importance in images. In Computer Vision and Pattern Recognition (CVPR), 2012
3. A. Biswas and D. Parikh. Simultaneous active learning of classifiers & attributes via relative feedback. In Computer Vision and Pattern Recognition (CVPR), 2013
4. J. Bragg, Mausam, and D. S. Weld. Crowdsourcing multilabel classification for taxonomy creation. In HCOMP'13, 2013
5. S. Branson, K. E. Hjorleifsson, and P. Perona. Active annotation translation. In Computer Vision and Pattern Recognition (CVPR), 2014
6. S. Branson, C. Wah, F. Schroff, B. Babenko, P. Welinder, P. Perona, and S. Belongie. Visual recognition with humans in the loop. In European Conference on Computer Vision (ECCV), 2010
7. Q. Chen, Z. Song, Z. Huang, Y. Hua, and S. Yan. Contextualizing object detection and classification. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014
8. X. Chen, A. Shrivastava, and A. Gupta. NEIL: Extracting Visual Knowledge from Web Data. In International Confer-ence on Computer Vision (ICCV), 2013
9. S. Cooper, F. Khatib, A. Treuille, J. Barbero, J. Lee, M. Beenen, A. Leaver-Fay, D. Baker, and Z. Popovic. Predicting protein structures with a multiplayer online game. Nature, 466:756-760, 2010
10. P. Dai, D. S. Weld, et al. Decision-theoretic control of crowdsourced workflows. In Twenty-Fourth AAAI Conference on Artificial Intelligence, 2010
11. T. Dean, M. Ruzon, M. Segal, J. Shlens, S. Vijayanarasimhan, and J. Yagnik. Fast, accurate detection of 100,000 object classes on a single machine. In Computer Vision and Pattern Recognition (CVPR), 2013
12. J. Deng, J. Krause, and L. Fei-Fei. Fine-grained crowdsourcing for fine-grained recognition. In Computer Vision and Pattern Recognition (CVPR), 2013
13. J. Deng, O. Russakovsky, J. Krause, M. Bernstein, A. C. Berg, and L. Fei-Fei. Scalable multi-label annotation. In CHI, 2014
14. C. Desai, D. Ramanan, and C. Fowlkes. Discriminative models for multi-class object layout. International Journal on Computer Vision, 2011
15. S. Divvala, A. Farhadi, and C. Guestrin. Learning everything about anything: Webly-supervised visual concept learning. In Computer Vision and Pattern Recognition (CVPR), 2014
16. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The Pascal Visual Object Classes (VOC) challenge. International Journal on Computer Vi-sion, 88(2):303-338, June 2010
17. P. Felzenszwalb, R. Girshick, D. McAllester, and D. Ramanan. Object detection with discriminatively trained part based models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 32, 2010
18. H. Gao, G. Barbier, and R. Goolsby. Harnessing the crowdsourcing power of social media for disaster relief. IEEE In-telligent Systems, 26(3):10-14, 2011
19. R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Computer Vision and Pattern Recognition (CVPR), 2014
20. R. B. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation (v4). CoRR, abs/1311. 2524. v4, 2013
21. B. Goodrich and I. Arel. Reinforcement learning based visual attention with application to face detection. In Computer Vision and Pattern Recognition (CVPR), 2012
22. M. R. Greene. Statistics of high-level scene context. Fron-tiers in Psychology, 4, 2013
23. G. Hartmann, M. Grundmann, J. Hoffman, D. Tsai, V. Kwatra, O. Madani, S. Vijayanarasimhan, I. Essa, J. Rehg, and R. Sukthankar. Weakly supervised learning of object segmentations from web-scale video. InWorkshop onWeb-scale Vision and Social Media, European Conference on Computer Vision (ECCV), 2012
24. J. Hoffman, S. Guadarrama, E. Tzeng, J. Donahue, R. Girshick, T. Darrell, and K. Saenko. LSDA: Large scale detection through adaptation. CoRR, abs/1407. 5035, 2014
25. S. J. Hwang and K. Grauman. Reading between the lines: Object localization using implicit cues from image tags. IEEE Transactions on Pattern Analysis and Machine Intel-ligence, 34(6):1145-1158, 2012
26. S. D. Jain and K. Grauman. Predicting sufficient annotation strength for interactive foreground segmentation. December 2013
27. Y. Jia. Caffe: An open source convolutional architecture for fast feature embedding. http://caffe. berkeleyvision. org/, 2013
28. E. Kamar, S. Hacker, and E. Horvitz. Combining human and machine intelligence in large-scale crowdsourcing. In AAMAS, 2012
29. S. Karayev, M. Fritz, and T. Darrell. Anytime recognition of objects and scenes. In Computer Vision and Pattern Recog-nition (CVPR), 2014
30. S. Kazemzadeh, V. Ordonez, M. Matten, and T. L. Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, 2014
31. M. Khodabandeh, A. Vahdat, G. Zhou, H. Hajimirsadeghi, M. J. Roshtkhari, G. Mori, and S. Se. Discovering human interactions in videos with limited data labeling. CoRR, abs/1502. 03851, 2015
32. A. Kovashka, S. Vijayanarasimhan, and K. Grauman. Actively selecting annotations among objects and attributes. In-ternational Conference on Computer Vision (ICCV), 2011
33. A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS, 2012
34. S. Lad and D. Parikh. Interactively guiding semi-supervised clustering via attribute-based explanations. In European Conference on Computer Vision (ECCV). 2014
35. C. Li, D. Parikh, and T. Chen. Automatic discovery of groups of objects for scene understanding. In Computer Vision and Pattern Recognition (CVPR), 2012
36. L.-J. Li, R. Socher, and L. Fei-Fei. Towards total scene understanding:classification, annotation and segmentation in an automatic framework. In Computer Vision and Pattern Recognition (CVPR), 2009
37. T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollr, and C. L. Zitnick. Microsoft COCO: Common Objects in Context. In European Conference on Com-puter Vision (ECCV), 2014
38. C. J. Lintott, K. Schawinski, A. Slosar, K. Land, S. Bamford, D. Thomas, M. J. Raddick, R. C. Nichol, A. Szalay, D. Andreescu, P. Murray, and J. Vandenberg. Galaxy zoo: morphologies derived from visual inspection of galaxies from the sloan digital sky survey. MNRAS, 389(3):1179-1189, 2008
39. D. Parikh and K. Grauman. Relative attributes. In Interna-tional Conference on Computer Vision (ICCV), pages 503-510. IEEE, 2011
40. A. Parkash and D. Parikh. Attributes for classifier feedback. In European Conference on Computer Vision (ECCV), 2012
41. J. Peng, Q. Liu, A. Ihler, and B. Berger. Crowdsourcing for structured labeling with applications to protein folding. In Proc. ICML Machine Learning Meets Crowdsourcing Work-shop, 2013
42. A. Prest, C. Leistner, J. Civera, C. Schmid, and V. Ferrari. Learning object class detectors fromweakly annotated video. In Computer Vision and Pattern Recognition (CVPR), 2012
43. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. CoRR, abs/1409. 0575, 2014
44. M. A. Sadeghi and A. Farhadi. Recognition using visual phrases. 2011
45. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun. Overfeat: Integrated recognition, localization and detection using convolutional networks. CoRR, abs/1312. 6229, 2013
46. V. S. Sheng, F. Provost, and P. G. Ipeirotis. Get another label? Improving data quality and data mining using multiple, noisy labelers. In SIGKDD, 2008
47. B. Siddiquie and A. Gupta. Beyond active noun tagging: Modeling contextual interactions for multi-class active learning. In Computer Vision and Pattern Recognition (CVPR), 2010
48. Z. Song, Q. Chen, Z. Huang, Y. Hua, and S. Yan. Contextualizing object detection and classification. In Computer Vision and Pattern Recognition (CVPR), 2011
49. M. Spain and P. Perona. Some objects are more equal than others: Measuring and predicting importance. In European Conference on Computer Vision (ECCV), 2008
50. H. Su, J. Deng, and L. Fei-Fei. Crowdsourcing annotations for visual object detection. In AAAI Human Computation Workshop, 2012
51. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. CoRR, abs/1409. 4842, 2014
52. K. Tang, R. Sukthankar, J. Yagnik, and L. Fei-Fei. Discriminative segment annotation in weakly labeled video. In Com-puter Vision and Pattern Recognition (CVPR), 2013
53. K. E. A. van de Sande, C. G. M. Snoek, and A. W. M. Smeulders. Fisher and vlad with flair. In Computer Vision and Pattern Recognition (CVPR), 2014
54. S. Vijayanarasimhan and K. Grauman. Multi-level active prediction of useful image annotations for recognition. In NIPS, 2009
55. S. Vijayanarasimhan and K. Grauman. Predicting effort vs. informativeness for multi-label image annotations. In Com-puter Vision and Pattern Recognition (CVPR), 2009
56. S. Vijayanarasimhan and K. Grauman. Large-scale live active learning: Training object detectors with crawled data and crowds. International Journal on Computer Vision, 108(1-2), 2014
57. L. von Ahn and L. Dabbish. Esp: Labeling images with a computer game. In AAAI Spring Symposium: Knowledge Collection from Volunteer Contributors, 2005
58. C. Vondrick, D. Patterson, and D. Ramanan. Efficiently scaling up crowdsourced video annotation. International Journal on Computer Vision, 2013
59. C. Wah, S. Branson, P. Perona, and S. Belongie. Multiclass recognition and part localization with humans in the loop. In International Conference on Computer Vision (ICCV), 2011
60. C. Wah, G. V. Horn, S. Branson, S. Maji, P. Perona, and S. Belongie. Similarity comparisons for interactive finegrained categorization. In Computer Vision and Pattern Recognition (CVPR), 2014
61. X. Wang, M. Yang, S. Zhu, and Y. Lin. Regionlets for generic object detection. In International Conference on Computer Vision (ICCV), 2013
62. D. S. Weld, P. Dai, et al. Human intelligence needs artificial intelligence. In Workshops at the Twenty-Fifth AAAI Confer-ence on Artificial Intelligence, 2011
63. C. Wojek, S. Walk, S. Roth, K. Schindler, and B. Schiele. Monocular visual scene understanding: Understanding multi-object traffic scenes. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013
64. J. Xiao, J. Hays, K. Ehinger, A. Oliva, and A. Torralba. SUN database: Large-scale scene recognition from Abbey to Zoo. Computer Vision and Pattern Recognition (CVPR), 2010
65. P. Zhang, J. Wang, A. Farhadi, M. Hebert, and D. Parikh. Predicting failures of vision systems. In Computer Vision and Pattern Recognition (CVPR), 2014.