Much Ado About Time: Exhaustive Annotation of Temporal Data

Proceedings of the 4th AAAI Conference on Human Computation and Crowdsourcing, HCOMP 2016

Volumn , Issue , 2016, Pages 219-228

  Sigurdsson, Gunnar A ^a   Russakovsky, Olga ^a   Farhadi, Ali ^b,c   Laptev, Ivan ^d   Gupta, Abhinav ^a,c  

^a CARNEGIE MELLON UNIVERSITY   (United States)

^b UNIVERSITY OF WASHINGTON   (United States)

^c Allen Institute for Cell Science

^d INRIA   (France)

Author keywords

[No Author keywords available]

Indexed keywords

ARTIFICIAL INTELLIGENCE; COST EFFECTIVENESS; INVESTMENTS;

AI SYSTEMS; ANNOTATED DATASETS; IMAGE ANNOTATION; INPUT IMAGE; LARGE-SCALES; LEARN+; MULTI-LABEL ANNOTATION; TEMPORAL DATA; VIDEO-CLIPS; WORKERS';

COST BENEFIT ANALYSIS;

EID: 85167598864     PISSN: None     EISSN: None     Source Type: Conference Proceeding    
DOI: None     Document Type: Conference Paper

References (38)

1. Bigham, J. P.; Jayant, C.; Ji, H.; Little, G.; Miller, A.; Miller, R. C.; Miller, R.; et al. 2010. VizWiz: nearly real-time answers to visual questions. In User Interface Software and Technology (UIST).
2. Bragg, J.; Weld, D. S.; et al. 2013. Crowdsourcing multi-label classification for taxonomy creation. In Human Computation and Crowdsourcing (HCOMP).
3. Coan, J. A., and Allen, J. J. B. 2007. Handbook of Emotion Elicitation and Assessment. New York: Oxford University Press.
4. Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. ImageNet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition (CVPR).
5. Deng, J.; Russakovsky, O.; Krause, J.; Bernstein, M. S.; Berg, A.; and Fei-Fei, L. 2014. Scalable multi-label annotation. In SIGCHI Conference on Human Factors in Computing Systems.
6. Fathi, A.; Balcan, M.; Ren, X.; and Rehg, J. 2011. Combining self training and active learning for video segmentation. In British Machine Vision Conference (BMVC).
7. Geiger, A.; Lenz, P.; and Urtasun, R. 2012. Are we ready for autonomous driving? The KITTI vision benchmark suite. In Computer Vision and Pattern Recognition (CVPR).
8. Gorban, A.; Idrees, H.; Jiang, Y.-G.; Roshan Zamir, A.; and Laptev. 2015. THUMOS challenge: Action recognition with a large number of classes. http://www.thumos.info/.
9. Heilbron, F. C., and Niebles, J. C. 2014. Collecting and annotating human activities in web videos. In Proceedings of International Conference on Multimedia Retrieval, 377. ACM.
10. Heilbron, F. C.; Escorcia, V.; Ghanem, B.; and Niebles, J. C. 2015. ActivityNet: A large-scale video benchmark for human activity understanding. In Computer Vision and Pattern Recognition (CVPR).
11. Karpathy, A.; Toderici, G.; Shetty, S.; Leung, T.; et al. 2014. Largescale video classification with convolutional neural networks. In Computer Vision and Pattern Recognition (CVPR).
12. Krishna, R.; Hata, K.; Chen, S.; Kravitz, J.; Shamma, D. A.; et al. 2016a. Embracing error to enable rapid crowdsourcing. In SIGCHI Conference on Human Factors in Computing Systems.
13. Krishna, R.; Zhu, Y.; Groth, O.; Johnson, J.; Hata, K.; Kravitz, J.; et al. 2016b. Visual genome: Connecting language and vision using crowdsourced dense image annotations. CoRR abs/1602.07332.
14. Kuehne, H.; Jhuang, H.; Garrote, E.; Poggio, T.; and Serre, T. 2011. HMDB: a large video database for human motion recognition. In International Conference on Computer Vision (ICCV).
15. Lasecki, W. S.; Gordon, M.; Koutra, D.; Jung, M. F.; et al. 2014. Glance: Rapidly coding behavioral video with the crowd. In User Interface Software and Technology (UIST).
16. Li, T., and Ogihara, M. 2003. Detecting emotion in music. In Proceedings of the fourth international conference on music information retrieval (ICMIR), volume 3, 239-240.
17. Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; et al. 2014. Microsoft COCO: Common Objects in Context. In European Conference on Computer Vision (ECCV).
18. Miller, G. A. 1956. The magical number seven, plus or minus two: some limits on our capacity for processing information. Psychological review 63(2):81.
19. Noronha, J.; Hysen, E.; Zhang, H.; and Gajos, K. Z. 2011. Platemate: Crowdsourcing nutritional analysis from food photographs. In User Interface Software and Technology (UIST).
20. Patterson, G.; Xu, C.; Su, H.; and Hays, J. 2014. The SUN attribute database: Beyond categories for deeper scene understanding. International Journal of Computer Vision 108(1-2).
21. Patterson, G.; Van Horn, G.; Belongie, S.; Perona, P.; and Hays, J. 2015. Tropel: Crowdsourcing detectors with minimal training. In Human Computation and Crowdsourcing (HCOMP).
22. Salisbury, E.; Stein, S.; and Ramchurn, S. 2015. Crowdar: augmenting live video with a real-time crowd. In Human Computation and Crowdsourcing (HCOMP).
23. Schapire, R. E., and Singer, Y. 2000. Boostexter: A boosting-based system for text categorization. Machine learning 39(2):135-168.
24. Sheshadri, A., and Lease, M. 2013. Square: A benchmark for research on computing crowd consensus. In Human Computation and Crowdsourcing (HCOMP).
25. Sigurdsson, G. A.; Varol, G.; Wang, X.; et al. 2016. Hollywood in homes: Crowdsourcing data collection for activity understanding. In European Conference on Computer Vision (ECCV).
26. Soomro, K.; Roshan Zamir, A.; and Shah, M. 2012. UCF101: A dataset of 101 human actions classes from videos in the wild. In CRCV-TR-12-01.
27. Thorpe, S.; Fize, D.; Marlot, C.; et al. 1996. Speed of processing in the human visual system. Nature 381(6582):520-522.
28. Ueda, N., and Saito, K. 2002. Parametric mixture models for multilabeled text. In Neural Information Processing Systems (NIPS).
29. Vijayanarasimhan, S., and Grauman, K. 2012. Active frame selection for label propagation in videos. In European Conference on Computer Vision (ECCV).
30. Von Ahn, L., and Dabbish, L. 2004. Labeling images with a computer game. In SIGCHI conference on Human factors in computing systems.
31. Von Ahn, L.; Liu, R.; and Blum, M. 2006. Peekaboom: A game for locating objects in images. In SIGCHI Conference on Human Factors in Computing Systems.
32. Vondrick, C., and Ramanan, D. 2011. Video Annotation and Tracking with Active Learning. In Advances in Neural Information Processing Systems (NIPS).
33. Vondrick, C.; Patterson, D.; and Ramanan, D. 2013. Efficiently scaling up crowdsourced video annotation. International Journal of Computer Vision 101(1).
34. Xiao, J.; Ehinger, K. A.; Hays, J.; Torralba, A.; and Oliva, A. 2014. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision (IJCV).
35. Ye, G.; Li, Y.; Xu, H.; Liu, D.; and Chang, S.-F. 2015. EventNet: A large scale structured concept library for complex event detection in video. In ACM International Conference on Multimedia.
36. Yeung, S.; Russakovsky, O.; Jin, N.; Andriluka, M.; Mori, G.; and Fei-Fei, L. 2015. Every moment counts: Dense detailed labeling of actions in complex videos. CoRR abs/1507.05738.
37. Yuen, J.; Russell, B.; Liu, C.; and Torralba, A. 2009. LabelMe video: Building a video database with human annotations. In International Conference on Computer Vision (ICCV).
38. Zhong, Y.; Lasecki, W. S.; et al. 2015. Region Speak: Quick comprehensive spatial descriptions of complex images for blind users. In SIGCHI Conference on Human Factors in Computing Systems.