ImageSpirit: Verbal guided image parsing

ACM Transactions on Graphics

Volumn 34, Issue 1, 2014, Pages

  Cheng, Ming Ming ^a   Zheng, Shuai ^a   Lin, Wen Yan ^b   Vineet, Vibhav ^b   Sturgess, Paul ^b   Crook, Nigel ^b   Mitra, Niloy J ^c   Torr, Philip ^a  

^a UNIVERSITY OF OXFORD   (United Kingdom)

^b OXFORD BROOKES UNIVERSITY   (United Kingdom)

^c UNIVERSITY COLLEGE LONDON   (United Kingdom)

Author keywords

Image parsing; Multilabel CRF; Natural language control; Object class segmentation; Speech interface; Visual attributes

Indexed keywords

ECONOMIC AND SOCIAL EFFECTS; MAMMALS; NATURAL LANGUAGE PROCESSING SYSTEMS; PIXELS; SEMANTICS; VISUAL LANGUAGES;

IMAGE PARSING; MULTI-LABEL; NATURAL LANGUAGES; OBJECT CLASS; SPEECH INTERFACE; VISUAL ATTRIBUTES;

IMAGE SEGMENTATION;

EID: 84905732098     PISSN: 07300301     EISSN: 15577368     Source Type: Journal    
DOI: 10.1145/2682628     Document Type: Review

References (58)

1. A. Adams, J. Baek, and M. A. Davis. 2010. Fast high-dimensional filtering using the permutohedral lattice. Comput. Graph. Forum. 29, 2, 753-762.
2. C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman. 2009. Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Trans. Graph. 3, 24:1-11.
3. B. Berlin and P. Kay. 1991. Basic Color Terms: Their Universality and Evolution. University of California Press.
4. F. Berthouzoz, W. Li, M. Dontcheva, and M. Agrawala. 2011. A framework for content-adaptive photo manipulation macros: Application to face, landscape, and global manipulations. ACM Trans. Graph. 30, 5, 120.
5. th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'80). 262-270.
6. Y. Boykov and M.-P. Jolly. 2001. Interactive graph cuts for optimal boundary and region segmentation of objects in n-d images. In Proceedings of the IEEE International Conference on Computer Vision (ICCV'01). 105-112.
7. S. Branson, C. Wah, F. Schroff, B. Babenko, P. Welinder, P. Perona, and S. Belongie. 2010. Visual recognition with humans in the loop. In Proceedings of the European Conference on Computer Vision (ECCV'10). 438-451.
8. R. Carroll, A. Agarwala, and M. Agrawala. 2010. Image warps for artistic perspective manipulation. ACM Trans. Graph. 29, 4, 127.
9. T. Chen, M.-M. Cheng, P. Tan, A. Shamir, and S.-M. Hu. 2009. Sketch2photo: Internet image montage. ACM Trans. Graph. 28, 5, 124:1-10.
10. M.-M. Cheng, N. J. Mitra, X. Huang, P. Torr, and S.-M. Hu. 2014a. Global contrast based salient region detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'14). 409-416.
11. M.-M. Cheng, Z. Zhang, W.-Y. Lin, and P. H. S. Torr. 2014b. BING: Binarized normed gradients for objectness estimation at 300fps. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'14).
12. M.-M. Cheng, F.-L. Zhang, N. J. Mitra, X. Huang, and S.-M. Hu. 2010. RepFinder: Finding approximately repeated scene elements for image editing. ACM Trans. Graph. 29, 4, 83:1-8.
13. T. Dean, M. A. Ruzon, M. Segal, J. Shlens, S. Vijaya-Narasimhan, and J. Yagnik. 2013. Fast, accurate detection of 100, 000 object classes on a single machine. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'13). 1814-1821.
14. th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'01). 341-346.
15. A. Farhadi, I. Endres, and D. Hoiem. 2010. Attribute-centric recognition for cross-category generalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'10).
16. A. Farhadi, I. Endres, D. Hoiem, and D. Forsyth. 2009. Describing objects by their attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'09). 1778-1785.
17. P. Felzenszwalb and D. Huttenlocher. 2004. Efficient graph-based image segmentation. Int. J. Comput. Vis. 59, 2, 167-181.
18. V. Ferrari and A. Zisserman. 2007. Learning visual attributes. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS'07).
19. C. Goldberg, T. Chen, F.-L. Zhang, A. Shamir, and S.-M. Hu. 2012. Datadriven object manipulation in images. Comput. Graph. Forum 31, 2.1, 265-274.
20. S. Henderson. 2008. Augmented reality for maintenance and repair. http://www.youtube.com/watch?v=mn-zvymlSvk.
21. F. S. Khan, R. Anwer, J. Van De Weijer, A. Bagdanov, M. Van-Rell, and A. Lopez. 2012. Color attributes for object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'12). 3306-3313.
22. D. Koller and N. Friedman. 2009. Probabilistic Graphical Models: Principles and Techniques. MIT Press.
23. P. Krahenbuhl and V. Koltun. 2011. Efficient inference in fully connected crfs with gaussian edge potentials. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS'11).
24. G. Kulkarni, V. Premraj, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg. 2011. Baby talk: Understanding and generating simple image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'11). 1601-1608.
25. L. Ladicky, C. Russell, P. Kohli, and P. H. S. Torr. 2009. Associative hierarchical crfs for object class image segmentation. In Proceedings of the IEEE Conference on Computer Vision (ICCV'09).
26. L. Ladicky, P. Sturgess, C. Russell, S. Sengupta, Y. Bastanlar, W. F. Clocksin, and P. H. S. Torr. 2010. Joint optimisation for object class segmentation and dense stereo reconstruction. In Proceedings of the British Machine Vision Conference (BMVC'10). 104.1-104.11.
27. J. Lalonde, D. Hoiem, A. Efros, C. Rother, J. Winn, and A. Criminisi. 2007. Photo clip art. ACM Trans. Graph. 26, 3.
28. C. H. Lampert, H. Nickisch, and S. Harmeling. 2009. Learning to detect unseen object classes by between-class attribute transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'09). 951-958.
29. G. Laput, M. Dontcheva, G. Wilensky, W. Chang, A. Agarwala, J. Linder, and E. Adar. 2013. Pixeltone: A multimodal interface for image editing. In Extended Abstracts on Human Factors in Computing Systems (CHI/EA'13). 2829-2830.
30. V. Lempitsky, P. Kohli, C. Rother, and T. Sharp. 2009. Image segmentation with a bounding box prior. In Proceedings of the IEEE International Conference on Computer Vision (ICCV'09). 277-284.
31. A. Levin, D. Lischinski, and Y. Weiss. 2008. A closed-form solution to natural image matting. IEEE Trans. Pattern Mach. Intell. 30, 2, 228-242.
32. Y. Li, J. Sun, C.-K. Tang, and H.-Y. Shum. 2004. Lazy snapping. ACM Trans. Graph. 23, 3, 303-308.
33. J. Liu, J. Sun, and H.-Y. Shum. 2009. Paint selection. ACM Trans. Graph. 28, 3.
34. T. Malisiewicz and A. A. Efros. 2008. Recognition by association via learning per-exemplar distances. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08). 1-8.
35. Microsoft. 2012. Microsoft speech platform-Sdk. http://www.microsoft.com/download/details.aspx?id=27226.
36. G. Patterson and J. Hays. 2012. Sun attribute database: Discovering, annotating, and recognizing scene attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'12). 2751-2758.
37. R. B. Potts. 1952. Some generalized order-disorder transformations. Math. Proc. Cambridge Philos. Soc. 48, 1, 106-109.
38. A. Rabinovich, A. Vedaldi, C. Galleguillos, E. Wiewiora, and S. Belongie. 2007. Objects in context. In Proceedings of the IEEE International Conference on Computer Vision (ICCV'07). 1-8.
39. C. Rother, V. Kolmogorov, and A. Blake. 2004. Grabcut: Interactive foreground extraction using iterated graph cuts. ACM Trans. Graph. 23, 3, 309-314.
40. J. Shotton, J. Winn, C. Rother, and A. Criminisi. 2009. Textonboost for image understanding: Multi-class object recognition and segmentation by jointly modeling texture, layout, and context. Int. J. Comput. Vis. 81, 1, 2-23.
41. N. Silberman, D. Hoiem, P. Kohli, and R. Fergus. 2012. Indoor segmentation and support inference from rgbd images. In Proceedings of the European Conference on Computer Vision (ECCV'12). 746-760.
42. P. Sturgess, L. Ladicky, N. Crook, and P. H. S. Torr. 2012. Scalable cascade inference for semantic image segmentation. In Proceedings of the British Machine Vision Conference (BMVC'12).
43. J. Sun, L. Yuan, J. Jia, and H.-Y. Shum. 2005. Image completion with structure propagation. ACM Trans. Graph. 24, 3, 861-868.
44. Sunnybrook Hospital. 2008. Xbox kinect in the hospital operating room. http://www.youtube.com/watch?v=f5Ep3oqicVU.
45. C. Sutton, K. Rohanimanesh, and A. McCallum. 2004. Dynamic conditional random fields: Factorized probabilistic models for labeling and segmenting sequence data. In Proceedings of the International Conference on Machine Learning (ICML'04).
46. J. Tighe and S. Lazebnik. 2011. Understanding scenes on many levels. In Proceedings of the IEEE International Conference on Computer Vision (ICCV'11). 335-342.
47. J. Tighe and S. Lazebnik. 2013. SuperParsing: Scalable nonparametric image parsing with superpixels. Int. J. Comput. Vis. 101, 2, 329-349.
48. G. Tsoumakas, A. Dimou, E. Spyromitros-Xioufis, V. Mezaris, I. Kompatsiaris, and I. Vlahavas. 2009. Correlation-based pruning of stacked binary relevance models for multi-label learning. In Proceedings of the ECML/PKDD Workshop on Learning from Multi-Label Data (MLD'09).
49. Z. Tu, X. Chen, A. L. Yuille, and S.-C. Zhu. 2005. Image parsing: Unifying segmentation, detection, and recognition. Int. J. Comput. Vis. 63, 2, 113-140.
50. J. Valentin, V. Vineet, M.-M. Cheng, D. Kim, S. Izadi, J. Shotton, P. Kohli, M. Niessner, A. Criminisi, and P. Torr. 2014. SemanticPaint: Interactive 3d labeling and learning at your fingertips. ACM Trans. Graph. (to appear).
51. K. Van De Sande, T. Gevers, and C. G. M. Snoek. 2010. Evaluating color descriptors for object and scene recognition. IEEE Trans. Pattern Anal. Mach. Intell. 32, 9, 1582-1596.
52. J. Verbeek and W. Triggs. 2007. Scene segmentation with crfs learned from partially labeled images. Adv. Neural Inf. Process. Syst. 20, 1553-1560.
53. C. Wah, S. Branson, P. Perona, and S. Belongie. 2011. Multi-class recognition and part localization with humans in the loop. In Proceedings of the IEEE Conference on Computer Vision (ICCV'11). 2524-2531.
54. th European Conference on Computer Vision (ECCV'10). 155-168.
55. J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba. 2010. Sun database: Large-scale scene recognition from abbey to zoo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'10). 3485-3492.
56. S. Zheng, M.-M. Cheng, J. Warrell, P. Sturgess, V. Vineet, C. Rother, and P. Torr. 2014. Dense semantic image segmentation with objects and attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'14). 3214-3221.
57. Y. Zheng, X. Chen, M.-M. Cheng, K. Zhou, S.-M. Hu, and N. J. Mitra. 2012. Interactive images: Cuboid proxies for smart image manipulation. ACM Trans. Graph. 31, 4, 99:1-11.
58. S. Zhou, H. Fu, L. Liu, D. Ohen-Or, and X. Han. 2010. Parametric reshaping of human bodies in images. ACM Trans. Graph. 29, 4, 126:1-126:10.