Interactive multimodal visual search on mobile device

IEEE Transactions on Multimedia

Volumn 15, Issue 3, 2013, Pages 594-607

  Li, Houqiang ^a   Wang, Yang ^a   Mei, Tao ^b   Wang, Jingdong ^b   Li, Shipeng ^b  

^a UNIVERSITY OF SCIENCE AND TECHNOLOGY OF CHINA   (China)

^b MICROSOFT RESEARCH ASIA   (China)

Author keywords

Interactive search; Mobile device; Mobile visual search; Multimodal search

Indexed keywords

IMAGE REPRESENTATIONS; INTERACTIVE IMAGES; INTERACTIVE SEARCH; MOBILE VISUAL SEARCHES; MULTI-MODAL INTERACTIONS; MULTIMODAL SEARCH; NATURAL USER INTERACTIONS; SEARCH PERFORMANCE;

IMAGE MATCHING; IMAGE SEGMENTATION; MOBILE DEVICES;

SEARCH ENGINES;

EID: 84897584148     PISSN: 15209210     EISSN: None     Source Type: Journal    
DOI: 10.1109/TMM.2012.2234730     Document Type: Article

References (44)

1. [Online]. Available: http://www.pwc.com/gx/en/communications/review/ features/mobile-data.jhtml
2. H. Bay, T. Tuytelaars, and L. Van Gool, "SURF: Speeded-up robust features," in Proc. ECCV, 2008, vol. 110, no. 3, pp. 346-359.
3. N. Ben-Haim, B. Babendo, and S. Belongie, "Improving web-based image search via content based clustering," in Proc. IEEE Conf. Comput. Vis. Pattern Recogn. Workshop, 2006, p. 106.
4. Y. Cao, H. Wang, C. Wang, Z. Li, L. Zhang, and L. Zhang, "MindFinder: Interactive sketch-based image search on millions of images," in Proc. ACM Multimedia, 2010, pp. 1605-1608.
5. V. Chandrasekhar, D. M. Chen, A. Lin, G. Takacs, S. S. Tsai, N. M. Cheung, Y. Reznik, R. Grzeszczuk, and B. Girod, "Comparison of local feature descriptors for mobile visual search," in Proc. IEEE Int. Conf. Image Process., 2010, pp. 3885-3888.
6. V. Chandrasekhar, G. Takacs, D. Chen, S. Tsai, R. Grzeszczuk, and B. Girod, "CHoG: Compressed histogram of gradients a low bit-rate feature descriptor," in Proc. IEEE Conf. Comput. Vis. Pattern Recogn., Jun. 2009, pp. 2504-2511.
7. T. Chen, M.-M. Cheng, P. Tan, A. Shamir, and S.-M. Hu, "Sketch2photo: Internet image montage," ACM Trans. Graph., Vol. 28, no. 5, pp. 124:1-124:10, Dec. 2009.
8. K. Church, B. Smyth, P. Cotter, and K. Bradley, "Mobile information access: A study of emerging search behavior on the mobile internet," ACM Trans. Web, Vol. 1, no. 1, May 2007.
9. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "ImageNet: A large-scale hierarchical image database," in Proc. IEEE Conf. Comput. Vis. Pattern Recogn., Jun. 2009, pp. 248-255.
10. Digimarc Discover [Online]. Available: https://www.digimarc.com/discover/
11. M. Eitz, K. Hildebrand, T. Boubekeur, and M. Alexa, "An evaluation of descriptors for large-scale image retrieval from sketched feature lines," Comput. Graphics, Vol. 34, no. 5, pp. 482-498, 2010.
12. M. Eitz, K. Hildebrand, T. Boubekeur, and M. Alexa, "Sketch-based image retrieval: Benchmark and bag-of-features descriptors," IEEE Trans. Vis. Comput. Graphics, Vol. 17, no. 11, pp. 1624-1636, Nov. 2011.
13. P. Felzenszwalb and D. Huttenlocher, "Efficient graph-based image segmentation," Int. J. Comput. Vis., Vol. 59, no. 2, pp. 167-181, 2004.
14. Flickr. [Online]. Available: http://www.flickr.net/
15. B. Frey and D. Dueck, "Clustering by passing messages between data points," Science, Vol. 315, no. 5814, pp. 972-976, 2007. (Pubitemid 46281181)
16. M. Gales and S. Young, "The application of hidden Markov models in speech recognition," Foundations and Trends in Signal Process., Vol. 1, no. 3, pp. 195-304, 2008.
17. GazoPa. [Online]. Available: http://www.gazopa.com/
18. B. Girod, V. Chandrasekhar, D. Chen, N. Cheung, R. Grzeszczuk, Y. Reznik, G. Takacs, S. Tsai, and R. Vedantham, "Mobile visual search," IEEE Signal Process. Mag., Vol. 28, no. 4, pp. 61-76, 2011.
19. Google. [Online]. Available: http://www.google.com/
20. Google Goggles. [Online]. Available: http://www.google.com/mobile/ goggles/
21. D. Hoiem, A. Efros, and M. Hebert, "Recovering surface layout from an image," Int. J. Comput. Vis., Vol. 75, no. 1, pp. 151-172, 2007. (Pubitemid 47120250)
22. M. Jia, X. Fan, X. Xie, M. Li, and W. Ma, "Photo-to-search: Using camera phones to inquire of the surrounding world," in Proc. Mobile Data Manag., 2006.
23. F. Jing, M. Li, H.-J. Zhang, and B. Zhang, "An efficient and effective region-based image retrieval framework," IEEE Trans. Image Process., Vol. 13, no. 5, pp. 699-709, 2004.
24. J. Kim and K. Grauman, "Boundary preserving dense local regions," in Proc. IEEE Conf. Comput. Vis. Pattern Recogn., 2011, pp. 1553-1560.
25. M. Li, "Texture moment for content-based image retrieval," in Proc. IEEE Int. Conf. Multimedia Expo, 2007, pp. 508-511.
26. LinkMe Mobile. [Online]. Available: http://www.snapnow.co.uk/
27. X. Liu, Y. Lou, A. W. Yu, and B. Lang, "Search by mobile image based on visual and spatial consistency," in Proc. IEEE Int. Conf. Multimedia Expo, 2011, pp. 1-6.
28. D. Lowe, "Distinctive image features from scale-invariant keypoints," Int. J. Comput. Vis., Vol. 60, no. 2, pp. 91-110, 2004.
29. G. Miller, "WordNet: A lexical database for English," Commun. ACM, Vol. 38, no. 11, pp. 39-41, 1995.
30. NOKIA Point and Find. [Online]. Available: http://pointandfind.nokia.com/
31. M. Porter, "An algorithm for suffix stripping," Program: Electron. Library Inf. Syst., Vol. 14, no. 3, pp. 130-137, 1993.
32. C. Rother, V. Kolmogorov, and A. Blake, "GrabCut: Interactive foreground extraction using iterated graph cuts," ACM Trans. Graph., Vol. 23, no. 3, pp. 309-314, Aug. 2004.
33. E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, "ORB: An efficient alternative to SIFT or SURF," in Proc. IEEE Int. Conf. Comput. Vis., Nov. 2011, pp. 2564-2571.
34. J. Shi and J. Malik, "Normalized cuts and image segmentation," IEEE Trans. Pattern Anal. Mach. Intell., Vol. 22, no. 8, pp. 888-905, Aug. 2000.
35. Siri. [Online]. Available: http://www.apple.com/iphone/features/siri.html
36. Smart VisualTM Kooaba. [Online]. Available: http://www.kooaba.com/
37. SnapTell. [Online]. Available: http://www.snaptell.com/
38. TinEye. [Online]. Available: http://www.tineye.com/
39. S. S. Tsai, D. Chen, H. Chen, C.-H. Hsu, K.-H. Kim, J. P. Singh, and B. Girod, "Combining image and text features: a hybrid approach to mobile book spine recognition," in Proc. ACM Multimedia, 2011, pp. 1029-1032.
40. S. Tsai, H. Chen, D. Chen, G. Schroth, R. Grzeszczuk, and B. Girod, "Mobile visual search on printed documents using text and low bitrate features," in Proc. IEEE Int. Conf. Image Process., Sep. 2011, pp. 2601-2604.
41. C. Wang, Z. Li, and L. Zhang, "MindFinder: Image search by interactive sketching and tagging," in Proc. Int. Conf. World Wide Web, 2010, pp. 1309-1312.
42. Y. Wang, T. Mei, J. Wang, H. Li, and S. Li, "JIGSAW: Interactive mobile visual search with multimodal queries," in Proc. ACM Multimedia, 2011, pp. 73-82.
43. H. Xu, J. Wang, X. Hua, and S. Li, "Image search by concept map," in Proc. ACM SIGIR, 2010, pp. 275-282.
44. Z. Zha, L. Yang, T. Mei, M. Wang, and Z. Wang, "Visual query suggestion," in Proc. ACM Multimedia, 2009, pp. 15-24.