Deep Multimodal Distance Metric Learning Using Click Constraints for Image Ranking

IEEE Transactions on Cybernetics

Volumn 47, Issue 12, 2017, Pages 4014-4024

  Yu, Jun ^a   Yang, Xiaokang ^b   Gao, Fei ^a   Tao, Dacheng ^c  

^a HANGZHOU DIANZI UNIVERSITY   (China)

^b SHANGHAI JIAO TONG UNIVERSITY   (China)

^c UNIVERSITY OF TECHNOLOGY SYDNEY   (Australia)

Author keywords

Deep learning; distance metric learning (DML); image ranking; multimodal

Indexed keywords

SEMANTICS;

ALTERNATING OPTIMIZATIONS; APPROPRIATE DISTANCES; BENCHMARK DATA; DISTANCE METRIC LEARNING; DISTANCE METRICS; IMAGE RANKINGS; IMAGE SEARCHING; MULTIMODAL FEATURES;

SEARCH ENGINES;

ARTICLE; AUTOENCODER; LEARNING; SEMANTICS;

EID: 84981725681     PISSN: 21682267     EISSN: None     Source Type: Journal    
DOI: 10.1109/TCYB.2016.2591583     Document Type: Article

References (62)

1. W. W. Y. Ng, X. Tian, Y. Lv, D. S. Yeung, and W. Pedrycz, "Incremental hashing for semantic image retrieval in nonstationary environments," IEEE Trans. Cybern., to be published, doi: 10.1109/TCYB.2016.2582530.
2. Z. Wen, B. Hou, and L. Jiao, "Discriminative dictionary learning with two-level low rank and group sparse decomposition for image classification," IEEE Trans. Cybern., to be published, doi: 10.1109/TCYB.2016.2581861.
3. L. Chen, D. Xu, I. W. Tsang, and J. Luo, "Tag-based Web photo retrieval improved by batch mode re-tagging," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., San Francisco, CA, USA, 2010, pp. 3440-3446.
4. Y. Liu, D. Xu, I. W. Tsang, and J. Luo, "Textual query of personal photos facilitated by large-scale Web data," IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 5, pp. 1022-1036, May 2011.
5. L. Yang et al., "A boosting framework for visuality-preserving distance metric learning and its application to medical image retrieval," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 1, pp. 30-44, Jan. 2010.
6. M. S. Lew, N. Sebe, C. Djeraba, and R. Jain, "Content-based multimedia information retrieval: State of the art and challenges," ACM Trans. Multimedia Comput. Commun. Appl., vol. 2, no. 1, pp. 1-19, Feb. 2006.
7. W. Wei et al., "A ranking approach on large-scale graph with multidimensional heterogeneous information," IEEE Trans. Cybern., vol. 46, no. 4, pp. 930-944, Apr. 2016.
8. G. Zhu, S. Yan, and Y. Ma, "Image tag refinement towards lowrank, content-tag prior and error sparsity," in Proc. ACM Int. Conf. Multimedia, Florence, Italy, 2010, pp. 461-470.
9. A. W. M. Smeulders, M. Worring, S. Santini, A. Gupta, and R. Jain, "Content-based image retrieval at the end of the early years," IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 12, pp. 1349-1380, Dec. 2000.
10. R. Veltkamp and M. Tanase, "Content-based image retrieval systems: A survey," Dept. Comput. Sci., Utrecht Univ., Utrecht, The Netherlands, Tech. Rep. TR-UU-CS-2000-34, 2000.
11. M. Flickner et al., "Query by image and video content: The QBIC system," Computer, vol. 28, no. 9, pp. 23-32, Sep. 1995.
12. J. Bach et al., "Virage image search engine: An open framework for image management," in Proc. Stor. Retrieval Image Video Databases, San Jose, CA, USA, 1996, pp. 76-87.
13. A. Pentland, R. W. Picard, and S. Sclaroff, "Photobook: Content-based manipulation of image databases," Int. J. Comput. Vis., vol. 18, no. 3, pp. 233-254, 1996.
14. R. Rahmani, S. A. Goldman, H. Zhang, S. R. Cholleti, and J. E. Fritts, "Localized content-based image retrieval," IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 11, pp. 1902-1912, Nov. 2008.
15. L. Wu, S. C. H. Hoi, R. Jin, J. Zhu, and N. Yu, "Distance metric learning from uncertain side information with application to automated photo tagging," in Proc. ACM Int. Conf. Multimedia, Scottsdale, AZ, USA, 2009, pp. 135-144.
16. D. G. Lowe, "Object recognition from local scale-invariant features," in Proc. IEEE Int. Conf. Comput. Vis., 1999, pp. 1150-1157.
17. L. Wu, S. C. H. Hoi, and N. Yu, "Semantics-preserving bag-of-words models and applications," IEEE Trans. Image Process., vol. 19, no. 7, pp. 1908-1920, Jul. 2010.
18. Y. Xie et al., "Discriminative object tracking via sparse representation and online dictionary learning," IEEE Trans. Cybern., vol. 44, no. 4, pp. 539-553, Apr. 2014.
19. Y. Zhen, Y. Gao, D.-Y. Yeung, H. Zha, and X. Li, "Spectral multimodal hashing and its application to multimedia retrieval," IEEE Trans. Cybern., vol. 46, no. 1, pp. 27-38, Jan. 2016.
20. K. Q. Weinberger, J. Blitzer, and L. K. Saul, "Distance metric learning for large margin nearest neighbor classification," in Proc. Adv. Neural Inf. Process. Syst., Vancouver, BC, Canada, 2006, pp. 1473-1480.
21. Y. Hong, Q. Li, J. Jiang, and Z. Tu, "Learning a mixture of sparse distance metrics for classification and dimensionality reduction," in Proc. IEEE Int. Conf. Comput. Vis., Barcelona, Spain, 2011, pp. 906-913.
22. J. Mei, M. Liu, Y.-F. Wang, and H. Gao, "Learning a Mahalanobis distance-based dynamic time warping measure for multivariate time series classification," IEEE Trans. Cybern., vol. 46, no. 6, pp. 1363-1374, Jun. 2016.
23. M. Loudahi, J. Klein, J.-M. Vannobel, and O. Colot, "Evidential matrix metrics as distances between meta-data dependent bodies of evidence," IEEE Trans. Cybern., vol. 46, no. 1, pp. 109-122, Jan. 2016.
24. M. Guillaumin, J. Verbeek, and C. Schmid, "Multimodal semisupervised learning for image classification," in Proc. CVPR, San Francisco, CA, USA, 2010, pp. 902-909.
25. S. Wang, S. Jiang, Q. Huang, and Q. Tian, "S3MKL: Scalable semisupervised multiple kernel learning for image data mining," in Proc. MM, Florence, Italy, 2010, pp. 163-172.
26. X. Zhu, X. Li, and S. Zhang, "Block-row sparse multiview multilabel learning for image classification," IEEE Trans. Cybern., vol. 46, no. 2, pp. 450-461, Feb. 2016.
27. J. Li, Y. Wu, J. Zhao, and K. Lu, "Low-rank discriminant embedding for multiview learning," IEEE Trans. Cybern., to be published, doi: 10.1109/TCYB.2016.2565898.
28. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," in Proc. Adv. Neural Inf. Process. Syst., 2012, pp. 1097-1105.
29. W. Ouyang et al., "DeepID-Net: Deformable deep convolutional neural networks for object detection," in Proc. CVPR, Boston, MA, USA, 2015, pp. 2403-2412.
30. J. Tompson, R. Goroshin, A. Jain, Y. LeCun, and C. Bregler, "Efficient object localization using convolutional networks," in Proc. CVPR, Boston, MA, USA, 2015, pp. 648-656.
31. C. Szegedy et al., "Going deeper with convolutions," in Proc. CVPR, Boston, MA, USA, 2015, pp. 1-9.
32. R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Columbus, OH, USA, 2014, pp. 580-587.
33. J. Hu, J. Lu, and Y.-P. Tan, "Discriminative deep metric learning for face verification in the wild," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Columbus, OH, USA, 2014, pp. 1875-1882.
34. J. Bromley, I. Guyon, Y. Lecun, E. Sckinger, and R. Shah, "Signature verification using a 'Siamese' time delay neural network," in Proc. Neural Inf. Process. Syst., Denver, CO, USA, 1994.
35. Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle, "Greedy layerwise training of deep networks," in Proc. Adv. Neural Inf. Process. Syst., Vancouver, BC, Canada, 2007, pp. 153-160.
36. B. Du et al., "Stacked convolutional denoising auto-encoders for feature representation," IEEE Trans. Cybern., to be published, doi: 10.1109/TCYB.2016.2536638.
37. G. Dupret and C. Liao, "A model to estimate intrinsic document relevance from the clickthrough logs of a Web search engine," in Proc. ACM Int. Conf. Web Search Data Min., New York, NY, USA, 2010, pp. 181-190.
38. G. Smith and H. Ashman, "Evaluating implicit judgments from image search interactions," in Proc. Web Sci. Conf. Soc. On-Line, Athens, Greece, 2009, pp. 1-3.
39. M. Shokouhi, F. Scholer, and A. Turpin, "Investigating the effectiveness of clickthrough data for document reordering," in Proc. Eur. Conf. Inf. Retrieval, Glasgow, U.K., 2008, pp. 591-595.
40. T. Joachims, "Optimizing search engines using clickthrough data," in Proc. ACM SIGKDD Int. Conf. Knowl. Disc. Data Min., Edmonton, AB, Canada, 2002, pp. 133-142.
41. T.-Y. Liu, "Learning to rank for information retrieval," Found. Trends Inf. Retrieval, vol. 3, no. 3, pp. 225-331, 2009.
42. D. Cossock and T. Zhang, "Statistical analysis of Bayes optimal subset ranking," IEEE Trans. Inf. Theory, vol. 54, no. 11, pp. 5140-5154, Nov. 2008.
43. Z. Dou, R. Song, X. Yuan, and J.-R. Wen, "Are click-through data adequate for learning Web search rankings?" in Proc. ACM Conf. Inf. Knowl. Manag., Napa County, CA, USA, 2008, pp. 73-82.
44. N. Craswell and M. Szummer, "Random walks on the click graph," in Proc. SIGIR, Amsterdam, The Netherlands, 2007, pp. 239-246.
45. V. Jain and M. Varma, "Learning to re-rank: Query-dependent image re-ranking using click data," in Proc. Int. World Wide Web Conf., Hyderabad, India, 2011, pp. 277-286.
46. Y. Pan et al., "Click-through-based cross-view learning for image search," in Proc. 22nd ACM Int. Conf. Res. Develop. Inf. Retrieval, Gold Coast, QLD, Australia, 2014, pp. 717-726.
47. Y.-Y. Chen, Y.-H. Kuo, C.-C. Wu, and W. Hsu, "Visually interpreting names as demographic attributes by exploiting click-through data," in Proc. AAAI, Austin, TX, USA, 2015, pp. 44-50.
48. J. Goldberger, S. Roweis, G. Hinton, and R. Salakhutdinov, "Neighbourhood components analysis," in Proc. Adv. Neural Inf. Process. Syst., Vancouver, BC, Canada, 2005, pp. 513-520.
49. S. C. H. Hoi, W. Liu, M. R. Lyu, and W.-Y. Ma, "Learning distance metrics with contextual constraints for image retrieval," in Proc. Comput. Vis. Pattern Recognit., New York, NY, USA, 2006, pp. 2072-2078.
50. I. Tsochantaridis, T. Joachims, T. Hofmann, and Y. Altun, "Large margin methods for structured and interdependent output variables," J. Mach. Learn. Res., vol. 6, pp. 1453-1484, Sep. 2005.
51. B. Gengtí, L. Yang, C. Xu, and X.-S. Hua, "Content-aware ranking for visual search," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., San Francisco, CA, USA, 2010, pp. 3400-3407.
52. Y. Y. Tang, Y. Wang, L. Li, and C. L. P. Chen, "Structural atomic representation for classification," IEEE Trans. Cybern., vol. 45, no. 12, pp. 2905-2913, Dec. 2015.
53. D. Zhou, O. Bousquet, T. N. Lal, J. Weston, and B. Schölkopf, "Learning with local and global consistency," in Proc. Adv. Neural Inf. Process. Syst., Vancouver, BC, Canada, 2004, pp. 321-328.
54. L. Zhu, J. Shen, H. Jin, R. Zheng, and L. Xie, "Content-based visual landmark search via multimodal hypergraph learning," IEEE Trans. Cybern., vol. 45, no. 12, pp. 2756-2769, Dec. 2015.
55. J. Wu, S. Pan, X. Zhu, C. Zhang, and X. Wu, "Positive and unlabeled multi-graph learning," IEEE Trans. Cybern., to be published, doi: 10.1109/TCYB.2016.2527239.
56. Y. Bengio, "Learning deep architectures for AI," Found. Trends Mach. Learn., vol. 2, no. 1, pp. 1-127, 2009.
57. I. Tsochantaridis, T. Hofmann, T. Joachims, and Y. Altun, "Support vector machine learning for interdependent and structured output spaces," in Proc. Int. Mach. Learn. Conf., Banff, AB, Canada, 2004, pp. 823-830.
58. T. Joachims, "Training linear SVMs in linear time," in Proc. ACM Conf. Knowl. Disc. Data Min., Philadelphia, PA, USA, 2006, pp. 217-226.
59. M. Wang, H. Li, D. Tao, K. Lu, and X. Wu, "Multimodal graph-based reranking for Web image search," IEEE Trans. Image Process., vol. 21, no. 11, pp. 4649-4661, Nov. 2012.
60. Microsoft. (2014). MSR-Bing Image Retrieval Challenge. [Online]. Available: http://research.microsoft.com/en-us/projects/irc/
61. D. Zhou, J. Huang, and B. Schölkopf, "Learning with hypergraphs: Clustering, classification, and embedding," in Proc. Adv. Neural Inf. Process. Syst., Vancouver, BC, Canada, 2006, pp. 1601-1608.
62. K. Järvelin and J. Kekäläinen, "Cumulated gain-based evaluation of IR techniques," ACM Trans. Inf. Syst., vol. 20, no. 4, pp. 422-446, Oct. 2002.