Multi-focus image fusion with a deep convolutional neural network

Information Fusion

Volumn 36, Issue , 2017, Pages 191-207

  Liu, Yu ^a   Chen, Xun ^a   Peng, Hu ^a   Wang, Zengfu ^b  

^a HEFEI UNIVERSITY OF TECHNOLOGY   (China)

^b UNIVERSITY OF SCIENCE AND TECHNOLOGY OF CHINA   (China)

Author keywords

Activity level measurement; Convolutional neural networks; Deep learning; Fusion rule; Image fusion; Multi focus image fusion

Indexed keywords

CONVOLUTION; LEVEL MEASUREMENT; MAPPING; NEURAL NETWORKS;

ACTIVITY LEVELS; CONVOLUTIONAL NEURAL NETWORK; DEEP LEARNING; FUSION RULE; MULTIFOCUS IMAGE FUSION;

IMAGE FUSION;

EID: 85002945898     PISSN: 15662535     EISSN: None     Source Type: Journal    
DOI: 10.1016/j.inffus.2016.12.001     Document Type: Article

References (67)

1. [1] Stathaki, T., Image Fusion: Algorithms and Applications. 2008, Academic Press.
2. [2] Burt, P., Adelson, E., The Laplacian pyramid as a compact image code. IEEE Trans. Commun. 31:4 (1983), 532–540.
3. [3] Toet, A., A morphological pyramidal image decomposition. Pattern Recognit. Lett. 9:4 (1989), 255–261.
4. [4] Li, H., Manjunath, B., Mitra, S., Multisensor image fusion using the wavelet transform. Graphical Models Image Process. 57:3 (1995), 235–245.
5. [5] Lewis, J., Oí¯Callaghan, R., Nikolov, S., Bull, D., Canagarajah, N., Pixel- and region-based image fusion with complex wavelets. Inf. Fusion 8:2 (2007), 119–130.
6. [6] Zhang, Q., Guo, B., Multifocus image fusion using the nonsubsampled contourlet transform. Signal Process. 89:7 (2009), 1334–1346.
7. [7] Piella, G., A general framework for multiresolution image fusion: from pixels to regions. Inf. Fusion 4:4 (2003), 259–280.
8. [8] Qu, X., Yan, J., Xiao, H., Zhu, Z., Image fusion algorithm based on spatial frequency-motivated pulse coupled neural networks in nonsubsampled contourlet transform domain. Acta Autom. Sin. 34:12 (2008), 1508–1514.
9. [9] Tian, J., Chen, L., Adaptive multi-focus image fusion using a wavelet-based statistical sharpness measure. Signal Process. 92:9 (2012), 2137–2146.
10. [10] Li, X., Li, H., Yu, Z., Kong, Y., Multifocus image fusion scheme based on the multiscale curvature in nonsubsampled contourlet transform domain. Opt. Eng., 54, 2015, 073115.
11. [11] Liu, Y., Liu, S., Wang, Z., A general framework for image fusion based on multi-scale transform and sparse representation. Inf. Fusion 24:1 (2015), 147–164.
12. [12] Mitianoudis, N., Stathaki, T., Pixel-based and region-based image fusion schemes using ica bases. Inf. Fusion 8:2 (2007), 131–142.
13. [13] Yang, B., Li, S., Multifocus image fusion and restoration with sparse representation. IEEE Trans. Instrum. Meas. 59:4 (2010), 884–892.
14. [14] Liang, J., He, Y., Liu, D., Zeng, X., Image fusion using higher order singular value decomposition. IEEE Trans. Image Process. 21:5 (2012), 2898–2909.
15. [15] Jiang, Y., Wang, M., Image fusion with morphological component analysis. Inf. Fusion 18:1 (2014), 107–118.
16. [16] Liu, Z., Chai, Y., Yin, H., Zhou, J., Zhu, Z., A novel multi-focus image fusion approach based on image decomposition. Inf. Fusion 35 (2017), 102–116.
17. [17] Huang, W., Jing, Z., Evaluation of focus measures in multi-focus image fusion. Pattern Recognit. Lett. 28:4 (2007), 493–500.
18. [18] Li, S., Kwok, J., Wang, Y., Combination of images with diverse focuses using the spatial frequency. Inf. Fusion 2:3 (2001), 169–176.
19. [19] Li, S., Kwok, J., Wang, Y., Multifocus image fusion using artificial neural networks. Pattern Recognit. Lett. 23:8 (2002), 985–997.
20. [20] Aslantas, V., Kurban, R., Fusion of multi-focus images using differential evolution algorithm. Expert Syst. Appl. 37:12 (2010), 8861–8870.
21. [21] De, I., Chanda, B., Multi-focus image fusion using a morphology-based focus measure in a quad-tree structure. Inf. Fusion 14:2 (2013), 136–146.
22. [22] Bai, X., Zhang, Y., Zhou, F., Xue, B., Quadtree-based multi-focus image fusion using a weighted focus-measure. Inf. Fusion 22:1 (2015), 105–118.
23. [23] Li, M., Cai, W., Tan, Z., A region-based multi-sensor image fusion scheme using pulse-coupled neural network. Pattern Recognit. Lett. 27:16 (2006), 1948–1956.
24. [24] Li, S., Yang, B., Multifocus image fusion using region segmentation and spatial frequency. Image Vis. Comput. 26:7 (2008), 971–979.
25. [25] Li, S., Kang, X., Hu, J., Image fusion with guided filtering. IEEE Trans. Image Process. 22:7 (2013), 2864–2875.
26. [26] Li, S., Kang, X., Hu, J., Y, B., Image matting for fusion of multi-focus images in dynamic scenes. Inf. Fusion 14:2 (2013), 147–162.
27. [27] Zhou, Z., Li, S., Wang, B., Multi-scale weighted gradient-based fusion for multi-focus images. Inf. Fusion 20:1 (2014), 60–72.
28. [28] Guo, D., Yan, J., Qu, X., High quality multi-focus image fusion using self-similarity and depth information. Opt. Commun. 338:1 (2015), 138–144.
29. [29] Liu, Y., Liu, S., Wang, Z., Multi-focus image fusion with dense sift. Inf. Fusion 23:1 (2015), 139–155.
30. [30] Nejati, M., Samavi, S., Shirani, S., Multi-focus image fusion using dictionary-based sparse representation. Inf. Fusion 25:1 (2015), 72–84.
31. [31] Zhang, Y., Bai, X., Wang, T., Boundary finding based multi-focus image fusion through multi-scale morphological focus-measure. Inf. Fusion 35 (2017), 81–101.
32. [32] Li, S., Kang, X., Fang, L., Hu, J., Yin, H., Pixel-level image fusion: a survey of the state of the art. Inf. Fusion 33 (2017), 100–112.
33. [33] Gao, G., Xu, L., Feng, D., Multi-focus image fusion based on non-subsampled shearlet transform. IET Image Process. 7:6 (2013), 633–639.
34. [34] Zhao, H., Shang, Z., Tang, Y., Fang, B., Multi-focus image fusion based on the neighbor distance. Pattern Recognit. 46:3 (2013), 1002–1011.
35. [35] Yang, B., Li, S., Pixel-level image fusion with simultaneous orthogonal matching pursuit. Inf. Fusion 13:1 (2012), 10–19.
36. [36] Yu, N., Qiu, T., Bi, F., Wang, A., Image features extraction and fusion based on joint sparse representation. IEEE J. Sel. Top Signal Process. 5:5 (2011), 1074–1082.
37. [37] Liu, Y., Wang, Z., Simultaneous image fusion and denosing with adaptive sparse representation. IET Image Process. 9:5 (2015), 347–357.
38. [38] LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., Gradient-based leaning applied to document recognition. Proc. IEEE 86:11 (1998), 2278–2324.
39. [39] Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., L, Y., Overfeat: integrated recognition, localizaton and detection using convolutional networks. arXiv 1312.6299v4 (2014), 1–16.
40. [40] https://en.wikipedia.org/wiki/Deep_learning.
41. [41] Nair, V., Hinton, G., Rectified linear units improve restricted boltzmann machines. Proceedings of 27th International Conference on Machine Learning, 2010, 807–814.
42. [42] Krizhevsky, A., Sutskever, I., Hinton, G., Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 2012, 1097–1105.
43. [43] Farfade, S., Saberian, M., Li, L., Multi-view face detection using deep convolutional neural networks. Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, 2015, 643–650.
44. [44] Sun, Y., Wang, X., Tang, X., Deep learning face representation from predicting 10,000 classes. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, 1891–1898.
45. [45] Long, J., Shelhamer, E., Darrell, T., Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, 3431–3440.
46. [46] Dong, C., Loy, C., He, K., Tang, X., Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. Mach. Intell. 38:2 (2016), 295–307.
47. [47] Zagoruyko, S., Komodakis, N., Learning to compare image patches via convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, 4353–4361.
48. [48] Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., Guadarrama, S., Darrell, T., Caffe: convolutional architecture for fast feature embedding. Proceedings of the ACM International Conference on Multimedia, 2014, 675–678.
49. [49] http://www.vlfeat.org/matconvnet/.
50. [50] He, K., Sun, J., Tang, X., Guided image filtering. IEEE Trans. Pattern Anal. Mach. Intell. 35:6 (2013), 1397–1409.
51. [51] http://www.image-net.org/.
52. [52] Glorot, X., Bengio, Y., Understanding the difficulty of training deep feedforward neural networks. International Conference on Artificial Intelligence and Statistics, 2010.
53. [53] http://mansournejati.ece.iut.ac.ir/content/lytro-multi-focus-dataset.
54. [54] Li, S., Yang, B., Hu, J., Performance comparison of different multi-resolution transforms for image fusion. Inf. Fusion 12:2 (2011), 74–84.
55. [55] http://home.ustc.edu.cn/~liuyu1.
56. [56] http://xudongkang.weebly.com/index.html.
57. [57] https://github.com/lsauto/MWGF-Fusion.
58. [58] Petrovic, V., Dimitrijevic, V., Focused pooling for image fusion evalution. Inf. Fusion 22:1 (2015), 119–126.
59. [59] Liu, Z., Blasch, E., Xue, Z., Zhao, J., Laganiere, R., Wu, W., Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: a comparative study. IEEE Trans. Pattern Anal. Mach. Intell. 34:1 (2012), 94–109.
60. [60] Hossny, M., Nahavandi, S., Creighton, D., Comments on information measure for performance of image fusion. Electron. Lett. 44:18 (2008), 1066–1067.
61. [61] Xydeas, C.S., Petrovic, V.S., Objective image fusion performance measure. Electron. Lett. 36:4 (2000), 308–309.
62. [62] Yang, C., Zhang, J., Wang, X., Liu, X., A novel similarity based quality metric for image fusion. Inf. Fusion 9:2 (2008), 156–160.
63. [63] Chen, Y., Blum, R., A new automated quality assessment algorithm for image fusion. Image Vis. Comput. 27:10 (2009), 1421–1432.
64. [64] Wang, Z., Bovik, A., Sheikh, H., Simoncelli, E., Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13:4 (2004), 600–612.
65. [65] Hossny, M., Nahavandi, S., Creighton, D., Bhatti, A., Image fusion performance metric based on mutual information and entropy driven quadtree decomposition. Electron. Lett. 46:18 (2010), 1266–1268.
66. [66] Zhang, W., Cham, W.-K., Gradient-directed multiexposure composition. IEEE Trans. Image Process. 21:4 (2012), 2318–2323.
67. [67] Mertens, T., Kautz, J., Reeth, F.V., Exposure fusion. Proceedings of Pacific Graphics, 2007, 382–390.