Region-Based Convolutional Networks for Accurate Object Detection and Segmentation

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 38, Issue 1, 2016, Pages 142-158

  Girshick, Ross ^a   Donahue, Jeff ^b   Darrell, Trevor ^b   Malik, Jitendra ^b  

^a MICROSOFT RESEARCH   (United States)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Convolutional Networks; Deep Learning; Detection; Object Recognition; Semantic Segmentation; Transfer Learning

Indexed keywords

COMPLEX NETWORKS; CONTENT BASED RETRIEVAL; CONVOLUTION; ERROR DETECTION; IMAGE SEGMENTATION; OBJECT RECOGNITION; SEMANTICS;

CONVOLUTIONAL NETWORKS; DEEP LEARNING; DETECTION ALGORITHM; DETECTION PERFORMANCE; LABELED TRAINING DATA; LOW-LEVEL IMAGE FEATURES; SEMANTIC SEGMENTATION; TRANSFER LEARNING;

OBJECT DETECTION;

EID: 84961595279     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2015.2437384     Document Type: Article

References (76)

1. D. Lowe, "Distinctive image features from scale-invariant keypoints," Int. J. Comput. Vis., vol. 60, no. 2, pp. 91-110, 2004.
2. N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2005, pp. 886-893.
3. M. Everingham, L. van Gool, C. K. I. Williams, J. Winn, and A. Zisserman, "The PASCAL visual object classes (VOC) challenge," Int. J. Comput. Vis., vol. 80, no. 2, pp. 303-338, 2010.
4. K. Fukushima, "Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position," Biol. Cybern., vol. 36, no. 4, pp. 193-202, 1980.
5. D. E. Rumelhart, G. E. Hinton, and R. J. Williams, "Learning internal representations by error propagation," Parallel Distrib. Process., vol. 1, pp. 318-362, 1986.
6. Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard, W. Hubbard, and L. Jackel, "Backpropagation applied to handwritten zip code recognition," Neural Comput., vol. 1, no. 4, pp. 541-551, 1989.
7. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proc. IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998.
8. A. Krizhevsky, I. Sutskever, and G. Hinton, "ImageNet classification with deep convolutional neural networks," in Proc. Adv. Neural Inf. Process. Syst., 2012, pp. 1106-1114.
9. J. Deng, A. Berg, S. Satheesh, H. Su, A. Khosla, and L. Fei-Fei. Imagenet large scale visual recognition competition 2012 (ILSVRC2012) [Online]. Available: http://www.image-net.org/ challenges/LSVRC/2012/, 2012.
10. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "ImageNet: A large-scale hierarchical image database," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2009, pp. 248-255.
11. R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich feature hierarchies for accurate object detection and semantic segmentation," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 580-587.
12. C. Szegedy, A. Toshev, and D. Erhan, "Deep neural networks for object detection," in Proc. Adv. Neural Inf. Process. Syst., 2013, pp. 2553-2561.
13. D. Erhan, C. Szegedy, A. Toshev, and D. Anguelov, "Scalable object detection using deep neural networks," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 2155-2162.
14. H. A. Rowley, S. Baluja, and T. Kanade, "Neural network-based face detection," IEEE Trans. Pattern Anal. Mach. Intell., vol. 20, no. 1, pp. 23-28, Jan. 1998.
15. R. Vaillant, C. Monrocq, and Y. LeCun, "Original approach for the localisation of objects in images," IEE Proc. Vis., Image, Signal Process., vol. 141, no. 4, pp. 245-250, Aug. 1994.
16. J. Platt and S. Nowlan, "A convolutional neural network hand tracker," in Proc. Adv. Neural Inf. Process. Syst., 1995, pp. 901-908.
17. P. Sermanet, K. Kavukcuoglu, S. Chintala, and Y. LeCun, "Pedestrian detection with unsupervised multi-stage feature learning," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 3626-3633.
18. P. Felzenszwalb, R. Girshick, D. McAllester, and D. Ramanan, "Object detection with discriminatively trained part based models," IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 9, pp. 1627-1645, Sep. 2010.
19. P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, "OverFeat: Integrated recognition, localization and detection using convolutional networks," in Proc. Int. Conf. Learn. Representations, 2014, p. 16.
20. C. Gu, J. J. Lim, P. Arbelaez, and J. Malik, "Recognition using regions," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2009, pp. 1030-1037.
21. J. Uijlings, K. van de Sande, T. Gevers, and A. Smeulders, "Selective search for object recognition," Int. J. Comput. Vis., vol. 104, no. 3, pp. 154-171, 2013.
22. J. Carreira and C. Sminchisescu, "CPMC: Automatic object segmentation using constrained parametric min-cuts," IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 7, pp. 1312-1328, Jul. 2012.
23. R. Girshick, P. Felzenszwalb, and D. McAllester. Discriminatively trained deformable part models, release 5 [Online]. Available: http://www.cs.berkeley.edu/rbg/latent-v5/, 2012.
24. K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," in Proc. Int. Conf. Learn. Representations, 2015.
25. P. Agrawal, R. Girshick, and J. Malik, "Analyzing the performance of multilayer neural networks for object recognition," in Proc. 13th Eur. Conf. Comput. Vis., 2014, pp. 329-344.
26. T. Dean, J. Yagnik, M. Ruzon, M. Segal, J. Shlens, and S. Vijayanarasimhan, "Fast, accurate detection of 100,000 object classes on a single machine," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 1814-1821.
27. K. He, X. Zhang, S. Ren, and J. Sun, "Spatial pyramid pooling in deep convolutional networks for visual recognition," in Proc. 13th Eur. Conf. Comput. Vis., 2014, pp. 346-361.
28. R. Girshick, "Fast R-CNN," arXiv e-prints, vol. arXiv:1504.08083v1.cs.CV, 2015.
29. D. Hoiem, A. Efros, and M. Hebert, "Geometric context from a single image," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2005, pp. 654-661.
30. B. C. Russell, W. T. Freeman, A. A. Efros, J. Sivic, and A. Zisserman, "Using multiple segmentations to discover objects and their extent in image collections," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2006, pp. 1605-1614.
31. C. L. Zitnick and P. Dollar, "Edge boxes: Locating object proposals from edges," in Proc. 13th Eur. Conf. Comput. Vis., 2014, pp. 391-405.
32. M.-M. Cheng, Z. Zhang, W.-Y. Lin, and P. H. S. Torr, "BING: Binarized normed gradients for objectness estimation at 300fps," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 3286-3293.
33. J. Hosang, R. Benenson, P. Dollar, and B. Schiele, "What makes for effective detection proposals?" arXiv e-prints, vol. arXiv:1502.05082v1 [cs.CV], 2015.
34. A. Humayun, F. Li, and J. M. Rehg, "RIGOR: Reusing inference in graph cuts for generating object regions," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 336-343.
35. P. Arbelaez, J. Pont-Tuset, J. Barron, F. Marques, and J. Malik, "Multiscale combinatorial grouping," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 328-335.
36. P. Krahenbuhl, and V. Koltun, "Geodesic object proposals," in Proc. 13th Eur. Conf. Comput. Vis., 2014, pp. 725-739.
37. S. J. Pan and Q. Yang, "A survey on transfer learning," IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 10, pp. 1345-1359, Oct. 2010.
38. R. Caruana, "Multitask learning: A knowledge-based source of inductive bias," in Proc. 10th Int. Conf. Mach. Learn., 1993, pp. 41-48.
39. S. Thrun, "Is learning the n-th thing any easier than learning the first?" in Proc. Adv. Neural Inf. Process. Syst., 1996, pp. 640-646.
40. Y. Bengio, A. Courville, and P. Vincent, "Representation learning: A review and new perspectives," IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 8, pp. 1798-1828, Aug. 2013.
41. J. Donahue, Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell, "DeCAF: A deep convolutional activation feature for generic visual recognition," in Proc. Int. Conf. Mach. Learn., 2014, pp. 647-655.
42. J. Hoffman, S. Guadarrama, E. Tzeng, J. Donahue, R. Girshick, T. Darrell, and K. Saenko, "From large-scale object classifiers to large-scale object detectors: An adaptation approach," in Proc. Adv. Neural Inf. Process. Syst., 2014, pp. 3536-3544.
43. A. Karpathy, A. Joulin, and L. Fei-Fei, "Deep fragment embeddings for bidirectional image sentence mapping," in Proc. Adv. Neural Inf. Process. Syst., 2014, pp. 1889-1897.
44. G. Gkioxari, B. Hariharan, R. Girshick, and J. Malik, "R-CNNs for pose estimation and action detection," arXiv e-prints, vol. arXiv:1406.5212v1 [cs.CV], 2014.
45. B. Hariharan, P. Arbelaez, R. Girshick, and J. Malik, "Simultaneous detection and segmentation," in Proc. 13th Eur. Conf. Comput. Vis., 2014, pp. 297-312.
46. S. Gupta, R. Girshick, P. Arbelaez, and J. Malik, "Learning rich features from RGB-D images for object detection and segmentation," in Proc. 13th Eur. Conf. Comput. Vis., 2014, pp. 345-360.
47. H. O. Song, R. Girshick, S. Jegelka, J. Mairal, Z. Harchaoui, and T. Darrell, "On learning to localize objects with minimal supervision," in Proc. Int. Conf. Mach. Learn., 2014, pp. 1611-1619.
48. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, "ImageNet large scale visual recognition challenge," arXiv e-prints, vol. arXiv:1409.0575v1 [cs.CV], 2014.
49. C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, "Going deeper with convolutions," arXiv e-prints, vol. arXiv:1409.4842v1 [cs.CV], 2014.
50. C. Szegedy, S. Reed, D. Erhan, and D. Anguelov, "Scalable, highquality object detection," arXiv e-prints, vol. arXiv:1412.1441v2cs.CV, 2015.
51. B. Alexe, T. Deselaers, and V. Ferrari, "Measuring the objectness of image windows," IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 11, pp. 2189-2202, Nov. 2012.
52. I. Endres and D. Hoiem, "Category independent object proposals," in Proc. 11th Eur. Conf. Comput. Vis., 2010, pp. 575-588.
53. D. Cirȩsan, A. Giusti, L. Gambardella, and J. Schmidhuber, "Mitosis detection in breast cancer histology images with deep neural networks," in Proc. Medical Image Comput. Comput.-Assisted Intervention, 2013, pp. 411-418.
54. X. Wang, M. Yang, S. Zhu, and Y. Lin, "Regionlets for generic object detection," in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 17-24.
55. Y. Jia. (2013). Caffe: An open source convolutional architecture for fast feature embedding [Online]. Available: http://caffe. berkeleyvision.org/
56. T. Dean, M. A. Ruzon, M. Segal, J. Shlens, S. Vijayanarasimhan, and J. Yagnik, "Fast, accurate detection of 100,000 object classes on a single machine," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 1814-1821.
57. S. Fidler, R. Mottaghi, A. Yuille, and R. Urtasun, "Bottom-up segmentation for top-down detection," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 3294-3301.
58. K. Sung and T. Poggio, "Example-based learning for view-based human face detection," Artif. Intell. Lab., Massachussets Inst. Technol., Cambridge, MA, USA, Tech. Rep. A.I. Memo No. 1521, 1994.
59. J. Carreira, R. Caseiro, J. Batista, and C. Sminchisescu, "Semantic segmentation with second-order pooling," in Proc. 12th Eur. Conf. Comput. Vis., 2012, pp. 430-443.
60. K. E. van de Sande, C. G. Snoek, and A. W. Smeulders, "Fisher and vlad with flair," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2014, pp. 2377-2384.
61. J. J. Lim, C. L. Zitnick, and P. Dollar, "Sketch tokens: A learned mid-level representation for contour and object detection," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 3158-3165.
62. X. Ren and D. Ramanan, "Histograms of sparse codes for object detection," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2013, pp. 3246-3253.
63. M. Zeiler, G. Taylor, and R. Fergus, "Adaptive deconvolutional networks for mid and high level feature learning," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2011, pp. 2018-2025.
64. D. Hoiem, Y. Chodpathumwan, and Q. Dai, "Diagnosing error in object detectors," in Proc. 12th Eur. Conf. Comput. Vis., 2012, pp. 340-353.
65. J. Deng, O. Russakovsky, J. Krause, M. Bernstein, A. C. Berg, and L. Fei-Fei, "Scalable multi-label annotation," in Proc. SIGCHI Conf. Human Factors Comput. Syst., 2014, pp. 3099-3102.
66. H. Su, J. Deng, and L. Fei-Fei, "Crowdsourcing annotations for visual object detection," in Proc. AAAI 4th Human Comput. Workshop, 2012.
67. C. Farabet, C. Couprie, L. Najman, and Y. LeCun, "Learning hierarchical features for scene labeling," IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 8, pp. 1915-1929, Aug. 2013.
68. P. Arbelaez, B. Hariharan, C. Gu, S. Gupta, L. Bourdev, and J. Malik, "Semantic segmentation using regions and parts," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2012, pp. 3378-3385.
69. B. Hariharan, P. Arbelaez, L. Bourdev, S. Maji, and J. Malik, "Semantic contours from inverse detectors," in Proc. IEEE Int. Conf. Comput. Vis., 2011, pp. 991-998.
70. B. Hariharan, P. Arbelaez, R. Girshick, and J. Malik, "Hypercolumns for object segmentation and fine-grained localization," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 447-456.
71. L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, "Semantic image segmentation with deep convolutional nets and fully connected CRFs," in Proc. Int. Conf. Learn. Representations, 2015.
72. S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du, C. Huang, and P. Torr, "Conditional random fields as recurrent neural networks," arXiv e-print, vol. arXiv:1502.03240v2 [cs. CV], 2015.
73. J. Long, E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation," in Proc. IEEE Conf. Comput. Vis. Pattern Recog., 2015, pp. 3431-3440.
74. P. Krahenbühl and V. Koltun, "Efficient inference in fully connected CRFs with Gaussian edge potentials," in Proc. Adv. Neural Inf. Process. Syst., 2011, pp. 109-117.
75. A. Oliva and A. Torralba, "Modeling the shape of the scene: A holistic representation of the spatial envelope," Int. J. Comput. Vis., vol. 42, pp. 145-175, 2001.
76. M. Douze, H. Jegou, H. Sandhawalia, L. Amsaleg, and C. Schmid, "Evaluation of gist descriptors for web-scale image search," in Proc. ACM Int. Conf. Image Video Retrieval, 2009, p. 19.