Image Segmentation for Fruit Detection and Yield Estimation in Apple Orchards

Journal of Field Robotics

Volumn 34, Issue 6, 2017, Pages 1039-1060

  Bargoti, Suchet ^a   Underwood, James P ^a  

^a UNIVERSITY OF SYDNEY   (Australia)

Author keywords

[No Author keywords available]

Indexed keywords

FARMS; FRUITS; GRADING; GROUND VEHICLES; HOUGH TRANSFORMS; IMAGE PROCESSING; METADATA; NEURAL NETWORKS; ORCHARDS; PIXELS;

CIRCULAR HOUGH TRANSFORMS; CONTEXTUAL INFORMATION; CONVOLUTIONAL NEURAL NETWORK; MONOCULAR VISION SYSTEM; MULTILAYERED PERCEPTRONS; SEGMENTATION PERFORMANCE; SQUARED CORRELATION COEFFICIENTS; WATERSHED SEGMENTATION;

IMAGE SEGMENTATION;

EID: 85013149597     PISSN: 15564959     EISSN: 15564967     Source Type: Journal    
DOI: 10.1002/rob.21699     Document Type: Article

References (52)

1. Aggelopoulou, A. D., Bochtis, D., Fountas, S., Swain, K. C., Gemtos, T. A., & Nanos, G. D. (2011). Yield prediction in apple orchards based on image processing. Precision Agriculture, 12(3), 448–456.
2. Annamalai, P., Lee, W. S., & Burks, T. F. (2004). Color vision system for estimating citrus yield in real-time. In ASAE Annual International Meeting, St. Joseph, Michigan.
3. Atherton, T. J., & Kerbyson, D. J. (1999). Size invariant circle detection. Image and Vision Computing, 17(11), 795–803.
4. Bargoti, S., & Underwood, J. (2015). Utilising metadata to aid image classification in orchards. In IEEE International Conference on Intelligent Robots and Systems (IROS), Workshop on Alternative Sensing for Robot Perception (WASRoP), IROS 2015: Hamburg Germany.
5. Bargoti, S., & Underwood, J. (2016). Image classification with orchard metadata, In IEEE International Conference on Robotics and Automation (ICRA), Stockholm, 2016, pp. 5164–5170. Curran Associates.
6. Bargoti, S., Underwood, J. P., Nieto, J. I., & Sukkarieh, S. (2015). A pipeline for trunk detection in trellis structured apple orchards. Journal of Field Robotics, 32(8), 1075–1094.
7. Brown, M., & Susstrunk, S. (2011). Multi-spectral SIFT for scene category recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 177–184). Providence, USA: Curran Associates.
8. Brust, C. A., Sickert, S., Simon, M., Rodner, E., & Denzler, J. (2015). Convolutional patch networks with spatial prior for road detection and urban scene understanding. In International Conference on Computer Vision Theory and Applications (VISAPP) (pp. 510–517). Berlin, Germany: SCITEPRESS.
9. Chhabra, M., Gupta, A., Mehrotra, P., & Reel, S. (2012). Automated detection of fully and partially riped mango by machine vision. In Proceedings of the International Conference on Soft Computing for Problem Solving (SocProS): Vol. 131. Advances in intelligent and soft computing (pp. 153–164). Roorkee, India: Springer.
10. Ciresan, D., Giusti, A., Gambardella, L. M., & Schmidhuber, J. (2012). Deep neural networks segment neuronal membranes in electron microscopy images. In Advances in neural information processing systems 25 (pp. 2843–2851). Lake Tahoe, USA: Harrahs and Harveys.
11. Erhan, D., Bengio, Y., Courville, A., Manzagol, P.-A., Vincent, P., & Bengio, S. (2010). Why does unsupervised pre-training help deep learning? Journal of Machine Learning Research, 11, 625–660.
12. Farabet, C., Couprie, C., Najman, L., & LeCun, Y. (2013). Learning hierarchical features for scene labeling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8), 1915–1929.
13. Font, D., Tresanchez, M., Martínez, D., Moreno, J., Clotet, E., & Palacín, J. (2015). Vineyard yield estimation based on the analysis of high resolution images obtained with artificial illumination at night. Sensors, 15(4), 8284–8301.
14. 4-fields: Neural network nearest neighbor fields for image transforms. CoRR, abs/1406.6.
15. Gemtos, T., Fountas, S., Tagarakis, A., & Liakos, V. (2013). Precision agriculture application in fruit crops: Experience in handpicked fruits. Procedia Technology, 8, 324–332.
16. Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2016). Region-based convolutional networks for accurate object detection and segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(1), 142–158.
17. Goodfellow, I., & Warde-Farley, D. (2013). Pylearn2: A machine learning research library. arXiv preprint arXiv:1308.4214 (pages 1–9).
18. Hung, C., Nieto, J., Taylor, Z., Underwood, J., & Sukkarieh, S. (2013). Orchard fruit segmentation using multi-spectral feature learning. In IEEE International Conference on Intelligent Robots and Systems (IROS) (pp. 5314–5320), Toko, Japan: Curran Associates.
19. Hung, C., Underwood, J., Nieto, J., & Sukkarieh, S. (2015). A feature learning based approach for automated fruit yield estimation. In Field and Service Robotics (FSR) (pp. 485–498). Brisbane, Australia: Springer.
20. Ji, W., Zhao, D., Cheng, F., Xu, B., Zhang, Y., & Wang, J. (2012). Automatic recognition vision system guided for apple harvesting robot. Computers & Electrical Engineering, 38(5), 1186–1195.
21. Jimenez, A. R., Ceres, R., & Pons, J. L. (2000). A survey of computer vision methods for locating fruit on trees. Transactions of the ASAE-American Society of Agricultural Engineers, 43(6), 1911–1920.
22. Kapach, K., Barnea, E., Mairon, R., Edan, Y., & Ben-Shahar, O. (2012). Computer vision for fruit harvesting robots–state of the art and challenges ahead. International Journal of Computational Vision and Robotics, 3(1-2), 4–34.
23. Kim, D., Choi, H., Choi, J., Yoo, S. J., & Han, D. (2015). A novel red apple detection algorithm based on AdaBoost learning. IEIE Transactions on Smart Processing & Computing, 4(4), 265–271.
24. Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems (pp. 1097–1105). Lake Tahoe, USA: Curran Associates.
25. Kurtulmus, F., Lee, W., & Vardar, A. (2014). Immature peach detection in colour images acquired in natural illumination conditions using statistical classifiers and neural network. Precision Agriculture, 15(1), 57–79.
26. Li, P., Lee, S.-H., & Hsu, H.-Y. (2011). Study on citrus fruit image data separability by segmentation methods. Procedia Engineering, 23, 408–416.
27. Linker, R., Cohen, O., & Naor, A. (2012). Determination of the number of green apples in RGB images recorded in orchards. Computers and Electronics in Agriculture, 81, 45–57.
28. Liu, S., Whitty, M., & Cossell, S. (2015). Automatic grape bunch detection in vineyards for precise yield estimation. 14th IAPR International Conference on Machine Vision Applications (MVA), Tokyo, 2015, pp. 238-241, Curran Associates.
29. Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 3431–3440). Boston, USA: Curran Associates.
30. Martens, J. (2010). Deep learning via Hessian-free optimization. In Proceedings of the 27th International Conference on Machine Learning (ICML) (pp. 735–742). Haifa, Israel: Omnipress.
31. Moonrinta, J., Chaivivatrakul, S., Dailey, M. N., & Ekpanyapong, M. (2010). Fruit detection, tracking, and 3D reconstruction for crop mapping and yield estimation. In International Conference on Control Automation Robotics & Vision (ICARCV) (pp. 1181–1186). Singapore: Curran Associates.
32. Ning, F., Delhomme, D., LeCun, Y., Piano, F., Bottou, L., & Barbano, P. E. (2005). Toward automatic phenotyping of developing embryos from videos. IEEE Transactions on Image Processing, 14(9), 1360–1371.
33. Nuske, S., Wilshusen, K., Achar, S., Yoder, L., & Singh, S. (2014). Automated visual yield estimation in vineyards. Journal of Field Robotics, 31(5), 837–860.
34. Payne, A., & Walsh, K. (2014). Machine vision in estimation of fruit crop yield. In Plant image analysis: fundamentals and applications (pp. 329–374). Boca Raton, USA: CRC Press.
35. Payne, A., Walsh, K., Subedi, P., & Jarvis, D. (2014). Estimating mango crop yield using image analysis using fruit at “stone hardening” stage and night time imaging. Computers and Electronics in Agriculture, 100, 160–167.
36. Pinheiro, P. H. O., & Collobert, R. (2013). Recurrent convolutional neural networks for scene parsing. CoRR, abs/1306.2.
37. Qiang, L., Jianrong, C., Bin, L., Lie, D., & Yajing, Z. (2014). Identification of fruit and branch in natural scenes for citrus harvesting robot using machine vision and support vector machine. International Journal of Agricultural and Biological Engineering, 7(2), 115–121.
38. Rao, D., De Deuge, M., Nourani-Vatani, N., Douillard, B., Williams, S. B., & Pizarro, O. (2014). Multimodal learning for autonomous underwater vehicles from visual and bathymetric data. In IEEE International Conference on Robotics and Automation (ICRA) (pp. 3819–3825). Hong Kong: Curran Associates.
39. Regunathan, M. & Lee, W. S. (2005). Citrus fruit identification and size determination using machine vision and ultrasonic sensors. In 2005 ASAE Annual Meeting (p. 1) at St. Joseph, Michigan. American Society of Agricultural and Biological Engineers.
40. Ren, S., He, K., Girshick, R. B., & Sun, J. (2015). Faster {R-CNN:} towards real-time object detection with region proposal networks. CoRR, abs/1506.0.
41. Roerdink, J. B. T. M., & Meijster, A. (2000). The watershed transform: Definitions, algorithms and parallelization strategies. Fundamenta Informaticae, 41(1-2), 187–228.
42. Sa, I., McCool, C., Lehnert, C., & Perez, T. (2015). On visual detection of highly-occluded objects for harvesting automation in horticulture. In IEEE International Conference on Robotics and Automation (ICRA). ICRA: Seattle, USA; Curran Associates.
43. Regunathan, M. & Lee, W. S. (2005). Citrus fruit identification and size determination using machine vision and ultrasonic sensors. In 2005 ASAE Annual Meeting (p. 1) at St. Joseph, Michigan. American Society of Agricultural and Biological Engineers.
44. Sengupta, S., & Lee, W. S. (2014). Identification and determination of the number of immature green citrus fruit in a canopy under different ambient light conditions. Biosystems Engineering, 11, 51–61.
45. Sermanet, P., Eigen, D., Zhang, X., Mathieu, M., Fergus, R., & LeCun, Y. (2013). Overfeat: Integrated recognition, localization and detection using convolutional networks. arXiv preprint arXiv:1312.6229.
46. Silwal, A., Gongal, A., & Karkee, M. (2014). Identification of red apples in field environment with over the row machine vision system. Agricultural Engineering International: CIGR Journal, 16(4), 66–75.
47. Song, Y., Glasbey, C. A., Horgan, G. W., Polder, G., Dieleman, J. A., & van der Heijden, G. W. A. M. (2014). Automatic fruit recognition and counting from multiple images. Biosystems Engineering, 118(1), 203–215.
48. Stajnko, D., Rakun, J., & Blanke, M. M. (2009). Modelling apple fruit yield using image analysis for fruit colour, shape and texture. European Journal of Horticultural Science, 74(6), 260–267.
49. Tighe, J., & Lazebnik, S. (2013). Superparsing. International Journal of Computer Vision, 101(2), 329–349.
50. Wang, Q., Nuske, S., Bergerman, M., & Singh, S. (2013). Automated Crop yield estimation for apple orchards. In Experimental Robotics: Vol. 88. Springer Tracts in Advanced Robotics (pp. 745–758). Quebec City, Canada: Springer International Publishing.
51. Wijethunga, P., Samarasinghe, S., Kulasiri, D., & Woodhead, I. (2009). Towards a generalized colour image segmentation for kiwifruit detection. In International Conference Image and Vision Computing New Zealand (IVCNZ) (pp. 62–66). Wellington, NZ: Curran Associates.
52. Yamamoto, K., Guo, W., Yoshioka, Y., & Ninomiya, S. (2014). On plant detection of intact tomato fruits using image analysis and machine learning methods. Sensors, 14(7), 12191–12206.