A New Deep Learning-Based Food Recognition System for Dietary Assessment on An Edge Computing Service Infrastructure

IEEE Transactions on Services Computing

Volumn 11, Issue 2, 2018, Pages 249-261

  Liu, Chang ^a   Cao, Yu ^a   Luo, Yan ^a,b   Chen, Guanling ^a   Vokkarane, Vinod ^b   Yunsheng, Ma ^c   Chen, Songqing ^d   Hou, Peng ^a  

^a UNIVERSITY OF MASSACHUSETTS LOWELL   (United States)

^b University of Massachusetts   (United States)

^c UNIVERSITY OF MASSACHUSETTS MEDICAL SCHOOL   (United States)

^d Krasnow Institute for Advanced Study   (United States)

Author keywords

deep learning; edge computing; food recognition; Mobile applications; object recognition

Indexed keywords

DEEP LEARNING; EDGE COMPUTING; ENERGY UTILIZATION; HUMAN COMPUTER INTERACTION; OBJECT RECOGNITION;

ALGORITHM DEVELOPMENT; COMPUTING SERVICES; DIETARY ASSESSMENTS; FOOD RECOGNITION; MOBILE APPLICATIONS; RECOGNITION ACCURACY; SERVICE COMPUTING; STATE OF THE ART;

MOBILE CLOUD COMPUTING;

EID: 85042517103     PISSN: 19391374     EISSN: None     Source Type: Journal    
DOI: 10.1109/TSC.2017.2662008     Document Type: Article

References (63)

1. C. L. Ogden, M. D. Carroll, B. K. Kit, and K. M. Flegal, "Prevalence of childhood and adult obesity in the United States, 2011-2012, " Jama, vol. 311, pp. 806-814, 2014.
2. World Health Organization: Fact sheets on obesity and overweight, 2016. [Online]. Available: http://www.who.int/ mediacentre/factsheets/fs311/en/.
3. G. H. Beaton, et al., "Sources of variance in 24-hour dietary recall data: Implications for nutrition study design and interpretation, " Amer. J. Clinical Nutrition, vol. 32, pp. 986-995, 1979.
4. J. Cade, R. Thompson, V. Burley, and D. Warm, "Development, validation and utilisation of food-frequency questionnaires-a review, " Public Health Nutrition, vol. 5, pp. 567-587, 2002.
5. R. Steele, "An overview of the state of the art of automated capture of dietary intake information, " Critical Rev. Food Sci. Nutrition, vol. 55, pp. 1929-1938, 2013.
6. Y. Matsuda and K. Yanai, "Multiple-food recognition considering co-occurrence employing manifold ranking, " in Proc. 21st Int. Conf. Pattern Recognit., 2012, pp. 2017-2020.
7. S. Yang, M. Chen, D. Pomerleau, and R. Sukthankar, "Food recognition using statistics of pairwise local features, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2010, pp. 2249-2256.
8. F. Zhu, et al., "The use of mobile devices in aiding dietary assessment and evaluation, " IEEE J. Selected Topics Signal Process., vol. 4, no. 4, pp. 756-766, Aug. 2010.
9. B. L. Daugherty, et al., "Novel technologies for assessing dietary intake: Evaluating the usability of a mobile telephone food record among adults and adolescents, " J. Med. Internet Res., vol. 14, 2012, Art. no. e58.
10. C. Xu, Y. He, N. Khannan, A. Parra, C. Boushey, and E. Delp, "Image-based food volume estimation, " in Proc. 5th Int. Workshop Multimedia Cooking Eating Activities, 2013, pp. 75-80.
11. TADA: Technology Assisted Dietary Assessment at Purdue University, West Lafayette, Indiana, USA, (2010). [Online]. Available: http://www.tadaproject.org/
12. MyFitnessPal.com: Free Calorie Counter, Diet & Exercise Tracker, (2005). [Online]. Available: http://www.myfitnesspal.com/
13. MyNetDiary: The easiest and smartest free calorie counter and free food diary for iPhone, iPad, Android, and BlackBerry applications, (2008). [Online]. Available:, http://www.mynetdiary.com/
14. FatSecret: All Things Food and Diet, (2007). [Online]. Available:, http://www.fatsecret.com/
15. C. K. Martin, T. Nicklas, B. Gunturk, J. B. Correa, H. R. Allen, and C. Champagne, "Measuring food intake with digital photography, " J. Hum. Nutr. Diet., vol. 27 no. Suppl 1, pp. 72-81, Jan. 2014.
16. Y. Bengio, "Learning deep architectures for AI, " Found. Trends- Mach. Learning, vol. 2, pp. 1-127, 2009.
17. G. E.Hinton, S. Osindero, and Y.-W. Teh, "A fast learning algorithm for deep belief nets, " Neural Comput., vol. 18, pp. 1527-1554, 2006.
18. Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning, " Nature, vol. 521, pp. 436-444, 2015.
19. D. G. Lowe, "Distinctive image features from scale-invariant keypoints, " Int. J. Comput. Vis., vol. 60, pp. 91-110, Nov. 2004.
20. M. Satyanarayanan, P. Bahl, R. Caceres, and N. Davies, "The case for VM-based cloudlets in mobile computing, " IEEE Pervasive Comput., vol. 8, no. 4, pp. 14-23, Oct.-Dec. 2009.
21. S. Yi, Z. Hao, Z. Qin, and Q. Li, "Fog computing: Platform and applications, " in Proc. 3rd IEEE Workshop Hot Topics Web Syst. Technol., 2015, pp. 73-78.
22. W. Shi, J. Cao, Q. Zhang, Y. Li, and L. Xu, "Edge computing: vision and challenges, " IEEE Internet Things J., vol. 3, no. 6, pp. 637-646, Oct. 2016.
23. M. Satyanarayanan, Z. Chen, K.Ha, W.Hu, W. Richter, and P. Pillai, "Cloudlets: At the leading edge of mobile-cloud convergence, " in Proc. 6th Int. Conf.Mobile Comput. Appl. Services, 2014, pp. 1-9.
24. W. Shi and S. Dustdar, "The promise of edge computing, " IEEE Comput. Mag., vol. 29, no. 5, pp. 78-81, May 2016.
25. S. Yi, C. Li, and Q. Li, "A survey of fog computing: Concepts, applications and issues, " in Proc. Workshop Mobile Big Data, 2015, pp. 37-42.
26. P. Garcia Lopez, et al., "Edge-centric computing: Vision and challenges, " ACM SIGCOMM Comput. Commun. Rev., vol. 45, pp. 37-42, 2015.
27. R. R. Wing and S. Phelan, "Long-term weight loss maintenance, " Amer. J. Clinical Nutrition, vol. 82, pp. 222S-225S, 2005.
28. Meal Snap: Magical Meal Logging for iPhone. (2011). [Online]. Available:, http://mealsnap.com/
29. Eatly: Eat Smart (Snap a photo of your meal and get health ratings). (2014). [Online]. Available: https://itunes.apple.com/us/ app/eatly-eat-smart-snap-photo/id661113749
30. N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection, " in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2005, pp. 886-893.
31. A. Meyers, N. Johnston, V. Rathod, A. Korattikara, A. Gorban, et al., "Im2Calories: Towards an automated mobile vision food diary, " in Proc. IEEE Int. Conf. Comput. Vis. Pattern Recognit., 2015, pp. 1233-1241.
32. I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning, Cambridge, MA, USA: MIT Press, 2016.
33. A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks, " in Proc. Adv. Neural Inf. Process. Syst., 2012, Art. no. 4.
34. D. H. Hubel and T. N. Wiesel, "Receptive fields, binocular interaction and functional architecture in the cat's visual cortex, " J. Physiology, vol. 160, pp. 106-154, 1962.
35. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition, " Proc. IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998.
36. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2009, pp. 248-255.
37. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: A simple way to prevent neural networks fromoverfitting, " J.Mach. Learning Res., vol. 15, pp. 1929-1958, 2014.
38. K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition, " In Proc. IEEE Int. Conf. Learn. Represent., 2015. pp. 33-41.
39. M. D. Zeiler and R. Fergus, "Visualizing and understanding convolutional networks, " in Computer Vision-ECCV, Berlin, Germany: Springer, 2014, pp. 818-833.
40. C. Szegedy, et al., "Going deeper with convolutions, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 1-9.
41. K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 770-778.
42. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition, " Proc. IEEE, vol. 86, no. 11, pp. 2278-2324, Nov. 1998.
43. V. Stantchev, A. Barnawi, S. Ghulam, J. Schubert, and G. Tamm, "Smart Items, fog and cloud computing as enablers of servitization in healthcare, " Sensors Transducers, vol. 185, 2015, Art. no. 121.
44. Y. Cao, S. Chen, P. Hou, and D. Buhl-Brown, "FAST: A fog computing assisted distributed analytics system to monitor fall for stroke mitigation, " presented at Prof. 10th IEEE Int. Conf. Netw. Archit. Storage (Best Paper Award), Boston, MA, USA, 2015.
45. Y. Cao, et al., "Improving tuberculosis diagnostics using deep learning and mobile health technologies among resource-poor and marginalized communities, " in Proc. IEEE Conf. Connected Health: Appl. Syst. Eng. Technol., 2016, pp. 274-281.
46. Y. Jia, et al., "Caffe: Convolutional architecture for fast feature embedding, " in Proc. ACM Int. Conf. Multimedia, 2014, pp. 675- 678.
47. J. Oh, et al., "Measuring objective quality of colonoscopy, " IEEE Trans. Biomed. Eng., vol. 56, no. 9, pp. 2190-2196, Sep. 2009.
48. R. C. Gonzalez and R. E. Woods, Digital Image Processing. Upper Saddle River, NJ, USA: Prentice Hall, 2008.
49. M. Lin, Q. Chen, and S. Yan, "Network in network, " in Proc. Int. Conf. Learn. Represent., 2014, pp. 1-9.
50. K. He and J. Sun, "Convolutional neural networks at constrained time cost, " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 5353-5360.
51. A. Holovaty and J. Kaplan-Moss, The Definitive Guide to Django: Web Development Done Right. New York, NY, USA: Apress, 2009.
52. G. Bradski and A. Kaehler, Learning OpenCV: Computer Vision with the OpenCV Library. Sebastopol, CA, USA: O'Reilly Media, Inc., 2008.
53. Django: The Web framework for perfectionists with deadlines, (2005). [Online]. Available: https://www.djangoproject.com/
54. H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, "Speeded-up robust features (SURF), " Comput. Vis. Image Understanding, vol. 110, pp. 346-359, 2008.
55. N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection, " in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2005, pp. 886-893.
56. P. Viola and M. Jones, "Rapid object detection using a boosted cascade of simple features, " in Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., 2001, pp. 511-518.
57. Y. Kawano and K. Yanai, "Foodcam: A real-time food recognition system on a smartphone, " Multimedia Tools Appl., vol. 74, pp. 1-25, 2015.
58. L. Bossard, M. Guillaumin, and L. Van Gool, "Food-101-mining discriminative components with random forests, " in Computer Vision-ECCV, Berlin, Germany: Springer, 2014, pp. 446-461.
59. Y. Kawano and K. Yanai, "Food image recognition with deep convolutional features, " in Proc. ACM Int. Joint Conf. Pervasive Ubiquitous Comput.: Adjunct Publication, 2014, pp. 589-593.
60. C. Liu, Y. Cao, Y. Luo, G. Chen, V. Vokkarane, and Y. Ma, "DeepFood: Deep learning-based food image recognition for computer- aided dietary assessment, " presented at Prof. 14th Int. Conf. Smart homes Health Telematics, Wuhan, China, 2016.
61. T.-Y. Lin, et al., "Microsoft coco: Common objects in context, " in Computer Vision-ECCV, Berlin, Germany: Springer, 2014, pp. 740- 755.
62. J. Wu, C. Leng, Y. Wang, Q. Hu, and J. Cheng, "Quantized Convolutional Neural Networks for Mobile Devices, " In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 4820-4828.
63. T. Chen, M. Li, et al., "MXNet: A flexible and efficient machine learning library for heterogeneous distributed systems, " In Proc. Workshop Mach. Learning Syst. Neural Information Processing Syst., 2016.