A Novel Performance Evaluation Methodology for Single-Target Trackers

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 38, Issue 11, 2016, Pages 2137-2155

  Kristan, Matej ^a   Matas, Jiri ^b   Leonardis, Ales ^c   Vojir, Tomas ^b   Pflugfelder, Roman ^d   Fernandez, Gustavo ^d   Nebehay, Georg ^d   Porikli, Fatih ^e   Cehovin, Luka ^a  

^a UNIVERSITY OF LJUBLJANA   (Slovenia)

^b CZECH TECHNICAL UNIVERSITY IN PRAGUE   (Czech Republic)

^c UNIVERSITY OF BIRMINGHAM   (United Kingdom)

^d AIT AUSTRIAN INSTITUTE OF TECHNOLOGY   (Austria)

^e AUSTRALIAN NATIONAL UNIVERSITY   (Australia)

Author keywords

model free tracking; Performance analysis; single target tracking; tracker evaluation datasets; tracker evaluation methodology; tracker evaluation system

Indexed keywords

ARTIFICIAL INTELLIGENCE; COMPUTER VISION;

EVALUATION METHODOLOGIES; MODEL FREE; PERFORMANCE ANALYSIS; SINGLE TARGET TRACKING; TRACKER EVALUATION DATASETS; TRACKER EVALUATION SYSTEM;

TARGET TRACKING;

EID: 84992034892     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2016.2516982     Document Type: Article

References (81)

1. D. M. Gavrila, "The visual analysis of human movement: A survey, " Comput. Vis. Image Understanding, vol. 73, no. 1, pp. 82-98, 1999.
2. T. B. Moeslund and E. Granum, "A survey of computer visionbased human motion capture, " Comput. Vis. Image Understanding, vol. 81, no. 3, pp. 231-268, Mar. 2001.
3. P. Gabriel, J. Verly, J. Piater, A. Genon, "The state of the art in multiple object tracking under occlusion in video sequences, " in Proc. Int. Conf. Adv. Concepts Intell. Vision Syst., 2003, pp. 166-173.
4. W. Hu, T. Tan, L. Wang, S. Maybank, "A survey on visual surveillance of object motion and behaviors, " IEEE Trans. Syst., Man Cybern. C, Appl. Rev., vol. 34, no. 30, pp. 334-352, Aug. 2004.
5. T. B. Moeslund, A. Hilton, V. Kruger, "A survey of advances in vision-based human motion capture and analysis, " Comput. Vis. Image Understanding, vol. 103, nos. 2/3, pp. 90-126, Nov. 2006.
6. A. Yilmaz and M. Shah, "Object tracking: A survey, " J. ACM Comput. Surveys, vol. 38, no. 4, p. 13, 2006.
7. X. Li, W. Hu, C. Shen, Z. Zhang, A. R. Dick, A. Van den Hengel, "A survey of appearance models in visual object tracking, " arXiv:1303. 4803 [cs. CV], 2013.
8. D. P. Young and J. M. Ferryman, "PETS Metrics: On-line performance evaluation service, " in Proc. 14th Int. Conf. Comput. Commun. Netw., 2005, pp. 317-324.
9. N. Goyette, P. M. Jodoin, F. Porikli, J. Konrad, P. Ishwar, "Changedetection. net: A new change detection benchmark dataset, " in Proc. IEEE Comput. Soc. Conf. Comput. Vision Pattern Recog. Workshops, 2012, pp. 1-8.
10. P. J. Phillips, H. Moon, S. A. Rizvi, P. J. Rauss, "The FERET evaluation methodology for face-recognition algorithms, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 10, pp. 1090-1104, Oct. 2000.
11. R. Kasturi, D. B. Goldgof, P. Soundararajan, V. Manohar, J. S. Garofolo, R. Bowers, M. Boonstra, V. N. Korzhova, J. Zhang, "Framework for performance evaluation of face, text, vehicle detection and tracking in video: Data, metrics, protocol, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 2, pp. 319-336, Feb. 2009.
12. A. Geiger, P. Lenz, R. Urtasun, "Are we ready for autonomous driving The kitti vision benchmark suite, " in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2012, pp. 3354-3361.
13. K. Bernardin and R. Stiefelhagen, "Evaluating multiple object tracking performance: The CLEAR MOT metrics, " EURASIP J. Image Video Process. (Special Issue on Video Tracking in Complex Scenes for Surveillance Applications), vol. 2008, p. 1, 2008.
14. B. Karasulu and S. Korukoglu, "A software for performance evaluation and comparison of people detection and tracking methods in video processing, " Multimedia Tools Appl., vol. 55, no. 3, pp. 677-723, 2011.
15. S. Salti, A. Cavallaro, L. Di Stefano, "Adaptive appearance modeling for video tracking: Survey and evaluation, " IEEE Trans. Image Process., vol. 21, no. 10, pp. 4334-4348, Oct. 2012.
16. Y. Wu, J. Lim, M. H. Yang, "Online object tracking: A benchmark, " in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2013, pp. 2411-2418.
17. A. W. M. Smeulders, D. M. Chu, R. Cucchiara, S. Calderara, A. Dehghan, M. Shah, "Visual tracking: An experimental survey, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 7, pp. 1442-1468, Jul. 2014.
18. Y. Pang and H. Ling, "Finding the best from the second bests-Inhibiting subjective bias in evaluation of visual tracking algorithms, " in Proc. IEEE Int. Conf. Comput. Vision, 2013, pp. 2784-2791.
19. M. Kristan and L.-Cehovin, Visual Object Tracking Challenge (VOT2013) Evaluation Kit, Visual Object Tracking Challenge, 2013.
20. M. Kristan, R. Pflugfelder, A. Leonardis, J. Matas, F. Porikli, L. Cehovin, G. Nebehay, G. Fernandez, T. Vojir, "The VOT2013 challenge: Overview and additional results, " in Proc. Comput. Vision Winter Workshop, 2014, pp. 61-68.
21. L. Leal-Taix-e, A. Milan, I. Reid, S. Roth, K. Schindler. (2015, Apr. ). MOTChallenge 2015: Towards a benchmark for multi-target tracking, " arXiv:1504. 01942 [cs], arXiv: 1504. 01942. [Online]. Available: http://arxiv. org/abs/1504. 01942
22. F. Fleuret, J. Berclaz, R. Lengagne, P. Fua, "Multi-camera people tracking with a probabilistic occupancy map, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 30, no. 2, pp. 267-282, Feb. 2008.
23. M. Kristan, R. Pflugfelder, A. Leonardis, J. Matas, F. Porikli, L.-Cehovin, G. Nebehay, G. Fernandez, T. Vojir, A. Gatt, A. Khajenezhad, A. Salahledin, A. Soltani-Farani, A. Zarezade, A. Petrosino, A. Milton, B. Bozorgtabar, B. Li, C. S. Chan, C. Heng, D. Ward, D. Kearney, D. Monekosso, H. C. Karaimer, H. R. Rabiee, J. Zhu, J. Gao, J. Xiao, J. Zhang, J. Xing, K. Huang, K. Lebeda, L. Cao, M. E. Maresca, M. K. Lim, M. E. Helw, M. Felsberg, P. Remagnino, R. Bowden, R. Goecke, R. Stolkin, S. Y. Lim, S. Maher, S. Poullot, S. Wong, S. Satoh, W. Chen, W. Hu, X. Zhang, Y. Li, Z. Niu, "The Visual Object Tracking VOT2013 challenge results, " in Proc. IEEE Int. Conf. Comput. Vision Workshops, 2013, pp. 98-111.
24. M. Kristan, R. Pflugfelder, A. Leonardis, J. Matas, L.-Cehovin, G. Nebehay, T. Vojir, G. Fernandez, A. Lukezic, A. Dimitriev, A. Petrosino, A. Saffari, B. Li, B. Han, C. Heng, C. Garcia, D. Panger-si-c, G. Hager, F. Shahbaz Khan, F. Oven, H. Possegger, H. Bischof, H. Nam, J. Zhu, J. Li, J. Y. Choi, J.-W. Choi, J. a. Henriques, J. Van de Weijer, J. Batista, K. Lebeda, K. Ofjall, K. M. Yi, L. Qin, L. Wen, M. E. Maresca, M. Danelljan, M. Felsberg, M.-M. Cheng, P. Torr, Q. Huang, R. Bowden, S. Hare, S. YueYing Lim, S. Hong, S. Liao, S. Hadfield, S. Li, S. Duffner, S. Golodetz, T. Mauthner, V. Vineet, W. Lin, Y. Li, Y. Qi, Z. Lei, Z. Niu, "The Visual Object Tracking VOT2014 challenge results, " in Proc. ECCV2014 Workshops, Workshop on Visual Object Tracking Challenge, 2014, pp. 98-111.
25. D. A. Ross, J. Lim, R. S. Lin, M. H. Yang, "Incremental learning for robust visual tracking, " Int. J. Comput. Vision, vol. 77, nos. 1-3, pp. 125-141, 2008.
26. H. Li, C. Shen, Q. Shi, "Real-time visual tracking using compressive sensing, " in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2011, pp. 1305-1312.
27. J. Kwon and K. M. Lee, "Tracking of a non-rigid object via patchbased dynamic appearance modeling and adaptive basin hopping Monte Carlo sampling. " in Proc. IEEE Int. Conf. Comput. Vision Pattern Recog., 2009, pp. 1208-1215.
28. M. Kristan, J. Per-s, M. Per-se, S. Kova-ci-c, "Closed-world tracking of multiple interacting targets for indoor-sports applications, " Comput. Vision Image Understanding, vol. 113, no. 5, pp. 598-611, May 2009.
29. M. Kristan, S. Kovacic, A. Leonardis, J. Per-s, "A two-stage dynamic model for visual tracking, " IEEE Trans. Syst., Man, Cybern. B, Cybern., vol. 40, no. 6, pp. 1505-1520, Dec. 2010.
30. T. Nawaz and A. Cavallaro, "A protocol for evaluating video trackers under real-world conditions, " IEEE Trans. Image Process., vol. 22, no. 4, pp. 1354-1361, Apr. 2013.
31. P. Carvalho, J. S. Cardoso, L. Corte-Real, "Filling the gap in quality assessment of video object tracking, " Image Vision Comput., vol. 30, no. 9, pp. 630-640, 2012.
32. L.-Cehovin, M. Kristan, A. Leonardis, "Is my new tracker really better than yours" in Proc. IEEE Winter Conf. Appl. Comput. Vision, 2014, pp. 540-547.
33. L.-Cehovin, A. Leonardis, M. Kristan, "Visual object tracking performance measures revisited, " arXiv:1502. 05803 [cs. CV], 2015.
34. M. Everingham, L. Eslami, S. M. A. Van Gool, C. K. I. Williams, J. Winn, A. Zisserman, "The Pascal visual object classes challenge-A retrospective, " Int. J. Comput. Vision, vol. 111, pp. 98-136, 2014.
35. B. Babenko, M. H. Yang, S. Belongie, "Robust object tracking with online multiple instance learning, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 8, pp. 1619-1632, Aug. 2011.
36. H. Wu, A. C. Sankaranarayanan, R. Chellappa, "Online empirical evaluation of tracking algorithms, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 32, no. 8, pp. 1443-1458, Aug. 2010.
37. J. SanMiguel, A. Cavallaro, J. Mart-nez, "Adaptive on-line performance evaluation of video trackers, " IEEE Trans. Image Process., vol. 21, no. 5, pp. 2812-2823, May 2012.
38. D. M. Chu and A. W. M. Smeulders, "Thirteen hard cases in visual tracking, " in Proc. 7th IEEE Int. Conf. Adv. Video Signal Based Surveillance, 2010, pp. 103-110.
39. C. Jaynes, S. Webb, R. Steele, Q. Xiong, "An open development environment for evaluation of video surveillance systems, " in Proc. 3rd IEEE Int. Workshop Perform. Eval. Tracking Surveillance, 2002, pp. 32-39.
40. R. Collins, X. Zhou, S. K. Teh, "An open source tracking testbed and evaluation web site, " presented at the Int. Workshop Perform. Eval. Tracking Surveillance, Breckenridge, CO, USA, 2005.
41. D. Doermann and D. Mihalcik, "Tools and techniques for video performance evaluation, " in Proc. 15th Int. Conf. Pattern Recog., 2000, pp. 167-170.
42. M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, A. Zisserman, "The Pascal visual object classes (VOC) challenge, " Int. J. Comput. Vision, vol. 88, no. 2, pp. 303-338, Jun. 2010.
43. Y. Bar-Shalom, X. R. Li, T. Kirubarajan, Estimation with Applications to Tracking and Navigation. Hoboken, NJ, USA: Wiley, 2001, ch. 11, pp. 438-440.
44. T. W. Anderson and D. A. Darling, "Asymptotic theory of certain 'goodness of fit' criteria based on stochastic processes, " Ann. Math. Statist., vol. 23, no. 2, pp. 193-212, 1952.
45. J. Dem-sar, "Statistical comparisons of classifiers over multiple datasets, " J. Mach. Learning Res., vol. 7, pp. 1-30, 2006.
46. W. Navidi, Statistics for Engineers and Scientists. New York, NY, USA: McGraw-Hill, 2011.
47. J. Dem-sar, "On the appropriateness of statistical tests in machine learning, " in Proc. Workshop Evaluation Methods Mach. Learning, 2008, pp. 1-4.
48. G. Nebehay and R. Pflugfelder, "Consensus-based matching and tracking of keypoints for object tracking, " in Proc. Winter Conf. Appl. Comput. Vision, Mar. 2014, pp. 862-869.
49. M. E. Maresca and A. Petrosino, "Matrioska: A multi-level approach to fast tracking by learning, " in Proc. Int. Conf. Image Anal. Process., 2013, pp. 419-428.
50. T. Vojir and J. Matas, "Robustifying the flock of trackers, " in Proc. Comput. Vis. Winter Workshop, 2011, pp. 91-97.
51. K. Lebeda, R. Bowden, J. Matas, "Long-term tracking through failure cases, " in Proc. IEEE Int. Conf. Comput. Vision Workshops, 2013, pp. 153-160.
52. S. Duffner and C. Garcia, "Pixeltrack: A fast adaptive algorithm for tracking non-rigid objects, " in Proc. Int. Conf. Comput. Vision, 2013, pp. 2480-2487.
53. L.-Cehovin, M. Kristan, A. Leonardis, "Robust visual tracking using an adaptive coupled-layer visual model, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 4, pp. 941-953, Apr. 2013.
54. H. Nam, S. Hong, B. Han, "Online graph-based tracking, " in Proc. 13th Eur. Conf. Comput. Vision, 2014, pp. 112-126.
55. Z. Cai, L. Wen, Z. Lei, N. Vasconcelos, S. Li, "Robust deformable and occluded object tracking with dynamic graph, " IEEE Trans. Image Process., vol. 23, no. 12, pp. 5497-5509, Dec. 2014.
56. M. Maresca and A. Petrosino, "Clustering local motion estimates for robust and efficient object tracking, " in Proc. Workshop Visual Object Tracking Challenge/Eur. Conf. Comput. Vision, 2014, pp. 244-253.
57. A. Adam, E. Rivlin, I. Shimshoni, "Robust fragments-based tracking using the Integral Histogram, " in Proc. IEEE Conf. Comput. Vision Pattern Recog., Jun. 2006, vol. 1, pp. 798-805.
58. M. Felsberg, "Enhanced distribution field tracking using channel representations, " in Proc. IEEE Int. Conf. Comput. Vision Workshops, 2013, pp. 121-128.
59. K. Ofjall and M. Felsberg, "Weighted update and comparison for channel-based distribution field tracking, " in Proc. Workshop Visual Object Tracking Challenge/Eur. Conf. Comput. Vision, 2014, pp. 218-231.
60. L. Sevilla-Lara and E. G. Learned-Miller, "Distribution fields for tracking, " in Proc. IEEE Int. Conf. Comput. Vision Pattern Recog., 2012, pp. 1910-1917.
61. K. M. Yi, H. Jeong, S. W. Kim, J. Y. Choi, "Visual tracking with dual modeling, " in Proc. 27th Conf. Image Vision Comput. New Zealand, 2012, pp. 25-30.
62. T. Vojir, J. Noskova, J. Matas, "Robust scale-adaptive meanshift for tracking, " Pattern Recog. Lett., vol. 49, pp. 250-258, 2014.
63. M. Danelljan, F. S. Khan, M. Felsberg, J. Van de Weijer, "Adaptive color attributes for real-time visual tracking, " in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2014, pp. 1090-1097.
64. F. Henriques, R. Caseiro, P. Martins, J. Batista, "Exploiting the circulant structure of tracking-by-detection with kernels, " in Proc. 12th Eur. Conf. Comput. Vision, 2012, pp. 702-715.
65. K. Briechle and U. D. Hanebeck, "Template matching using fast normalized cross correlation, " Proc. SPIE, vol. 4387, pp. 95-102, 2001.
66. D. Comaniciu, V. Ramesh, P. Meer, "Kernel-based object tracking, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 25, no. 5, pp. 564-577, May 2003.
67. K. Nummiaro, E. Koller-Meier, L. Van Gool, "Color features for tracking non-rigid objects, " Chin. J. Autom., vol. 29, no. 3, pp. 345-355, May 2003.
68. P. P-erez, C. Hue, J. Vermaak, M. Gangnet, "Color-based probabilistic tracking, " in Proc. Eur. Conf. Comput. Vision, vol. 1, 2002, pp. 661-675.
69. K. Zhang, L. Zhang, M. H. Yang, "Real-time compressive tracking, " in Proc. Eur. Conf. Comput. Vision, 2012, pp. 864-877.
70. D. Comaniciu and P. Meer, "Mean shift: A robust approach toward feature space analysis, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 5, pp. 603-619, May 2002.
71. S. Duffner and C. Garcia, "Exploiting contextual motion cues for visual object tracking, " in Proc. Workshop Visual Object Tracking Challenge/Eur. Conf. Comput. Vision, 2014, pp. 1-12.
72. S. Hare, A. Saffari, P. H. S. Torr, "Struck: Structured output tracking with kernels, " in Proc. IEEE Int. Conf. Comput. Vision, 2011, pp. 263-270.
73. J. F. Henriques, R. Caseiro, P. Martins, J. Batista, "High-speed tracking with kernelized correlation filters, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 37, no. 3, pp. 583-596, Mar. 2015.
74. M. Danelljan, G. Hager, F. S. Khan, M. Felsberg, "Accurate scale estimation for robust visual tracking, " in Proc. Brit. Mach. Vision Conf., 2014, http://dx. doi. org/10. 5244/C. 28. 65
75. L. Yang and Z. Jianke, "A scale adaptive kernel correlation filter tracker with feature integration, " in Proc. Workshop Visual Object Tracking Challenge/Eur. Conf. Comput. Vision, 2014, pp. 254-265.
76. M. Kristan, J. Pers, M. Perse, S. Kovacic, "Bayes spectral entropy-based measure of camera focus, " in Proc. Comput. Vision Winter Workshop, Feb. 2005, pp. 155-164.
77. B. J. Frey and D. Dueck, "Clustering by passing messages between data points, " Science, vol. 315, pp. 972-976, 2007.
78. D. S. Bolme, J. R. Beveridge, B. A. Draper, Y. M. Lui, "Visual object tracking using adaptive correlation filters, " in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2010, pp. 2544-2550.
79. N. Dalal and B. Triggs, "Histograms of oriented gradients for human detection, " in Proc. IEEE Conf. Comput. Vision Pattern Recog., Jun. 2005, vol. 1, pp. 886-893.
80. A. Torralba and A. A. Efros, "Unbiased look at dataset bias, " in Proc. IEEE Conf. Comput. Vision Pattern Recog., 2011, pp. 1521-1528.
81. A. Li, M. Li, Y. Wu, M. Yang, S. Yan, "NUS-PRO: A new visual tracking challenge, " IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 2, pp. 335-349, Feb. 2016.