Segmentation of moving objects by long term video analysis

IEEE Transactions on Pattern Analysis and Machine Intelligence

Volumn 36, Issue 6, 2014, Pages 1187-1200

  Ochs, Peter ^a   Malik, Jitendra ^b   Brox, Thomas ^a  

^a UNIVERSITY OF FREIBURG   (Germany)

^b UNIVERSITY OF CALIFORNIA   (United States)

Author keywords

Computer vision; motion segmentation; point trajectories; variational methods

Indexed keywords

ARTIFICIAL INTELLIGENCE;

MOTION SEGMENTATION; MOVING OBJECTS; POINT TRAJECTORY; POSITIVE SIDES; POST PROCESSING; SHORT-TERM VARIATIONS; VARIATIONAL METHODS; VIDEO ANALYSIS;

COMPUTER VISION;

EID: 84901822916     PISSN: 01628828     EISSN: None     Source Type: Journal    
DOI: 10.1109/TPAMI.2013.242     Document Type: Article

References (75)

1. Large Displacement Optical Flow Tracker [Online]. Available: http://lmb.informatik.uni-freiburg.de/resources/binaries/eccv 2010-trackingLinux64.zip
2. Motion Segmentation Dataset (FBMS) [Online]. Available: http://lmb.informatik.uni-freiburg.de/resources/datasets/ FBMS-Trainingset.tar. gz and http://lmb.informatik.uni-freiburg. de/resources/datasets/FBMS-Testset. tar.gz
3. T. Amiaz and N. Kiryati, "Piecewise-smooth dense optical flow via level sets," Int. J. Comput. Vis., vol. 68, no. 2, pp. 111-124, Jun. 2006. (Pubitemid 43724241)
4. P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, "Contour detection and hierarchical image segmentation," IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 5, pp. 898-916, May 2011.
5. A. Ayvaci, M. Raptis, and S. Soatto, "Sparse occlusion detection with optical flow," Int. J. Comput. Vis., vol. 97, no. 3, pp. 322-338, 2012.
6. X. Bai and G. Sapiro, "A geodesic framework for fast interactive image and video segmentation and matting," in Proc. 11th Int. Conf. Computer Vision, Rio de Janeiro, Brazil, 2007.
7. S. Belongie and J. Malik, "Finding boundaries in natural images: A new method using point descriptors and area completion," in Proc. European Conf. Computer Vision, Freiburg, Germany, 1998, pp. 751-766, LNCS 1406. (Pubitemid 128093117)
8. D. Beymer, P. McLauchlan, B. Coifman, and J. Malik, "A realtime computer vision system for measuring traffic parameters," in Proc. Int. Conf. CVPR, San Juan, Puerto Rico, 1997, pp. 495-501.
9. S. Birchfield and S. Pundlik, "Joint tracking of features and edges," in Proc. Int. Conf. CVPR, Anchorage, AK, USA Jun. 2008.
10. T. Boult and L. Brown, "Factorization-based segmentation of motions," in Proc. IEEE Workshop Visual Motion, Princeton, NJ, USA, 1991, pp. 179-186. (Pubitemid 23618715)
11. Y. Boykov and G. Funka-Lea, "Graph cuts and efficient N-D image segmentation," Int. J. Comput. Vis., vol. 70, no. 2, pp. 109-131, 2006. (Pubitemid 44127252)
12. W. Brendel and S. Todorovic, "Video object segmentation by tracking regions," in Proc. Int. Conf. Computer Vision, Kyoto, Japan, 2009.
13. G. Brostow and R. Cipolla, "Unsupervised Bayesian detection of independent motion in crowds," in Proc. Int. Conf. Computer Vision Pattern Recognition, New York, NY, USA, 2006.
14. G. Brostow, J. Shotton, J. Fauqueur, and R. Cipolla, "Segmentation and recognition using structure from motion point clouds," in Proc. ECCV, Berlin, Germany, 2008, pp. 44-57.
15. T. Brox, A. Bruhn, and J. Weickert, "Variational motion segmentation with level sets," in Proc. ECCV, A. Leonardis, H. Bischof, and A. Pinz, Eds., Graz, Austria, 2006, pp. 471-483, LNCS 3951. (Pubitemid 44029800)
16. T. Brox and D. Cremers, "On local region models and the statistical interpretation of the piecewise smooth Mumford-Shah functional," Int. J. Comput. Vis., vol. 84, no. 2, pp. 184-193, 2009.
17. T. Brox and J. Malik, "Object segmentation by long-term analysis of point trajectories," in Proc. ECCV, Heraklion, Greece, Sept. 2010.
18. T. Brox and J. Malik, "Large displacement optical flow: Descriptor matching in variational motion estimation," IEEE Trans. Pattern Anal. Mach. Intell., vol. 33, no. 3, pp. 500-513, Mar. 2011.
19. A. Chambolle, D. Cremers, and T. Pock, "A convex approach to minimal partitions," SIAM J. Appl. Math., vol. 5, no. 4, pp. 1113-1158, 2012.
20. A. Chambolle and T. Pock, "A first-order primal-dual algorithm for convex problems with applications to imaging," J. Math. Imaging Vis., vol. 40, no. 1, pp. 120-145, 2011.
21. T. Chan, S. Esedoglu, and M. Nikolova, "Algorithms for finding global minimizers of image segmentation and denoising models," SIAM J. Appl. Math., vol. 66, no. 5, pp. 1632-1648, 2006. (Pubitemid 44599935)
22. A. Cheriyadat and R. Radke, "Non-negative matrix factorization of partial track data for motion segmentation," in Proc. ICCV, 2009.
23. J. Costeira and T. Kanande, "A multi-body factorization method for motion analysis," in Proc. ICCV, 1995, pp. 1071-1076.
24. D. Cremers, T. Pock, K. Kolev, and A. Chambolle, "Convex relaxation techniques for segmentation, stereo and multiview reconstruction," in Advances in Markov Random Fields for Vision and Image Processing. MIT Press, 2011.
25. D. Cremers and S. Soatto, "Motion competition: A variational framework for piecewise parametric motion segmentation," Int. J. Comput. Vis., vol. 62, no. 3, pp. 249-265, May 2005. (Pubitemid 40356777)
26. E. Elhamifar and R. Vidal, "Sparse subspace clustering: Algorithm, theory, and applications," IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 11, pp. 2765-2781, Nov. 2013.
27. M. Fradet, P. Robert, and P. Pérez, "Clustering point trajectories with various life-spans," in Proc. Eur. Conf. Visual Media Production, London, U.K., 2009, pp. 7-14.
28. K. Fragkiadaki and J. Shi, "Detection free tracking: Exploiting motion and topology for segmenting and tracking under entanglement," in Proc. Int. Conf. CVPR, 2011.
29. K. Fragkiadaki, G. Zhang, and J. Shi, "Video segmentation by tracing discontinuities in a trajectory embedding," in Proc. Int. Conf. CVPR, Providence, RI, USA, 2012.
30. K. Fragkiadaki, W. Zhang, G. Zhang, and J. Shi, "Two granularity tracking: Mediating trajectory and detection graphs for tracking under occlusions," in Proc. ECCV, Florence, Italy, 2012, pp. 552-565.
31. C. Gear, "Multibody grouping from motion images," Int. J. Comput. Vis., vol. 29, no. 2, pp. 133-150, Aug. 1998. (Pubitemid 128517908)
32. L. Grady, "Random walks for image segmentation," IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no. 11, pp. 1768-1783, Nov. 2006. (Pubitemid 46405425)
33. M. Grundmann, V. Kwatra, M. Han, and I. Essa, "Efficient hierarchical graph-based video segmentation," in Proc. Int. Conf. CVPR, San Francisco, CA, USA, 2010, pp. 2141-2148.
34. K. Koffka, Principles of Gestalt Psychology. New York, NY, USA: Hartcourt Brace Jovanovich, 1935.
35. N. Komodakis and G. Tziritas, "Approximate labeling via graphcuts based on linear programming," IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 8, pp. 1436-1453, Jun. 2007. (Pubitemid 47040441)
36. N. Komodakis, G. Tziritas, and N. Paragios, "Performance vs computational efficiency for optimizing single and dynamic MRFs: Setting the state of the art with primal-dual strategies," Comput. Vis. Image Understand., vol. 112, no. 1, pp. 14-29, Oct. 2008.
37. F. Lauer and C. Schnörr, "Spectral clustering of linear subspaces for motion segmentation," in Proc. ICCV, Kyoto, Japan, 2009, pp. 678-685.
38. J. Lellmann, F. Becker, and C. Schnörr, "Convex optimization for multi-class image labeling with a novel family of total variation based regularizers," in Proc. ICCV, Kyoto, Japan, 2009, pp. 646-653.
39. J. Lezama, K. Alahari, J. Sivic, and I. Laptev, "Track to the future: Spatio-temporal video segmentation with long-range motion cues," in Proc. Int. Conf. CVPR, Providence, RI, USA, Jun. 2011.
40. T. Li, V. Kallem, D. Singaraju, and R. Vidal, "Projective factorization of multiple rigid-body motions," in Proc. Int. Conf. CVPR, Minneapolis, MN, USA, Jun. 2007, pp. 1-6.
41. G. Liu, Z. Lin, and Y. Yu, "Robust subspace segmentation by low-rank representation," in Proc. ICML, Haifa, Israel, 2010, pp. 663-670.
42. R. Liu, Z. Lin, F. De la Torre, and Z. Su, "Fixed-rank representation for unsupervised visual learning," in Proc. Int. Conf. CVPR, Providence, RI, USA, Jun. 2012, pp. 598-605.
43. B. Lucas and T. Kanade, "An iterative image registration technique with an application to stereo vision," in Proc. 7th Int. Joint Conf. Artificial Intell., Canada, U.K., Aug. 1981, pp. 674-679. (Pubitemid 14478744)
44. A. Y. Ng, M. I. Jordan, and Y. Weiss, "On spectral clustering: Analysis and an algorithm," in Proc. NIPS, 2002, pp. 849-856.
45. C. Nieuwenhuis, B. Berkels, M. Rumpf, and D. Cremers, "Interactive motion segmentation," in Proc. 32nd DAGM, Darmstadt, Germany, 2010, pp. 483-492, LNCS 6376.
46. P. Ochs and T. Brox, "Object segmentation in video: A hierarchical variational approach for turning point trajectories into dense regions," in Proc. ICCV, Barcelona, Spain, Nov. 2011, pp. 1583-1590.
47. P. Ochs and T. Brox, "Higher order motion models and spectral clustering," in Proc. Int. Conf. CVPR, Providence, RI, USA, Jun. 2012, pp. 614-621.
48. B. Ommer, T. Mader, and J. Buhmann, "Seeing the objects behind the dots: Recognition in videos from a moving camera," Int. J. Comput. Vis., vol. 83, no. 1, pp. 57-71, 2009.
49. Y. Ostrovsky, E. Meyers, S. Ganesh, U. Mathur, and P. Sinha, "Visual parsing after recovery from blindness," Psychol. Sci., vol. 20, no. 12, pp. 1484-1491, 2009.
50. M. Pawan Kumar, P. Torr, and A. Zisserman, "Learning layered motion segmentations of video," Int. J. Comput. Vis., vol. 76, pp. 301-319, 2008.
51. A. Prest, C. Leistner, J. Civera, C. Schmid, and V. Ferrari, "Learning object class detectors from weakly annotated video," in Proc. Int. Conf., Providence, RI, USA, Jun. 2012, pp. 3282-3289.
52. B. L. Price, B. S. Morse, and S. Cohen, "LIVEcut: Learningbased interactive video segmentation by evaluation of multiple propagated cues," in Proc. ICCV, Kyoto, Japan, 2009, pp. 779-786.
53. S. R. Rao, R. Tron, R. Vidal, and Y. Ma, "Motion segmentation via robust subspace separation in the presence of outlying, incomplete, or corrupted trajectories," in Proc. Int. Conf., Anchorage, AK, USA, Jun. 2008, pp. 1-8.
54. P. Sand and S. Teller, "Particle video: Long-range motion estimation using point trajectories," Int. J. Comput. Vis., vol. 80, no. 1, pp. 72-91, Oct. 2008.
55. K. Schindler, D. Suter, and H. Wang, "A model-selection framework for multibody structure-and-motion of image sequences," Int. J. Comput. Vis., vol. 79, no. 2, pp. 159-177, 2008.
56. J. Shi and J. Malik, "Motion segmentation and tracking using normalized cuts," in Proc. 6th Int. Conf. Computer Vision, Bombay, India, Jan. 1998, pp. 1154-1160.
57. J. Shi and J. Malik, "Normalized cuts and image segmentation," IEEE Trans. Pattern Anal. Mach. Intell., vol. 22, no. 8, pp. 888-905, Aug. 2000.
58. J. Shi and C. Tomasi, "Good features to track," in Proc. Int. Conf. CVPR, Seattle, WA, USA, Jun. 1994, pp. 593-600.
59. S. Sinha, J.-M. Frahm, M. Pollefeys, and Y. Genc, "Feature tracking and matching in video using programmable graphics hardware," Mach. Vis. Appl., vol. 22, no. 1, pp. 207-217, Jan. 2011.
60. A. Sinop and L. Grady, "A seeded image segmentation framework unifying graph cuts and random walker which yields a new algorithm," in Proc. ICCV, Rio de Janeiro, Brazil, Oct. 2007.
61. J. Sivic, F. Schaffalitzky, and A. Zisserman, "Object level grouping for video shots," Int. J. Comput. Vis., vol. 67, no. 2, pp. 189-210, Jan. 2006.
62. P. Sturgess, K. Alahari, L. Ladicky, and P. Torr, "Combining appearance and structure from motion features for road scene understanding," in Proc. BMVC, 2009.
63. D. Sun, E. B. Sudderth, and M. Black, "Layered segmentation and optical flow estimation over time," in Proc. Int. Conf. CVPR, Providence, RI, USA, Jun. 2012, pp. 1768-1775.
64. N. Sundaram, T. Brox, and K. Keutzer, "Dense point trajectories by GPU-accelerated large displacement optical flow," in Proc. ECCV, Heraklion, Greece, 2010, LNCS.
65. R. Tron and R. Vidal, "A benchmark for the comparison of 3- D motion segmentation algorithms," in Proc. Int. Conf. CVPR, Minneapolis, MN, USA, Jun. 2007, pp. 1-8.
66. M. Unger, T. Pock, W. Trobin, D. Cremers, and H. Bischof, "TVSeg - Interactive total variation based image segmentation," in Proc. BMVC, 2008.
67. A. Vazquez-Reina, S. Avidan, H. Pfister, and E. Miller, "Multiple hypothesis video segmentation from superpixel flows," in Proc. ECCV, Berlin, Germany, 2010, LNCS.
68. R. Vidal, R. Tron, and R. Hartley, "Multiframe motion segmentation with missing data using powerfactorization and GPCA," Int. J. Comput. Vis., vol. 79, no. 1, pp. 85-105, 2008. (Pubitemid 351578661)
69. J. Y. A. Wang and E. H. Adelson, "Representing moving images with layers," IEEE Trans. Image Proces., vol. 3, no. 5, pp. 625-638, Sept. 1994.
70. C. Wojek, S. Walk, S. Roth, K. Schindler, and B. Schiele, "Monocular visual scene understanding: Understanding multiobject traffic scenes," IEEE Trans. Pattern Anal. Mach. Intell., vol. 35, no. 4, pp. 882-897, Apr. 2013.
71. J. Xiao and M. Shah, "Motion layer extraction in the presence of occlusion using graph cuts," IEEE Trans. Pattern Anal. Mach. Intell., vol. 27, no. 10, pp. 1644-1659, Oct. 2005. (Pubitemid 41555408)
72. J. Yan and M. Pollefeys, "A general framework for motion segmentation: Independent, articulated, rigid, non-rigid, degenerate and non-degenerate," in Proc. ECCV, Graz, Austria, pp. 94-106, 2006, LNCS 3954.
73. C. Zach, D. Gallup, and J.-M. Frahm, "Fast gain-adaptive KLT tracking on the GPU," in Proc. CVPR Workshop Visual Computer Vision GPUs, Anchorage, AK, USA, 2008, pp. 1-7.
74. L. Zappella, X. Lladó, E. Provenzi, and J. Salvi, "Enhanced local subspace affinity for feature-based motion segmentation," Pattern Recognit., vol. 44, no. 2, pp. 454-470, Feb. 2011.
75. G. Zhang, Z. Yuan, D. Chen, Y. Liu, and N. Zheng, "Video object segmentation by clustering region trajectories," in Proc. Int. Conf. Pattern Recognition, Tsukuba, Japan, 2012.