A $BV$ functional and its relaxation for joint motion estimation and image sequence recovery

Conti, Sergio; Ginster, Janusz; Rumpf, Martin

ESAIM: Mathematical Modelling and Numerical Analysis , Tome 49 (2015) no. 5, pp. 1463-1487.

The estimation of motion in an image sequence is a fundamental task in image processing. Frequently, the image sequence is corrupted by noise and one simultaneously asks for the underlying motion field and a restored sequence. In smoothly shaded regions of the restored image sequence the brightness constancy assumption along motion paths leads to a pointwise differential condition on the motion field. At object boundaries which are edge discontinuities both for the image intensity and for the motion field this condition is no longer well defined. In this paper a total-variation type functional is discussed for joint image restoration and motion estimation. This functional turns out not to be lower semicontinuous, and in particular fine-scale oscillations may appear around edges. By the general theory of vector valued $BV$ functionals its relaxation leads to the appearance of a singular part of the energy density, which can be determined by the solution of a local minimization problem at edges. Based on bounds for the singular part of the energy and under appropriate assumptions on the local intensity variation one can exclude the existence of microstructures and obtain a model well-suited for simultaneous image restoration and motion estimation. Indeed, the relaxed model incorporates a generalized variational formulation of the brightness constancy assumption. The analytical findings are related to ambiguity problems in motion estimation such as the proper distinction between foreground and background motion at object edges.

L. Ambrosio and A. Braides, Functionals defined on partitions in sets of finite perimeter. I. Integral representation and $\Gamma$-convergence. J. Math. Pures Appl. 69 (1990) 285–305. | Zbl

L. Ambrosio and A. Braides, Functionals defined on partitions in sets of finite perimeter. II. Semicontinuity, relaxation and homogenization. J. Math. Pures Appl. 69 (1990) 307–333. | Zbl

L. Ambrosio and G. Dal Maso, On the relaxation in BV(Ω;R^m) of quasi-convex integrals. J. Funct. Anal. 109 (1992) 76–97. | DOI | Zbl

L. Ambrosio, N. Fusco and D. Pallara, Functions of Bounded Variation and Free Discontinuity Problems. Oxford Math. Monogr. Oxford University Press, New York (2000). | Zbl

G. Aubert and P. Kornprobst, A mathematical study of the relaxed optical flow problem in the space $BV\left(\Omega \right)$. SIAM J. Math. Anal. 30 (1999) 1282–1308. | DOI | Zbl

G. Aubert, R. Deriche and P. Kornprobst, Computing optical flow via variational techniques. SIAM J. Appl. Math. 60 (1999) 156–182. | DOI | Zbl

P. Aviles and Y. Giga, Variational integrals on mappings of bounded variation and their lower semicontinuity. Arch. Ration. Mech. Anal. 115 (1991) 201–255. | DOI | Zbl

J. Bigun and G.H. Granlund, Optical flow based on the inertia matrix of the frequency domain. In Proc. of SSAB Symposium on Picture Processing: Lund University, Sweden (1988) 132–135.

G. Bouchitté, I. Fonseca, G. Leoni and L. Mascarenhas, A global method for relaxation in $W^{1,p}$ and $SB{V}_p$. Arch. Ration. Mech. Anal. 165 (2002) 187–242. | DOI | Zbl

G. Bouchitté, I. Fonseca and L. Mascarenhas, A global method for relaxation. Arch. Ration. Mech. Anal. 145 (1998) 51–98. | DOI | Zbl

T. Brox, A. Bruhn and J. Weickert, Variational motion segmentation with level sets. In Computer Vision – ECCV 2006. Edited by A.P. H. Bischof and A. Leonardis. Vol. 3951 of Lect. Notes Comput. Sci. Springer (2005) 471–483

C. Brune, H. Maurer and M. Wagner, Detection of intensity and motion edges within optical flow via multidimensional control. SIAM J. Img. Sci. 2 (2009) 1190–1210. | DOI | Zbl

C. Carstensen and P. Plech Àč, Numerical solution of the double-well problem allowing for microstructure. Math. Comput. 66 (1997) 997–1026. | DOI | Zbl

V. Caselles and B. Coll, Snakes in movement. SIAM J. Numer. Anal. 33 (1996) 2445–2456. | DOI | Zbl

A. Chambolle and T. Pock, A first-order primal-dual algorithm for convex problems with an applications to imaging. J. Math. Imaging and Vision 40 (2011) 120–145. | DOI | Zbl

I. Cohen, Nonlinear variational method for optical flow computation. In Proc. of the 8th SCIA, June 93 (1993) 523–530.

S. Conti, A. Garroni and A. Massaccesi, Modeling of dislocations and relaxation of functionals on 1-currents with discrete multiplicity. Calc. Var. Partial Differ. Eqs. (2015) DOI: 10.1007/s00526-015-0846-x.

D. Cremers and C. Schnörr, Motion competition: Variational integration of motion segmentation and shape regularization. In Pattern Recognition − Proc. of the DAGM, edited by L. Van Gool. Vol. 2449 of Lect. Notes in Comput. Sci. (2002) 472–480. | Zbl

D. Cremers and C. Schnörr, Statistical shape knowledge in variational motion segmentation. Image and Vision Computing 21 (2003) 77–86. | DOI

D. Cremers and S. Soatto, Motion competition: A variational framework for piecewise parametric motion segmentation. Int. J. Comput. Vision 62 (2005) 249–265. | DOI | Zbl

B. Dacorogna, Direct Methods in the Calculus of Variations. Springer-Verlag, New York (1989). | Zbl

L.C. Evans and R.F. Gariepy, Measure Theory and Fine Properties of Functions. CRC Press (1992). | Zbl

D. Fleet and Y. Weiss, Optical flow estimation. In Handbook of Mathematical Models in Computer Vision. Springer, New York (2006) 239–257.

I. Fonseca and S. Müller, Relaxation of quasiconvex functionals in BV(Ω,R^p) for integrands f(x,u,∇u). Arch. Ration. Mech. Anal. 123 (1993) 1–49. | DOI | Zbl

N. Fusco, M. Gori and F. Maggi, A remark on Serrin’s theorem. NoDEA Nonlin. Differ. Eqs. Appl. 13 (2006) 425–433. | DOI | Zbl

C. Goffman and J. Serrin, Sublinear functions of measures and variational integrals. Duke Math. J. 31 (1964) 159–178. | DOI | Zbl

F. Guichard, A morphological, affine and galilean invariant scale–space for movies. IEEE Trans. Image Process. 7 (1998) 444–456. | DOI

W. Hinterberger, O. Scherzer, C. Schnörr and J. Weickert, Analysis of optical flow models in the framework of calculus of variations. Technical Report No. 8, University of Mannheim, Germany (2001). | Zbl

B.K.P. Horn and B.G. Schunck, Determining optical flow. Artificial Intelligence 17 (1981) 185–203. | DOI | Zbl

T.Y. Hou and X.H. Wu, A multiscale finite element method for elliptic problems in composite materials and porous media. J. Comput. Phys. 134 (1997) 169–189. | DOI | Zbl

K. Ito, An optimal optical flow, SIAM J. Control Optim. 44 (2005) 728–742. | DOI | Zbl

C. Kanglin and D.A. Lorenz, Image sequence interpolation based on optical flow, segmentation and optimal control. Image Process., IEEE Trans. 21 (2012) 1020–1030.

P. Kornprobst, R. Deriche and G. Aubert, Image sequence analysis via partial differential equations. J. Math. Imaging Vision 11 (1999) 5–26. | DOI

J. Kristensen and F. Rindler, Relaxation of signed integral functionals in BV. Calc. Var. Partial Differ. Equ. 37 (2010) 29–62. | DOI | Zbl

L. Le Tarnec, F. Destrempes, G. Cloutier and D. Garcia, A proof of convergence of the Horn–Schunck optical flow algorithm in arbitrary dimension. SIAM J. Imaging Sci. 7 (2014) 277–293. | DOI | Zbl

B. D. Lucas and T. Kanade, An iterative image registration technique with an application to stereo vision. In Proc. of Seventh International Joint Conference on Artificial Intelligence. Vancouver, Canada (1981) 674–679.

E. Memin and P. Perez, A Multigrid Approach for Hierarchical Motion Estimation. In Proc. of ICCV (1998) 933–938.

H.H. Nagel and W. Enkelmann, An investigation of smoothness constraints for the estimation of displacement vector fields from image sequences. IEEE Trans. Pattern Anal. Mach. Intell. 8 (1986) 565–593. | DOI

H.H. Nagel and M. Otte, Optical flow estimation: Advances and comparisons. In Proc. of the 3rd European Conference on Computer Vision. Edited by J.-O. Eklundh. Vol. 800 of Lect. Notes Comput. Sci. Springer (1994) 51–70.

P. Nesi, Variational approach to optical flow estimation managing discontinuities. Image and Vision Comput. 11 (1993) 419–439. | DOI

J.-M. Odobez and P. Bouthemy, Direct incremental model-based image motion segmentation for video analysis. Signal Process. 66 (1998) 143–155. | DOI | Zbl

N. Papenberg, A. Bruhn, T. Brox, S. Didas and J. Weickert, Highly accurate optic flow computation with theoretically justified warping. Int. J. Computer Vision 67 (2006) 141–158. | DOI

N. Paragios and R. Deriche, Geodesic active contours and level sets for the detection and tracking of moving objects. IEEE Trans. Pattern Anal. Mach. Intel. 22 (2000) 266–280. | DOI

J.S. Perez, E. Meinhardt-Llopis and G. Facciolo, TV-L1 Optical flow estimation. Image Processing On Line 3 (2013) 137–150. | DOI

Y. Rathi, N. Vaswani, A. Tannenbaum and A. Yezzi, Particle filtering for geometric active contours with application to tracking moving and deforming objects. In CVPR IEEE Computer Society (2005) 1–8.

Ju. G. Rešetnjak, General theorems on semicontinuity and convergence with functionals. Sibirsk. Mat. Ž. 8 (1967) 1051–1069. | Zbl

F. Rindler, Lower semicontinuity and Young measures in BV without Alberti’s rank-one theorem. Adv. Calc. Var. 5 (2012) 127–159. | DOI | Zbl

L. Rudin, S. Osher and E. Fatemi, Nonlinear total variation based noise removal algorithms. Physica D 60 (1992) 259–268. | DOI | Zbl

M. Tristarelli, Computation of coherent optical flow by using multiple constraints. In Proc. of International Conference on Computer Vision. IEEE Computer Society Press (1995) 263–268.

A. Wedel, T. Pock, C. Zach, H. Bischof and D. Cremers, An improved algorithm for TV-L1 optical flow. In Statistical and Geometrical Approaches to Visual Motion Analysis, edited by D. Cremers, B. Rosenhahn, A.L. Yuille and F.R. Schmidt. Vol. 5604 of Lect. Notes Comput. Sci. Springer Berlin Heidelberg (2009) 23–45.

J. Weickert, A. Bruhn and C. Schnörr, Lucas/Kanade meets Horn/Schunck: Combining local and global optic flow methods. International. J. Computer Vision 61 (2005) 211–231. | Zbl

J. Weickert and Ch. Schnörr, Variational optic flow computation with a spatio-temporal smoothness constraint. J. Math. Imaging Vision 14 (2001) 245–255. | DOI | Zbl

E. Weinan, P.B. Ming and P.W. Zhang, Analysis of the heterogeneous multiscale method for elliptic homogenization problems. J. Amer. Math. Soc. 18 (2005) 121–156. | Zbl

J. Yuan, C. Schnörr and G. Steidl, Simultaneous higher-order optical flow estimation and decomposition. SIAM J. Sci. Comput. 29 (2007) 2283–2304. | DOI | Zbl