Optical Flow

Estimate object velocities

• 

Libraries:
Computer Vision Toolbox / Analysis & Enhancement

Description

The Optical Flow block estimates the direction and speed of object motion between two images or between one video frame to another frame using either the Horn-Schunck or the Lucas-Kanade method.

Examples

Compute Optical Flow Velocities

Compute the optical flow velocities for a moving object in a video or image sequence.

• Open Script

Ports

Input

I/I1 — Image or video frame

scalar | vector | matrix

Image or video frame, specified as a scalar, vector, or matrix. If the Compute optical flow between parameter is set to Two images, the name of this port changes to I1.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | fixed point

I2 — Image

scalar | vector | matrix

Image or video frame, specified as a scalar, vector, or matrix.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | fixed point

Output

|V|^2 — Velocity magnitudes

scalar | vector | matrix

Velocity magnitudes, returned as a scalar, vector, or matrix.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | fixed point

V — Velocity components in complex form

scalar | vector | matrix

Velocity components in complex form, specified as a scalar, vector, or matrix.

Dependencies

To enable this port, set the Velocity output parameter to Horizontal and vertical components in complex form.

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | fixed point

Parameters

Main Tab

Method — Optical flow calculation method

Horn-Schunck (default) | Lucas-Kanade

Select the method to use to calculate the optical flow. Options include Horn-Schunck or Lucas-Kanade.

Compute optical flow between — Compute optical flow

Current frame and N-th frame back (default) | Two images

Select how to compute the optical flow. Select Two images to compute the optical flow between two images. Select Current frame and N-th frame back to compute the optical flow between two video frames that are N frames apart.

Dependencies

To enable this parameter, set the:

• Method parameter to Horn-Schunck

• Method parameter to Lucas-Kanade and the Temporal gradient filter to Difference filter [-1 1]

N — Number of frames

1 (default) | scalar

Enter a scalar value that represents the number of frames between the reference frame and the current frame.

Dependencies

To enable this parameter, set the Compute optical flow between parameter to Current frame and N-th frame back.

Smoothness factor — Smoothness factor

1 (default) | positive scalar

Specify the smoothness factor. Enter a large positive scalar value for high relative motion between the two images or video frames. Enter a small positive scalar value for low relative motion.

Dependencies

To enable this parameter, set the Method parameter to Horn-Schunck.

Stop iterative solution — Stop iterative solution

When maximum number of iterations is reached (default) | When velocity difference falls below threshold | Whichever comes first

Specify the method to control when the block's iterative solution process stops. If you want the process to stop when the velocity difference is below a certain threshold value, select When velocity difference falls below threshold. If you want the process to stop after a certain number of iterations, choose When maximum number of iterations is reached. You can also select Whichever comes first.

Dependencies

To enable this parameter, set the Method parameter to Horn-Schunck.

Maximum number of iterations — Maximum number of iterations

10 (default) | scalar

Specify the maximum number of iterations for the block to perform.

Dependencies

To enable this parameter, set the Method parameter to Horn-Schunck and the Stop iterative solution parameter to When maximum number of iterations is reached or Whichever comes first.

Velocity output — Optical flow output

Magnitude-squared (default) | Horizontal and vertical components in complex form

Specify how to output an optical flow. If you select Magnitude-squared, the block outputs an optical flow matrix where each element is in the form $$u^2+v^2$$. If you select Horizontal and vertical components in complex form, the block outputs the optical flow matrix where each element is in the form $$u+jv$$.

Temporal gradient filter — Filter used for temporal gradient

Difference filter [-1 1] (default) | Derivative of Gaussian

Specify whether the block solves for u and v using a difference filter or a derivative of a Gaussian filter.

Dependencies

To enable this parameter, set the Method parameter to Lucas-Kanade.

Number of input frames to buffer — Number of input frames to buffer for smoothing

3 (default) | scalar

Specify the number of input frames to buffer for smoothing. Use this parameter for temporal filter characteristics such as the standard deviation and number of filter coefficients.

Dependencies

To enable this parameter, set the Temporal gradient filter parameter to Derivative of Gaussian.

Standard deviation for image smoothing filter — Standard deviation for image smoothing filter

1.5 (default) | scalar

Specify the standard deviation for the image smoothing filter.

Dependencies

To enable this parameter, set the Temporal gradient filter parameter to Derivative of Gaussian.

Standard deviation for gradient smoothing filter — Standard deviation for gradient smoothing filter

1 (default) | scalar

Specify the standard deviation for the gradient smoothing filter.

Dependencies

To enable this parameter, set the Temporal gradient filter parameter to Derivative of Gaussian.

Discard normal flow estimates when constraint equation is ill-conditioned — Discard normal flow estimates

off (default) | on

Select this parameter to set the motion vector to zero when the optical flow constraint equation is ill-conditioned.

Dependencies

To enable this parameter, set the Temporal gradient filter parameter to Derivative of Gaussian.

Output image corresponding to motion vectors (accounts for block delay) — Output image corresponding to motion vectors

off (default) | on

Select this parameter to output the image that corresponds to the motion vector outputted by the block.

Dependencies

To enable this parameter, set the Temporal gradient filter parameter to Derivative of Gaussian.

Threshold for noise reduction — Threshold for noise reduction

0.0039 (default) | scalar

Specify a scalar value that determines the motion threshold between each image or video frame. The higher the number, the less small movements impact the optical flow calculation.

Dependencies

To enable this parameter, set the Method parameter to Lucas-Kanade.

Data Types Tab

For details on the fixed-point block parameters, see Specify Fixed-Point Attributes for Blocks (DSP System Toolbox).

Block Characteristics

Data Types	double | single
Multidimensional Signals	no
Variable-Size Signals	no

Algorithms

Optical Flow Equation

To compute the optical flow between two images, you must solve this optical flow constraint equation:

$$I_{x}u+I_{y}v+I_t=0$$

.

• $$I_x$$, $$I_y$$, and $$I_t$$ are the spatiotemporal image brightness derivatives.

• u is the horizontal optical flow.

• v is the vertical optical flow.

Horn-Schunck Method

By assuming that the optical flow is smooth across the entire image, the Horn-Schunck method estimates a velocity field, $$[\begin{array}{cc}u& v{]}^T\end{array}$$, that minimizes this equation:

$$E={\displaystyle \iint (I_{x}u+I_{y}v}+I_t{)}^{2}dxdy+\alpha {\displaystyle \iint \left\{{\left(\frac{\partial u}{\partial x}\right)}^2+{\left(\frac{\partial u}{\partial y}\right)}^2+{\left(\frac{\partial v}{\partial x}\right)}^2+{\left(\frac{\partial v}{\partial y}\right)}^2\right\}}dxdy$$

.

In this equation, $$\frac{\partial u}{\partial x}$$ and $$\frac{\partial u}{\partial y}$$ are the spatial derivatives of the optical velocity component, u, and $$\alpha$$ scales the global smoothness term. The Horn-Schunck method minimizes the previous equation to obtain the velocity field, [u v], for each pixel in the image. This method is given by the following equations:

$$\begin{array}{l}{u}_{x,y}^{k+1}={\overline{u}}_{x,y}^k-\frac{I_x[I_x{\overline{u}}^k{}_{x,y}+I_y{\overline{v}}^k{}_{x,y}+I_t]}{{\alpha }^2+I_x^2+I_y^2}\\ v_{x,y}^{k+1}={\overline{v}}_{x,y}^k-\frac{I_y[I_x{\overline{u}}^k{}_{x,y}+I_y{\overline{v}}^k{}_{x,y}+I_t]}{{\alpha }^2+I_x^2+I_y^2}\end{array}$$

.

In these equations, $$\left[\begin{array}{cc}{u}_{x,y}^k& v_{x,y}^k\end{array}\right]$$ is the velocity estimate for the pixel at (x,y), and $$\left[\begin{array}{cc}{\overline{u}}_{x,y}^k& {\overline{v}}_{x,y}^k\end{array}\right]$$ is the neighborhood average of $$\left[\begin{array}{cc}{u}_{x,y}^k& v_{x,y}^k\end{array}\right]$$. For k = 0, the initial velocity is 0.

To solve u and v using the Horn-Schunck method:

1. Compute $$I_x$$ and $$I_y$$ by using the Sobel convolution kernel, $$\left[\begin{array}{ccc}-1& -2& \begin{array}{ccc}\begin{array}{ccc}\begin{array}{ccc}-1;& 0& 0\end{array}& 0;& 1\end{array}& 2& 1\end{array}\end{array}\right]$$, and its transposed form for each pixel in the first image.

2. Compute $$I_t$$ between images 1 and 2 using the $$\left[\begin{array}{cc}-1& 1\end{array}\right]$$ kernel.

3. Assume the previous velocity to be 0, and compute the average velocity for each pixel using $$\left[\begin{array}{ccc}0& 1& \begin{array}{ccc}0;& 1& \begin{array}{ccc}0& 1;& \begin{array}{ccc}0& 1& 0\end{array}\end{array}\end{array}\end{array}\right]$$ as a convolution kernel.

4. Iteratively solve for u and v.

Lucas-Kanade Method

To solve the optical flow constraint equation for u and v, the Lucas-Kanade method divides the original image into smaller sections and assumes a constant velocity in each section. Then it performs a weighted, least-square fit of the optical flow constraint equation to a constant model for $${\left[\begin{array}{cc}u& v\end{array}\right]}^T$$ in each section $$\Omega$$. The method achieves this fit by minimizing this equation:

$${\displaystyle \sum _{x\in \Omega }{W}^2{[I_{x}u+I_{y}v+I_t]}^2}$$

W is a window function that emphasizes the constraints at the center of each section. The solution to the minimization problem is

$$\left[\begin{array}{cc}{\displaystyle \sum W^2{I}_x^2}& {\displaystyle \sum W^2{I}_x{I}_y}\\ {\displaystyle \sum W^2{I}_y{I}_x}& {\displaystyle \sum W^2{I}_y^2}\end{array}\right]\left[\begin{array}{c}u\\ v\end{array}\right]=-\left[\begin{array}{c}{\displaystyle \sum W^2{I}_x{I}_t}\\ {\displaystyle \sum W^2{I}_y{I}_t}\end{array}\right]$$

.

Lucas-Kanade Difference Filter

When you set the Temporal gradient filter to Difference filter [-1 1], u and v are solved as follows:

1. Compute $$I_x$$ and $$I_y$$ using the kernel $$\left[\begin{array}{cccc}-1& 8& 0& \begin{array}{cc}-8& 1\end{array}\end{array}\right]/12$$ and its transposed form.

   If you are working with fixed-point data types, the kernel values are signed fixed-point values with a word length equal to 16 and a fraction length equal to 15.

2. Compute $$I_t$$ between images 1 and 2 by using the $$\left[\begin{array}{cc}-1& 1\end{array}\right]$$ kernel.

3. Smooth the gradient components, $$I_x$$, $$I_y$$, and $$I_t$$, by using a separable and isotropic 5-by-5 element kernel whose effective 1-D coefficients are $$\left[\begin{array}{cccc}\begin{array}{cc}1& 4\end{array}& 6& 4& 1\end{array}\right]/16$$. If you are working with fixed-point data types, the kernel values are unsigned fixed-point values with a word length equal to 8 and a fraction length equal to 7.

4. Solve the 2-by-2 linear equations for each pixel using the following method:

   • If $$A=\left[\begin{array}{cc}a& b\\ b& c\end{array}\right]=\left[\begin{array}{cc}{\displaystyle \sum W^2{I}_x^2}& {\displaystyle \sum W^2{I}_x{I}_y}\\ {\displaystyle \sum W^2{I}_y{I}_x}& {\displaystyle \sum W^2{I}_y^2}\end{array}\right]$$

     then the eigenvalues of A are $${\lambda }_i=\frac{a+c}{2}\pm \frac{\sqrt{4b^2+{(a-c)}^2}}{2};i=1,2$$

     In the fixed-point diagrams, $$P=\frac{a+c}{2},Q=\frac{\sqrt{4b^2+{(a-c)}^2}}{2}$$

   • The eigenvalues are compared to the threshold, $$\tau$$, that corresponds to the value you enter for the threshold for noise reduction. The results fall into one of the following cases.

     Case 1: $${\lambda }_1\ge \tau$$ and $${\lambda }_2\ge \tau$$

     A is nonsingular, and the system of equations is solved using Cramer's rule.

     Case 2: $${\lambda }_1\ge \tau$$ and $${\lambda }_2<\tau$$

     A is singular (noninvertible), and the gradient flow is normalized to calculate u and v.

     Case 3: $${\lambda }_1<\tau$$ and $${\lambda }_2<\tau$$

     The optical flow, u and v, is 0.

Derivative of Gaussian

If you set the temporal gradient filter to Derivative of Gaussian, u and v are solved using these steps.

1. Compute $$I_x$$ and $$I_y$$.

   1. Use a Gaussian filter to perform temporal filtering. Specify the temporal filter characteristics, such as the standard deviation and number of filter coefficients, by using the Number of frames to buffer for temporal smoothing parameter.

   2. Use a Gaussian filter and the derivative of a Gaussian filter to smooth the image by using spatial filtering. Specify the standard deviation and length of the image smoothing filter by using the Standard deviation for image smoothing filter parameter.

2. Compute $$I_t$$ between images 1 and 2.

   1. Use the derivative of a Gaussian filter to perform temporal filtering. Specify the temporal filter characteristics, such as the standard deviation and number of filter coefficients, by using the Number of frames to buffer for temporal smoothing parameter.

   2. Use the filter described in step 1b to perform spatial filtering on the output of the temporal filter.

3. Smooth the gradient components, $$I_x$$, $$I_y$$, and $$I_t$$, by using a gradient smoothing filter. Use the Standard deviation for gradient smoothing filter parameter to specify the standard deviation and the number of filter coefficients for the gradient smoothing filter.

4. Solve the 2-by-2 linear equations for each pixel using this method:

   • If $$A=\left[\begin{array}{cc}a& b\\ b& c\end{array}\right]=\left[\begin{array}{cc}{\displaystyle \sum W^2{I}_x^2}& {\displaystyle \sum W^2{I}_x{I}_y}\\ {\displaystyle \sum W^2{I}_y{I}_x}& {\displaystyle \sum W^2{I}_y^2}\end{array}\right]$$

     then the eigenvalues of A are $${\lambda }_i=\frac{a+c}{2}\pm \frac{\sqrt{4b^2+{(a-c)}^2}}{2};i=1,2$$

   • When the block finds the eigenvalues, it compares them to the threshold, $$\tau$$, that corresponds to the value you enter for the Threshold for noise reduction parameter. The results fall into one of the following cases.

     Case 1: $${\lambda }_1\ge \tau$$ and $${\lambda }_2\ge \tau$$

     A is nonsingular, so the block solves the system of equations by using Cramer's rule.

     Case 2: $${\lambda }_1\ge \tau$$ and $${\lambda }_2<\tau$$

     A is singular (noninvertible), so the block normalizes the gradient flow to calculate u and v.

     Case 3: $${\lambda }_1<\tau$$ and $${\lambda }_2<\tau$$

     the optical flow, u and v, is 0.

Extended Capabilities

C/C++ Code Generation
Generate C and C++ code using Simulink® Coder™.

Fixed-Point Conversion
Design and simulate fixed-point systems using Fixed-Point Designer™.

Version History

Introduced before R2006a