SteadyFlow: Spatially Smooth Optical Flow for Video Stabilization

Shuaicheng Liu1         Lu yuan2         Ping Tan1         Jian Sun2

1. National University of Singapore               2. Microsoft Research

Abstract:

We propose a novel motion model, SteadyFlow, to represent the motion between neighboring video frames for stabilization. A SteadyFlow is a specific optical flow by enforcing strong spatial coherence, such that smoothing feature trajectories can be replaced by smoothing pixel profiles, which are motion vectors collected at the same pixel location in the SteadyFlow over time. In this way, we can avoid brittle feature tracking in a video stabilization system. Besides, SteadyFlow is a more general 2D motion model which can deal with spatially-variant motion. We initialize the SteadyFlow by optical flow and then discard discontinuous motions by a spatial-temporal analysis and fill in missing regions by motion completion. Our experiments demonstrate the effectiveness of our stabilization on real-world challenging videos.

 

Paper [PDF]

Related Projects

Shuaicheng Liu, Lu yuan, Ping Tan, Jian Sun. Bundled Camera Paths for Video Stabilization. ACM Transactions on Graphics (Proceeding of SIGGRAPH) 2013. [PDF][project page]

Shuaicheng Liu, Yinting Wang, Lu Yuan, Jiajun Bu, Ping Tan, Jian Sun: Video Stabilization with a Depth Camera. IEEE Conference on Computer Vision and Patten Recognition(CVPR) 2012 [PDF][project page]

 

Video Spotlight

Full demo Video: download [129M]

 

 

Downloads:

Example 1: Rolling shutter together with large occlusion

     
Input        Our method

Example 2: The synthesized example

      
Input        Liu et al 2013        Our method

Example 3: Two rolling shutter examples

     
Input
  Baker et al.2010    Our method
     
Input   Karpenko et al 2011    Our method

Example 4: Videos contain large foreground , moving towards camera

     
Input            Our method

     
Input        Our method

Example 5: Video contains quick camera zooming, comparison between with and without adaptive smoothing.

     
Input
          Our method
     
Without adaptive smoothing

Example 6: Stabilize by raw optical flow          

     
Input
           Our method
     
Raw optical flow result

Example 7: Motion completion by strong gaussian smoothing

     
Input
           Our method
     
Motion completion by gaussian smoothing

Example 8: more examples in the paper

     
Input            Our method
     

 

Limitations

Our spatial-temporal analysis failed to distinguish foreground and background when videos contain dominate foregrounds, (foregrounds occupy more than half area of a frame and exist for a long time).