Skip to content

Latest commit

 

History

History
171 lines (162 loc) · 8.31 KB

README.md

File metadata and controls

171 lines (162 loc) · 8.31 KB
Raw

LiteFlowNet2

A Lightweight Optical Flow CNN - Revisiting Data Fidelity and Regularization

Abstract

Over four decades, the majority addresses the problem of optical flow estimation using variational methods. With the advance of machine learning, some recent works have attempted to address the problem using convolutional neural network (CNN) and have showed promising results. FlowNet2, the state-of-the-art CNN, requires over 160M parameters to achieve accurate flow estimation. Our LiteFlowNet2 outperforms FlowNet2 on Sintel and KITTI benchmarks, while being 25.3 times smaller in the model size and 3.1 times faster in the running speed. LiteFlowNet2 is built on the foundation laid by conventional methods and resembles the corresponding roles as data fidelity and regularization in variational methods. We compute optical flow in a spatial-pyramid formulation as SPyNet but through a novel lightweight cascaded flow inference. It provides high flow estimation accuracy through early correction with seamless incorporation of descriptor matching. Flow regularization is used to ameliorate the issue of outliers and vague flow boundaries through feature-driven local convolutions. Our network also owns an effective structure for pyramidal feature extraction and embraces feature warping rather than image warping as practiced in FlowNet2 and SPyNet. Comparing to LiteFlowNet, LiteFlowNet2 improves the optical flow accuracy on Sintel Clean by 23.3%, Sintel Final by 12.8%, KITTI 2012 by 19.6%, and KITTI 2015 by 18.8%, while being 2.2 times faster. Our network protocol and trained models are made publicly available on https://github.com/twhui/LiteFlowNet2.

Results and Models

Models Training datasets FlyingChairs Sintel (training) KITTI2012 (training) KITTI2015 (training) Log Config Download
clean final EPE Fl-all EPE
LiteFlowNet2-pre-M6S6 Flying Chairs 4.20 - - - - - log Config Model
LiteFlowNet2-pre-M6S6R6 Flying Chairs 3.94 - - - - - log Config Model
LiteFlowNet2-pre-M5S5R5 Flying Chairs 2.85 - - - - - log Config Model
LiteFlowNet2-pre-M4S4R4 Flying Chairs 2.07 - - - - - log Config Model
LiteFlowNet2-pre (LiteFlowNet2-pre-M3S3R3) Flying Chairs 1.57 2.78 4.24 5.79 39.42% 14.34 log Config Model
LiteFlowNet2 Flying Chairs + Flying Thing3d subset - 2.35 3.86 4.84 32.87% 12.07 log Config Model
LiteFlowNet2 Flying Chairs + Flying Thing3d subset + Sintel + KITTI - 1.32 1.65 - - - log Config Model
LiteFlowNet2 Flying Chairs + Flying Thing3d subset + KITTI - - - 0.89 4.31% 1.24 log Config Model

Citation

@article{hui2020lightweight,
  title={A lightweight optical flow CNN—Revisiting data fidelity and regularization},
  author={Hui, Tak-Wai and Tang, Xiaoou and Loy, Chen Change},
  journal={IEEE transactions on pattern analysis and machine intelligence},
  volume={43},
  number={8},
  pages={2555--2569},
  year={2020},
  publisher={IEEE}
}