Skip to content

Latest commit

 

History

History
100 lines (74 loc) · 4.83 KB

README.md

File metadata and controls

100 lines (74 loc) · 4.83 KB
Raw

TubeR: Tubelet Transformer for Video Action Detection

This repo contains the supported code to reproduce spatio-temporal action detection results of TubeR: Tubelet Transformer for Video Action Detection.

Updates

08/08/2022 Initial commits

Results and Models

AVA 2.1 Dataset

Backbone Pretrain #view mAP FLOPs config model
CSN-50 Kinetics-400 1 view 27.2 78G config S3
CSN-50 (with long-term context) Kinetics-400 1 view 28.8 78G config Comming soon
CSN-152 Kinetics-400+IG65M 1 view 29.7 120G config S3
CSN-152 (with long-term context) Kinetics-400+IG65M 1 view 31.7 120G config Comming soon

AVA 2.2 Dataset

Backbone Pretrain #view mAP FLOPs config model
CSN-152 Kinetics-400+IG65M 1 view 31.1 120G config S3
CSN-152 (with long-term context) Kinetics-400+IG65M 1 view 33.4 120G config Comming soon

JHMDB Dataset

Backbone #view [email protected] [email protected] config model
CSN-152 1 view 87.4 82.3 config S3

Usage

The project is developed based on GluonCV-torch. Please refer to tutorial for details.

Dependency

The project is tested working on:

  • Torch 1.12 + CUDA 11.3
  • timm==0.4.5
  • tensorboardX

Dataset

Please download the asset.zip and unzip them at ./datasets.

[AVA] Please refer to DATASET.md for AVA dataset downloading and pre-processing. [JHMDB] Please refer to JHMDB for JHMDB dataset and Dataset Section for UCF dataset. You also can refer to ACT-Detector to prepare the two datasets.

Inference

To run inference, first modify the config file:

  • set the correct WORLD_SIZE, GPU_WORLD_SIZE, DIST_URL, WOLRD_URLS based on experiment setup.
  • set the LABEL_PATH, ANNO_PATH, DATA_PATH to your local directory accordingly.
  • Download the pre-trained model and set PRETRAINED_PATH to model path.
  • make sure LOAD and LOAD_FC are set to True

Then run:

# run testing
python3  eval_tuber_ava.py <CONFIG_FILE> 

# for example, to evaluate ava from scratch, run:
python3 eval_tuber_ava.py configuration/TubeR_CSN152_AVA21.yaml

Training

To train TubeR from scratch, first modify the configfile:

  • set the correct WORLD_SIZE, GPU_WORLD_SIZE, DIST_URL, WOLRD_URLS based on experiment setup.
  • set the LABEL_PATH, ANNO_PATH, DATA_PATH to your local directory accordingly.
  • Download the pre-trained feature backbone and transformer weights and set PRETRAIN_BACKBONE_DIR (CSN50, CSN152), PRETRAIN_TRANSFORMER_DIR (DETR) accordingly.
  • make sure LOAD and LOAD_FC are set to False

Then run:

# run training from scratch
python3  train_tuber.py <CONFIG_FILE>

# for example, to train ava from scratch, run:
python3 train_tuber_ava.py configuration/TubeR_CSN152_AVA21.yaml

TODO

[ ]Add tutorial and pre-trained weights for TubeR with long-term memory

[ ]Add weights for UCF24

Citing TubeR

@inproceedings{zhao2022tuber,
  title={TubeR: Tubelet transformer for video action detection},
  author={Zhao, Jiaojiao and Zhang, Yanyi and Li, Xinyu and Chen, Hao and Shuai, Bing and Xu, Mingze and Liu, Chunhui and Kundu, Kaustav and Xiong, Yuanjun and Modolo, Davide and others},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={13598--13607},
  year={2022}
}