360_object_tracking

This is a package used for object detection, object tracking and overtaking behaviour detection on panoramic (360) videos of equirectangular projection, which is implemented according to Jingwei Guo's thesis.

YOLO v5 and Faster RCNN models pre-trained on COCO dataset are used as the detectors in this package. Projection transformation from equirectangular to perspective is realized using Perspective-and-Equirectangular and the implementation of DeepSORT was adapted from HERE.

Multiple Object Detection and Tracking in Panoramic Videos for Cycling Safety Analysis

By Jingwei Guo, Meihui Wang, Ilya Ilyankou, Natchapon Jongwiriyanurak, Xiaowei Gao, Nicola Christie, James Haworth

If you find the project useful in your research, please consider citing:

@misc{guo2024multipleobjectdetectiontracking,
      title={Multiple Object Detection and Tracking in Panoramic Videos for Cycling Safety Analysis},
      author={Jingwei Guo and Meihui Wang and Ilya Ilyankou and Natchapon Jongwiriyanurak and Xiaowei Gao and Nicola Christie and James Haworth},
      year={2024},
      eprint={2407.15199},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2407.15199},
}

Dependencies and Installation

The library should be run under Python 3.3+ with the following libraries installed:

detectron2 (version updated before Aug 5, 2022 only)

First, clone the repository:

git clone https://github.com/cuppp1998/360_object_tracking.git

To install all the dependencies (except Detectron2), run the following command in a new conda environment:

cd 360_object_tracking
pip install -r requirements.txt

Since in the new versions of Detectron2 (updated after Aug 5, 2022), some APIs have been modified, here we install an old version of it:

pip install -e git+https://github.com/facebookresearch/detectron2.git@5aeb252b194b93dc2879b4ac34bc51a31b5aee13#egg=detectron2

Download the pre-trained ReID network used in DeepSORT:

cd deep_sort/deep/checkpoint
pip install gdown
gdown 'https://drive.google.com/uc?export=download&id=1_qwTWdzT9dWNudpusgKavj_4elGgbkUN'
cd ../../../

Instruction of the Main Functionalities

The implementation process of each functionality is explained in detail in Code Explanation.ipynb.

360 Object Detection

To realize object detection on panoramic videos of equirectangular projection, execute Object_Detection.py in the Terminal as below:

python Object_Detection.py [--input_video_path INPUT_VIDEO_PATH] [--output_video_path OUTPUT_VIDEO_PATH] [--classes_to_detect CLASSES_TO_DETECT] [--FOV FOV] [--THETAs THETAS] [--PHIs PHIS] [--sub_image_width SUB_IMAGE_WIDTH] [--model_type MODEL_TYPE] [--score_threshold SCORE_THRESHOLD] [--nms_threshold NMS_THRESHOLD] [--use_mymodel USE_MYMODEL]

The following arguments are provided:

Argument	Description	Required?	Defaults
INPUT_VIDEO_PATH	Path of the input video	✔️
OUTPUT_VIDEO_PATH	Path of the output video	✔️
CLASSES_TO_DETECT	Index numbers of the categories to detect in the COCO dataset		[0, 1, 2, 3, 5, 7, 9]
FOV	Field of view of the sub images		120
THETAS	A list which contains the theta of each sub image (The length should be the same as the number of sub images)		[0, 90, 180, 270]
PHIS	A list which contains the Phi of each sub image (The length should be the same as the number of sub images)		[-10, -10, -10, -10]
SUB_IMAGE_WIDTH	Width (or height) of the sub images		640
MODEL_TYPE	A string that determines which detector to use ("YOLO" or "Faster RCNN")		"YOLO"
SCORE_THRESHOLD	The threshold of the confidence score		0.4
NMS_THRESHOLD	The threshold of the Non Maximum Suppression		0.45
USE_MYMODEL	A boolean value which determines whether to use the improved object detection model, if False, instead of being split into 4 parts, the image will be detected as a whole		True