UltraDet

This is the official implementation of Mining Negative Temporal Contexts For False Positive Suppression In Real-Time Ultrasound Lesion Detection. We provide a real-time breast ultrasound detection solution to reduce about 50% FPs at a recall rate of 0.90. Also, we provide high-quality bounding box labels for the CVA-Net breast ultrasound video dataset. Our paper has been early accepted by MICCAI 2023!

[Arxiv] [PDF] [Code]

Paper Abstract

During ultrasonic scanning processes, real-time lesion detection can assist radiologists in accurate cancer diagnosis. However, this essential task remains challenging and underexplored. General-purpose real-time object detection models can mistakenly report obvious false positives (FPs) when applied to ultrasound videos, potentially misleading junior radiologists. One key issue is their failure to utilize negative symptoms in previous frames, denoted as negative temporal contexts (NTC). To address this issue, we propose to extract contexts from previous frames, including NTC, with the guidance of inverse optical flow. By aggregating extracted contexts, we endow the model with the ability to suppress FPs by leveraging NTC. We call the resulting model UltraDet. The proposed UltraDet demonstrates significant improvement over previous state-of-the-arts and achieves real-time inference speed.

Experiment Results

The NTCA module leverages RoI-level NTC which are crucial for radiologists but ignored in previous works, thereby effectively improving the detection performance in a reliable and interpretable way. We plug the NTCA module into a basic real-time detector to form UltraDet. Experiments on CVA-BUS dataset demonstrate that UltraDet, with real-time inference speed, significantly outperforms previous works, reducing about 50% FPs at a recall rate of 0.90.

Quantitative results of real-time lesion detection on CVA-BUS.

Model	Type	Pr80	Pr90	FP80	FP90	AP50	R@16	FPS	Model Weights
One-Stage Detectors
YOLOX	Image	69.73.7	43.47.7	23.84.8	87.624.5	80.41.6	97.50.5	59.8	-
RetinaNet	Image	75.72.5	57.22.9	9.32.0	32.86.5	84.51.0	95.10.6	53.6	-
FCOS	Image	87.22.2	72.25.1	11.02.4	23.03.7	89.51.4	98.80.3	56.1	-
DeFCN	Image	81.51.8	67.52.3	21.13.2	33.44.3	86.41.3	99.30.3	51.2	-
Track-YOLO	Video	75.12.7	47.03.1	18.11.9	74.214.7	80.11.0	94.70.9	46.0	-
DETR-Based Detectors
DeformDETR	Image	90.13.2	72.710.6	5.62.2	37.820.9	90.52.0	98.70.3	33.8	-
TransVOD	Video	92.52.2	77.57.2	3.11.3	23.711.5	90.11.8	98.40.4	24.2	-
CVA-Net	Video	92.32.6	80.26.1	4.72.6	19.65.6	91.61.9	98.60.8	23.1	-
PTSEFormer	Video	93.31.9	85.46.0	2.81.1	12.59.8	91.51.6	97.91.2	9.1	-
FasterRCNN-Based Detectors
FasterRCNN	Image	91.30.9	75.23.6	6.91.4	34.46.7	88.01.4	92.41.0	49.2	-
RelationNet	Image	91.41.3	79.22.9	6.22.0	24.45.6	87.61.7	92.40.9	42.7	-
FGFA	Video	92.91.5	82.24.1	4.41.6	13.33.7	90.51.1	93.60.9	33.8	-
SELSA	Video	91.61.7	80.22.5	7.51.5	23.35.5	89.21.1	92.60.8	43.8	-
MEGA	Video	93.91.5	86.92.3	3.11.7	11.73.0	90.91.0	93.60.7	40.2	-
BasicDet (RDN)	Video	92.41.0	83.62.2	3.81.2	13.43.2	88.71.4	92.70.6	42.2	-
UltraDet (Ours)	Video	95.71.2	90.81.4	1.90.4	5.71.6	91.61.6	93.81.3	30.4	Link

Citing UltraDet

If you find UltraDet useful in your research, please cite our paper:

@article{yu2023mining,
  title={Mining Negative Temporal Contexts For False Positive Suppression In Real-Time Ultrasound Lesion Detection},
  author={Yu, Haojun and Li, Youcheng and Wu, QuanLin and Zhao, Ziwei and Chen, Dengbo and Wang, Dong and Wang, Liwei},
  journal={arXiv preprint arXiv:2305.18060},
  year={2023}
}

Requirments

The whole project is based on detectron2.

Installation

# install anaconda3 with python3.9
conda create -n <env_name> python=3.9
conda activate <env_name>

# use tuna source for pypi (optional)
python -m pip install --upgrade pip

# install packages
conda install pytorch==1.10.1 torchvision==0.11.2 torchaudio==0.10.1 cudatoolkit=11.3 -c pytorch -c conda-forge
python -m pip install detectron2 -f https://dl.fbaipublicfiles.com/detectron2/wheels/cu113/torch1.10/index.html
pip install pandas
pip install path
pip install sqlmodel
pip install coloredlogs
pip install opencv-p
pip install mmcv
pip install prettytable
pip install fire
pip install scipy
pip install timm

# install ultrasound_vid
git clone https://github.com/HaojunYu1998/UltraDet.git
cd UltraDet
pip install -e .

# issue shooting
# AttributeError: module 'distutils' has no attribute 'version'
pip uninstall setuptools
pip install setuptools==58.0.4

Checkpoints

We are pleased to share the weights of the models required for UltraDet inference. You can access them through the following links:

Weigths
flownet.ckpt
r34.pkl
ultradet.pth

After downloading the weights, please store them in the following path:

UltraDet/
  pretrained_models/
    flownet.ckpt
    r34.pkl
    ultradet.pth

Dataset Preparation

We provide high quality annotation files. You can access them through the following links:

Json File
test.json
trainval.json

Please download PNG images from CVA-Net and arrange the images and labels in the COCO format. If you encounter any data-related issues, please contact us via [email protected].

Note: The authors of CVA-Net dataset have made corrections to the benign/malignant classification annotations (please see CVA-Net). To use our high-quality annotations, please first correct the video and image location in rawframes/.

UltraDet/
  datasets/
    bus_data_cva/
      rawframes/
        benign/
          2cda21c3aab26332/
            000000.png
            000001.png
            ...
          ...
        malignant/
          1dc9ca2f1748c2ec/
            000000.png
            000001.png
            ...
          ...
      test.json
      trainval.json

Training and Inference

You can train or infer with UltraDet by running the following code:

# Training
python train_net.py --num-gpus {GPU_NUM} \
--config-file {CONFIG_FILES}

# Validation
python train_net.py --num-gpus {GPU_NUM} --eval-only --resume \
--config-file {CONFIG_FILES}

# Test
python train_net.py --num-gpus {GPU_NUM} --eval-only  --resume \
--config-file {CONFIG_FILES} \
DATASETS.SPLIT "test" \
DATASETS.NUM_FOLDS 1 \
# MODEL.WEIGHTS 'pretrained_models/ultradet.pth' # If you wish to load the pre-trained weights of UltraDet that we have released, please include this line of code