Problems in training text detection models #1833

Yangget · 2023-04-04T03:53:53Z

Yangget
Apr 4, 2023

Hello, Community Developers.

I'm having some issues training my own text detection model. I would be very grateful if you could bring some advice.

The following is my process:
First, I changed the dataset to the form of the sample in the official document

{'metainfo': {'dataset_type': 'BenetechMakingGraphsAccessible',
  'task_name': 'textdet',
  'category': [{'id': 0, 'name': 'text'}]},
 'data_list': [{'img_path': '/home/.../**.jpg',
   'height': 523,
   'width': 320,
   'instances': [{'bbox': [45, 5, 411, 34],
     'bbox_label': 0,
     'polygon': [45, 5, 411, 5, 411, 31, 60, 34],
     'text': 'Rural population long-run future projection in Wallis and Futuna',
     'ignore': False}, ...],
...],
}

Then I modified the following configuration:

vim configs/textdet/dbnetpp/dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015.py

_base_ = [
    'dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015.py',
]

load_from = None

_base_.model.backbone = dict(
    type='CLIPResNet',
    init_cfg=dict(
        type='Pretrained',
        checkpoint='https://download.openmmlab.com/'
        'mmocr/backbone/resnet50-oclip-7ba0c533.pth'))

_base_.train_dataloader.num_workers = 16
_base_.optim_wrapper.optimizer.lr = 0.002

param_scheduler = [
    dict(type='LinearLR', end=200, start_factor=0.001),
    dict(type='PolyLR', power=0.9, eta_min=1e-7, begin=200, end=1200),
]

vim configs/textdet/dbnetpp/dbnetpp_resnet50-dcnv2_fpnc_1200e_icdar2015.py

_base_ = [
    '_base_dbnetpp_resnet50-dcnv2_fpnc.py',
    '../_base_/default_runtime.py',
    '../_base_/datasets/icdar2015.py',
    '../_base_/schedules/schedule_sgd_1200e.py',
]

load_from = 'https://download.openmmlab.com/mmocr/textdet/dbnetpp/tmp_1.0_pretrain/dbnetpp_r50dcnv2_fpnc_100k_iter_synthtext-20220502-352fec8a.pth'  # noqa

# dataset settings
train_list = [_base_.icdar2015_textdet_train]
test_list = [_base_.icdar2015_textdet_test]

train_dataloader = dict(
    batch_size=4,
    num_workers=8,
    persistent_workers=True,
    sampler=dict(type='DefaultSampler', shuffle=True),
    dataset=dict(
        type='ConcatDataset',
        datasets=train_list,
        pipeline=_base_.train_pipeline))

val_dataloader = dict(
    batch_size=4,
    num_workers=8,
    persistent_workers=True,
    sampler=dict(type='DefaultSampler', shuffle=False),
    dataset=dict(
        type='ConcatDataset',
        datasets=test_list,
        pipeline=_base_.test_pipeline))

test_dataloader = val_dataloader

auto_scale_lr = dict(base_batch_size=4)

vim configs/textdet/_base_/default_runtime.py

default_scope = 'mmocr'
env_cfg = dict(
    cudnn_benchmark=False,
    mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),
    dist_cfg=dict(backend='nccl'),
)
randomness = dict(seed=None)

default_hooks = dict(
    timer=dict(type='IterTimerHook'),
    logger=dict(type='LoggerHook', interval=1),
    param_scheduler=dict(type='ParamSchedulerHook'),
    checkpoint=dict(type='CheckpointHook', interval=2),
    sampler_seed=dict(type='DistSamplerSeedHook'),
    sync_buffer=dict(type='SyncBuffersHook'),
    visualization=dict(
        type='VisualizationHook',
        interval=1,
        enable=False,
        show=False,
        draw_gt=False,
        draw_pred=False),
)

# Logging
log_level = 'INFO'
log_processor = dict(type='LogProcessor', window_size=10, by_epoch=True)

load_from = None
resume = False

# Evaluation
val_evaluator = dict(type='HmeanIOUMetric')
test_evaluator = val_evaluator

# Visualization
vis_backends = [dict(type='LocalVisBackend')]
visualizer = dict(
    type='TextDetLocalVisualizer',
    name='visualizer',
    vis_backends=vis_backends)

Modify my data address very name

vim configs/textdet/_base_/datasets/icdar2015.py

icdar2015_textdet_data_root = '/home/mydata/'

icdar2015_textdet_train = dict(
    type='OCRDataset',
    data_root=icdar2015_textdet_data_root,
    ann_file='ocr_train.json',
    filter_cfg=dict(filter_empty_gt=True, min_size=16),
    pipeline=None)

icdar2015_textdet_test = dict(
    type='OCRDataset',
    data_root=icdar2015_textdet_data_root,
    ann_file='ocr_test.json',
    test_mode=True,
    pipeline=None)

In this configuration, I set epoch to 120

vim configs/textdet/_base_/schedules/schedule_sgd_1200e.py

optim_wrapper = dict(
    type='OptimWrapper',
    optimizer=dict(type='SGD', lr=0.007, momentum=0.9, weight_decay=0.0001))
train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=120, val_interval=1)
val_cfg = dict(type='ValLoop')
test_cfg = dict(type='TestLoop')
# learning policy
param_scheduler = [
    dict(type='PolyLR', power=0.9, eta_min=1e-7, end=120),
]

Finally, I ran the following command

CUDA_VISIBLE_DEVICES=2,3,4,5,6,7 PORT=29501 bash ./tools/dist_train.sh  configs/textdet/dbnetpp/dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015.py 6

The training result I got

2023/04/03 12:39:41 - mmengine - INFO - 
------------------------------------------------------------
System environment:
    sys.platform: linux
    Python: 3.7.10 (default, Feb 26 2021, 18:47:35) [GCC 7.3.0]
    CUDA available: True
    numpy_random_seed: 443960725
    GPU 0,1,2,3,4,5: NVIDIA GeForce RTX 3080
    CUDA_HOME: /usr/local/cuda
    NVCC: Cuda compilation tools, release 11.1, V11.1.105
    GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
    PyTorch: 1.9.0
    PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) oneAPI Math Kernel Library Version 2021.2-Product Build 20210312 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v2.1.2 (Git Hash 98be7e8afa711dc9b66c8ff3504129cb82013cdb)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_37,code=compute_37
  - CuDNN 8.0.5
  - Magma 2.5.2
  - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.1, CUDNN_VERSION=8.0.5, CXX_COMPILER=/opt/rh/devtoolset-7/root/usr/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=1.9.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

    TorchVision: 0.10.0
    OpenCV: 4.7.0
    MMEngine: 0.7.0

Runtime environment:
    cudnn_benchmark: False
    mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0}
    dist_cfg: {'backend': 'nccl'}
    seed: None
    Distributed launcher: pytorch
    Distributed training: True
    GPU number: 6
------------------------------------------------------------

2023/04/03 12:39:41 - mmengine - INFO - Config:
model = dict(
    type='DBNet',
    backbone=dict(
        type='CLIPResNet',
        init_cfg=dict(
            type='Pretrained',
            checkpoint=
            'https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth'
        )),
    neck=dict(
        type='FPNC',
        in_channels=[256, 512, 1024, 2048],
        lateral_channels=256,
        asf_cfg=dict(attention_type='ScaleChannelSpatial')),
    det_head=dict(
        type='DBHead',
        in_channels=256,
        module_loss=dict(type='DBModuleLoss'),
        postprocessor=dict(
            type='DBPostprocessor', text_repr_type='quad',
            epsilon_ratio=0.002)),
    data_preprocessor=dict(
        type='TextDetDataPreprocessor',
        mean=[123.675, 116.28, 103.53],
        std=[58.395, 57.12, 57.375],
        bgr_to_rgb=True,
        pad_size_divisor=32))
train_pipeline = [
    dict(type='LoadImageFromFile', color_type='color_ignore_orientation'),
    dict(
        type='LoadOCRAnnotations',
        with_bbox=True,
        with_polygon=True,
        with_label=True),
    dict(
        type='TorchVisionWrapper',
        op='ColorJitter',
        brightness=0.12549019607843137,
        saturation=0.5),
    dict(
        type='ImgAugWrapper',
        args=[['Fliplr', 0.5], {
            'cls': 'Affine',
            'rotate': [-10, 10]
        }, ['Resize', [0.5, 3.0]]]),
    dict(type='RandomCrop', min_side_ratio=0.1),
    dict(type='Resize', scale=(640, 640), keep_ratio=True),
    dict(type='Pad', size=(640, 640)),
    dict(
        type='PackTextDetInputs',
        meta_keys=('img_path', 'ori_shape', 'img_shape'))
]
test_pipeline = [
    dict(type='LoadImageFromFile', color_type='color_ignore_orientation'),
    dict(type='Resize', scale=(4068, 1024), keep_ratio=True),
    dict(
        type='LoadOCRAnnotations',
        with_polygon=True,
        with_bbox=True,
        with_label=True),
    dict(
        type='PackTextDetInputs',
        meta_keys=('img_path', 'ori_shape', 'img_shape', 'scale_factor',
                   'instances'))
]
default_scope = 'mmocr'
env_cfg = dict(
    cudnn_benchmark=False,
    mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),
    dist_cfg=dict(backend='nccl'))
randomness = dict(seed=None)
default_hooks = dict(
    timer=dict(type='IterTimerHook'),
    logger=dict(type='LoggerHook', interval=1),
    param_scheduler=dict(type='ParamSchedulerHook'),
    checkpoint=dict(type='CheckpointHook', interval=2),
    sampler_seed=dict(type='DistSamplerSeedHook'),
    sync_buffer=dict(type='SyncBuffersHook'),
    visualization=dict(
        type='VisualizationHook',
        interval=1,
        enable=False,
        show=False,
        draw_gt=False,
        draw_pred=False))
log_level = 'INFO'
log_processor = dict(type='LogProcessor', window_size=10, by_epoch=True)
load_from = None
resume = False
val_evaluator = dict(type='HmeanIOUMetric')
test_evaluator = dict(type='HmeanIOUMetric')
vis_backends = [dict(type='LocalVisBackend')]
visualizer = dict(
    type='TextDetLocalVisualizer',
    name='visualizer',
    vis_backends=[dict(type='LocalVisBackend')])
icdar2015_textdet_data_root = '/home/mydata/input/'
icdar2015_textdet_train = dict(
    type='OCRDataset',
    data_root='/home/mydata/input/',
    ann_file='ocr_train.json',
    filter_cfg=dict(filter_empty_gt=True, min_size=16),
    pipeline=None)
icdar2015_textdet_test = dict(
    type='OCRDataset',
    data_root='/home/mydata/input/',
    ann_file='ocr_test.json',
    test_mode=True,
    pipeline=None)
optim_wrapper = dict(
    type='OptimWrapper',
    optimizer=dict(type='SGD', lr=0.002, momentum=0.9, weight_decay=0.0001))
train_cfg = dict(type='EpochBasedTrainLoop', max_epochs=120, val_interval=1)
val_cfg = dict(type='ValLoop')
test_cfg = dict(type='TestLoop')
param_scheduler = [
    dict(type='LinearLR', end=200, start_factor=0.001),
    dict(type='PolyLR', power=0.9, eta_min=1e-07, begin=200, end=1200)
]
train_list = [
    dict(
        type='OCRDataset',
        data_root='/home/mydata/input/',
        ann_file='ocr_train.json',
        filter_cfg=dict(filter_empty_gt=True, min_size=16),
        pipeline=None)
]
test_list = [
    dict(
        type='OCRDataset',
        data_root='/home/mydata/input/',
        ann_file='ocr_test.json',
        test_mode=True,
        pipeline=None)
]
train_dataloader = dict(
    batch_size=4,
    num_workers=16,
    persistent_workers=True,
    sampler=dict(type='DefaultSampler', shuffle=True),
    dataset=dict(
        type='ConcatDataset',
        datasets=[
            dict(
                type='OCRDataset',
                data_root='/home/mydata/input/',
                ann_file='ocr_train.json',
                filter_cfg=dict(filter_empty_gt=True, min_size=16),
                pipeline=None)
        ],
        pipeline=[
            dict(
                type='LoadImageFromFile',
                color_type='color_ignore_orientation'),
            dict(
                type='LoadOCRAnnotations',
                with_bbox=True,
                with_polygon=True,
                with_label=True),
            dict(
                type='TorchVisionWrapper',
                op='ColorJitter',
                brightness=0.12549019607843137,
                saturation=0.5),
            dict(
                type='ImgAugWrapper',
                args=[['Fliplr', 0.5], {
                    'cls': 'Affine',
                    'rotate': [-10, 10]
                }, ['Resize', [0.5, 3.0]]]),
            dict(type='RandomCrop', min_side_ratio=0.1),
            dict(type='Resize', scale=(640, 640), keep_ratio=True),
            dict(type='Pad', size=(640, 640)),
            dict(
                type='PackTextDetInputs',
                meta_keys=('img_path', 'ori_shape', 'img_shape'))
        ]))
val_dataloader = dict(
    batch_size=4,
    num_workers=8,
    persistent_workers=True,
    sampler=dict(type='DefaultSampler', shuffle=False),
    dataset=dict(
        type='ConcatDataset',
        datasets=[
            dict(
                type='OCRDataset',
                data_root='/home/mydata/input/',
                ann_file='ocr_test.json',
                test_mode=True,
                pipeline=None)
        ],
        pipeline=[
            dict(
                type='LoadImageFromFile',
                color_type='color_ignore_orientation'),
            dict(type='Resize', scale=(4068, 1024), keep_ratio=True),
            dict(
                type='LoadOCRAnnotations',
                with_polygon=True,
                with_bbox=True,
                with_label=True),
            dict(
                type='PackTextDetInputs',
                meta_keys=('img_path', 'ori_shape', 'img_shape',
                           'scale_factor', 'instances'))
        ]))
test_dataloader = dict(
    batch_size=4,
    num_workers=8,
    persistent_workers=True,
    sampler=dict(type='DefaultSampler', shuffle=False),
    dataset=dict(
        type='ConcatDataset',
        datasets=[
            dict(
                type='OCRDataset',
                data_root='/home/mydata/input/',
                ann_file='ocr_test.json',
                test_mode=True,
                pipeline=None)
        ],
        pipeline=[
            dict(
                type='LoadImageFromFile',
                color_type='color_ignore_orientation'),
            dict(type='Resize', scale=(4068, 1024), keep_ratio=True),
            dict(
                type='LoadOCRAnnotations',
                with_polygon=True,
                with_bbox=True,
                with_label=True),
            dict(
                type='PackTextDetInputs',
                meta_keys=('img_path', 'ori_shape', 'img_shape',
                           'scale_factor', 'instances'))
        ]))
auto_scale_lr = dict(base_batch_size=4)
launcher = 'pytorch'
work_dir = './work_dirs/dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015'

2023/04/03 12:39:43 - mmengine - INFO - Hooks will be executed in the following order:
before_run:
(VERY_HIGH   ) RuntimeInfoHook                    
(BELOW_NORMAL) LoggerHook                         
 -------------------- 
before_train:
(VERY_HIGH   ) RuntimeInfoHook                    
(NORMAL      ) IterTimerHook                      
(VERY_LOW    ) CheckpointHook                     
 -------------------- 
before_train_epoch:
(VERY_HIGH   ) RuntimeInfoHook                    
(NORMAL      ) IterTimerHook                      
(NORMAL      ) DistSamplerSeedHook                
 -------------------- 
before_train_iter:
(VERY_HIGH   ) RuntimeInfoHook                    
(NORMAL      ) IterTimerHook                      
 -------------------- 
after_train_iter:
(VERY_HIGH   ) RuntimeInfoHook                    
(NORMAL      ) IterTimerHook                      
(BELOW_NORMAL) LoggerHook                         
(LOW         ) ParamSchedulerHook                 
(VERY_LOW    ) CheckpointHook                     
 -------------------- 
after_train_epoch:
(NORMAL      ) IterTimerHook                      
(NORMAL      ) SyncBuffersHook                    
(LOW         ) ParamSchedulerHook                 
(VERY_LOW    ) CheckpointHook                     
 -------------------- 
before_val_epoch:
(NORMAL      ) IterTimerHook                      
 -------------------- 
before_val_iter:
(NORMAL      ) IterTimerHook                      
 -------------------- 
after_val_iter:
(NORMAL      ) IterTimerHook                      
(NORMAL      ) VisualizationHook                  
(BELOW_NORMAL) LoggerHook                         
 -------------------- 
after_val_epoch:
(VERY_HIGH   ) RuntimeInfoHook                    
(NORMAL      ) IterTimerHook                      
(BELOW_NORMAL) LoggerHook                         
(LOW         ) ParamSchedulerHook                 
(VERY_LOW    ) CheckpointHook                     
 -------------------- 
before_test_epoch:
(NORMAL      ) IterTimerHook                      
 -------------------- 
before_test_iter:
(NORMAL      ) IterTimerHook                      
 -------------------- 
after_test_iter:
(NORMAL      ) IterTimerHook                      
(NORMAL      ) VisualizationHook                  
(BELOW_NORMAL) LoggerHook                         
 -------------------- 
after_test_epoch:
(VERY_HIGH   ) RuntimeInfoHook                    
(NORMAL      ) IterTimerHook                      
(BELOW_NORMAL) LoggerHook                         
 -------------------- 
after_run:
(BELOW_NORMAL) LoggerHook                         
 -------------------- 
2023/04/03 12:39:43 - mmengine - WARNING - "FileClient" will be deprecated in future. Please use io functions in https://mmengine.readthedocs.io/en/latest/api/fileio.html#file-io
2023/04/03 12:39:43 - mmengine - WARNING - "HardDiskBackend" is the alias of "LocalBackend" and the former will be deprecated in future.
2023/04/03 12:39:54 - mmengine - INFO - load model from: https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth
2023/04/03 12:39:54 - mmengine - INFO - Loads checkpoint by http backend from path: https://download.openmmlab.com/mmocr/backbone/resnet50-oclip-7ba0c533.pth
Name of parameter - Initialization information

....

The value is the same before and after calling `init_weights` of DBNet  
2023/04/03 12:39:56 - mmengine - INFO - Checkpoints will be saved to /mmocr/work_dirs/dbnetpp_resnet50-oclip_fpnc_1200e_icdar2015.
2023/04/03 12:40:00 - mmengine - INFO - Epoch(train)   [1][   1/2146]  lr: 2.0000e-06  eta: 11 days, 21:32:08  time: 3.9917  data_time: 2.1671  memory: 4524  loss: 17.5933  loss_prob: 13.7704  loss_thr: 2.8410  loss_db: 0.9819
2023/04/03 12:40:00 - mmengine - INFO - Epoch(train)   [1][   2/2146]  lr: 2.0000e-06  eta: 6 days, 14:32:17  time: 2.2163  data_time: 1.1019  memory: 4729  loss: 17.3339  loss_prob: 13.4220  loss_thr: 2.9319  loss_db: 0.9800
2023/04/03 12:40:01 - mmengine - INFO - Epoch(train)   [1][   3/2146]  lr: 2.0000e-06  eta: 4 days, 19:42:40  time: 1.6176  data_time: 0.7355  memory: 4729  loss: 16.4382  loss_prob: 12.5648  loss_thr: 2.9031  loss_db: 0.9703
2023/04/03 12:40:01 - mmengine - INFO - Epoch(train)   [1][   4/2146]  lr: 2.0000e-06  eta: 3 days, 21:24:28  time: 1.3058  data_time: 0.5555  memory: 4729  loss: 16.3932  loss_prob: 12.5339  loss_thr: 2.8904  loss_db: 0.9688
2023/04/03 12:40:02 - mmengine - INFO - Epoch(train)   [1][   5/2146]  lr: 2.0000e-06  eta: 3 days, 9:00:22  time: 1.1324  data_time: 0.4641  memory: 4729  loss: 16.7636  loss_prob: 12.8642  loss_thr: 2.9285  loss_db: 0.9709
2023/04/03 12:40:02 - mmengine - INFO - Epoch(train)   [1][   6/2146]  lr: 2.0000e-06  eta: 3 days, 0:18:45  time: 1.0109  data_time: 0.3872  memory: 4729  loss: 16.8796  loss_prob: 12.9825  loss_thr: 2.9252  loss_db: 0.9719
2023/04/03 12:40:02 - mmengine - INFO - Epoch(train)   [1][   7/2146]  lr: 2.0000e-06  eta: 2 days, 18:47:17  time: 0.9337  data_time: 0.3323  memory: 4729  loss: 16.9751  loss_prob: 13.0699  loss_thr: 2.9334  loss_db: 0.9718
2023/04/03 12:40:03 - mmengine - INFO - Epoch(train)   [1][   8/2146]  lr: 2.0000e-06  eta: 2 days, 13:50:06  time: 0.8645  data_time: 0.2911  memory: 4729  loss: 16.9655  loss_prob: 13.0505  loss_thr: 2.9429  loss_db: 0.9721
2023/04/03 12:40:03 - mmengine - INFO - Epoch(train)   [1][   9/2146]  lr: 2.0000e-06  eta: 2 days, 10:02:16  time: 0.8114  data_time: 0.2591  memory: 4729  loss: 17.0130  loss_prob: 13.0781  loss_thr: 2.9625  loss_db: 0.9724
2023/04/03 12:40:04 - mmengine - INFO - Epoch(train)   [1][  10/2146]  lr: 2.0000e-06  eta: 2 days, 7:17:47  time: 0.7730  data_time: 0.2334  memory: 4729  loss: 17.2032  loss_prob: 13.2335  loss_thr: 2.9959  loss_db: 0.9738
2023/04/03 12:40:04 - mmengine - INFO - Epoch(train)   [1][  11/2146]  lr: 2.0000e-06  eta: 2 days, 5:00:43  time: 0.4161  data_time: 0.0170  memory: 4729  loss: 17.2217  loss_prob: 13.2229  loss_thr: 3.0248  loss_db: 0.9740
2023/04/03 12:40:04 - mmengine - INFO - Epoch(train)   [1][  12/2146]  lr: 2.0000e-06  eta: 2 days, 2:47:12  time: 0.4087  data_time: 0.0136  memory: 4729  loss: 17.2021  loss_prob: 13.2003  loss_thr: 3.0279  loss_db: 0.9739
2023/04/03 12:40:05 - mmengine - INFO - Epoch(train)   [1][  13/2146]  lr: 2.0000e-06  eta: 2 days, 1:12:29  time: 0.4090  data_time: 0.0136  memory: 4729  loss: 17.3482  loss_prob: 13.3327  loss_thr: 3.0398  loss_db: 0.9757
2023/04/03 12:40:05 - mmengine - INFO - Epoch(train)   [1][  14/2146]  lr: 2.0000e-06  eta: 1 day, 23:34:30  time: 0.4088  data_time: 0.0124  memory: 4729  loss: 17.2537  loss_prob: 13.2340  loss_thr: 3.0445  loss_db: 0.9752
2023/04/03 12:40:06 - mmengine - INFO - Epoch(train)   [1][  15/2146]  lr: 2.0000e-06  eta: 1 day, 22:14:03  time: 0.4033  data_time: 0.0028  memory: 4729  loss: 17.0010  loss_prob: 13.0152  loss_thr: 3.0114  loss_db: 0.9743
2023/04/03 12:40:06 - mmengine - INFO - Epoch(train)   [1][  16/2146]  lr: 2.0000e-06  eta: 1 day, 21:15:37  time: 0.4059  data_time: 0.0064  memory: 4729  loss: 16.7903  loss_prob: 12.8051  loss_thr: 3.0123  loss_db: 0.9729
2023/04/03 12:40:06 - mmengine - INFO - Epoch(train)   [1][  17/2146]  lr: 2.0000e-06  eta: 1 day, 20:37:03  time: 0.4068  data_time: 0.0064  memory: 4729  loss: 16.4949  loss_prob: 12.5287  loss_thr: 2.9957  loss_db: 0.9704
2023/04/03 12:40:07 - mmengine - INFO - Epoch(train)   [1][  18/2146]  lr: 2.0000e-06  eta: 1 day, 19:50:29  time: 0.4117  data_time: 0.0101  memory: 4729  loss: 16.4251  loss_prob: 12.4622  loss_thr: 2.9924  loss_db: 0.9706
2023/04/03 12:40:07 - mmengine - INFO - Epoch(train)   [1][  19/2146]  lr: 2.0000e-06  eta: 1 day, 19:19:25  time: 0.4206  data_time: 0.0101  memory: 4729  loss: 16.1116  loss_prob: 12.2047  loss_thr: 2.9379  loss_db: 0.9690
2023/04/03 12:40:08 - mmengine - INFO - Epoch(train)   [1][  20/2146]  lr: 2.0000e-06  eta: 1 day, 18:34:51  time: 0.4176  data_time: 0.0112  memory: 4729  loss: 15.8800  loss_prob: 12.0030  loss_thr: 2.9089  loss_db: 0.9681
2023/04/03 12:40:08 - mmengine - INFO - Epoch(train)   [1][  21/2146]  lr: 2.0000e-06  eta: 1 day, 18:02:13  time: 0.4190  data_time: 0.0211  memory: 4729  loss: 15.5596  loss_prob: 11.7250  loss_thr: 2.8684  loss_db: 0.9662
2023/04/03 12:40:09 - mmengine - INFO - Epoch(train)   [1][  22/2146]  lr: 2.0000e-06  eta: 1 day, 17:26:42  time: 0.4227  data_time: 0.0211  memory: 4729  loss: 15.4810  loss_prob: 11.6443  loss_thr: 2.8705  loss_db: 0.9662
2023/04/03 12:40:09 - mmengine - INFO - Epoch(train)   [1][  23/2146]  lr: 2.0000e-06  eta: 1 day, 17:01:04  time: 0.4246  data_time: 0.0211  memory: 4729  loss: 15.4489  loss_prob: 11.6112  loss_thr: 2.8707  loss_db: 0.9670
2023/04/03 12:40:09 - mmengine - INFO - Epoch(train)   [1][  24/2146]  lr: 2.0000e-06  eta: 1 day, 16:27:45  time: 0.4265  data_time: 0.0211  memory: 4729  loss: 15.4141  loss_prob: 11.5909  loss_thr: 2.8550  loss_db: 0.9683
2023/04/03 12:40:10 - mmengine - INFO - Epoch(train)   [1][  25/2146]  lr: 2.0000e-06  eta: 1 day, 15:59:53  time: 0.4285  data_time: 0.0211  memory: 4729  loss: 15.3092  loss_prob: 11.4734  loss_thr: 2.8674  loss_db: 0.9684
2023/04/03 12:40:10 - mmengine - INFO - Epoch(train)   [1][  26/2146]  lr: 2.0000e-06  eta: 1 day, 15:38:22  time: 0.4285  data_time: 0.0175  memory: 4729  loss: 15.2319  loss_prob: 11.4158  loss_thr: 2.8467  loss_db: 0.9694
2023/04/03 12:40:11 - mmengine - INFO - Epoch(train)   [1][  27/2146]  lr: 2.0000e-06  eta: 1 day, 15:14:28  time: 0.4209  data_time: 0.0175  memory: 4729  loss: 15.2968  loss_prob: 11.4776  loss_thr: 2.8462  loss_db: 0.9730
2023/04/03 12:40:11 - mmengine - INFO - Epoch(train)   [1][  28/2146]  lr: 2.0000e-06  eta: 1 day, 14:50:42  time: 0.4174  data_time: 0.0138  memory: 4729  loss: 15.1300  loss_prob: 11.3416  loss_thr: 2.8156  loss_db: 0.9728
2023/04/03 12:40:11 - mmengine - INFO - Epoch(train)   [1][  29/2146]  lr: 2.0000e-06  eta: 1 day, 14:27:54  time: 0.4088  data_time: 0.0138  memory: 4729  loss: 15.0930  loss_prob: 11.3071  loss_thr: 2.8124  loss_db: 0.9735
2023/04/03 12:40:12 - mmengine - INFO - Epoch(train)   [1][  30/2146]  lr: 2.0000e-06  eta: 1 day, 14:09:02  time: 0.4096  data_time: 0.0128  memory: 4729  loss: 14.9176  loss_prob: 11.1351  loss_thr: 2.8091  loss_db: 0.9734
2023/04/03 12:40:12 - mmengine - INFO - Epoch(train)   [1][  31/2146]  lr: 2.0000e-06  eta: 1 day, 13:52:44  time: 0.4076  data_time: 0.0028  memory: 4729  loss: 14.8606  loss_prob: 11.0955  loss_thr: 2.7906  loss_db: 0.9746
2023/04/03 12:40:13 - mmengine - INFO - Epoch(train)   [1][  32/2146]  lr: 2.0000e-06  eta: 1 day, 13:41:27  time: 0.4115  data_time: 0.0061  memory: 4729  loss: 14.6614  loss_prob: 10.9282  loss_thr: 2.7597  loss_db: 0.9735
2023/04/03 12:40:13 - mmengine - INFO - Epoch(train)   [1][  33/2146]  lr: 2.0000e-06  eta: 1 day, 13:29:28  time: 0.4108  data_time: 0.0061  memory: 4729  loss: 14.4315  loss_prob: 10.7155  loss_thr: 2.7444  loss_db: 0.9716
2023/04/03 12:40:14 - mmengine - INFO - Epoch(train)   [1][  34/2146]  lr: 2.0000e-06  eta: 1 day, 13:23:06  time: 0.4195  data_time: 0.0092  memory: 4729  loss: 14.0829  loss_prob: 10.4022  loss_thr: 2.7145  loss_db: 0.9662
2023/04/03 12:40:14 - mmengine - INFO - Epoch(train)   [1][  35/2146]  lr: 2.0000e-06  eta: 1 day, 13:11:39  time: 0.4221  data_time: 0.0092  memory: 4729  loss: 14.0465  loss_prob: 10.3682  loss_thr: 2.7116  loss_db: 0.9667
2023/04/03 12:40:14 - mmengine - INFO - Epoch(train)   [1][  36/2146]  lr: 2.0000e-06  eta: 1 day, 12:54:07  time: 0.4165  data_time: 0.0101  memory: 4729  loss: 13.9724  loss_prob: 10.2957  loss_thr: 2.7099  loss_db: 0.9668
2023/04/03 12:40:15 - mmengine - INFO - Epoch(train)   [1][  37/2146]  lr: 2.0000e-06  eta: 1 day, 12:46:41  time: 0.4213  data_time: 0.0189  memory: 4729  loss: 13.8134  loss_prob: 10.1524  loss_thr: 2.6946  loss_db: 0.9664
2023/04/03 12:40:15 - mmengine - INFO - Epoch(train)   [1][  38/2146]  lr: 2.0000e-06  eta: 1 day, 12:36:29  time: 0.4243  data_time: 0.0188  memory: 4729  loss: 13.7975  loss_prob: 10.1175  loss_thr: 2.7131  loss_db: 0.9670
2023/04/03 12:40:16 - mmengine - INFO - Epoch(train)   [1][  39/2146]  lr: 2.0000e-06  eta: 1 day, 12:27:43  time: 0.4286  data_time: 0.0188  memory: 4729  loss: 13.6821  loss_prob: 10.0080  loss_thr: 2.7076  loss_db: 0.9665
2023/04/03 12:40:16 - mmengine - INFO - Epoch(train)   [1][  40/2146]  lr: 2.0000e-06  eta: 1 day, 12:20:35  time: 0.4324  data_time: 0.0189  memory: 4729  loss: 13.5460  loss_prob: 9.8949  loss_thr: 2.6852  loss_db: 0.9660
2023/04/03 12:40:17 - mmengine - INFO - Epoch(train)   [1][  41/2146]  lr: 2.0000e-06  eta: 1 day, 12:07:59  time: 0.4296  data_time: 0.0189  memory: 4729  loss: 13.4187  loss_prob: 9.7584  loss_thr: 2.6958  loss_db: 0.9645
2023/04/03 12:40:17 - mmengine - INFO - Epoch(train)   [1][  42/2146]  lr: 2.0000e-06  eta: 1 day, 11:56:19  time: 0.4242  data_time: 0.0156  memory: 4729  loss: 13.2822  loss_prob: 9.6402  loss_thr: 2.6779  loss_db: 0.9642
2023/04/03 12:40:17 - mmengine - INFO - Epoch(train)   [1][  43/2146]  lr: 2.0000e-06  eta: 1 day, 11:45:44  time: 0.4203  data_time: 0.0157  memory: 4729  loss: 13.2092  loss_prob: 9.5797  loss_thr: 2.6648  loss_db: 0.9646
2023/04/03 12:40:18 - mmengine - INFO - Epoch(train)   [1][  44/2146]  lr: 2.0000e-06  eta: 1 day, 11:36:35  time: 0.4136  data_time: 0.0126  memory: 4729  loss: 13.3232  loss_prob: 9.6949  loss_thr: 2.6588  loss_db: 0.9696
2023/04/03 12:40:18 - mmengine - INFO - Epoch(train)   [1][  45/2146]  lr: 2.0000e-06  eta: 1 day, 11:28:35  time: 0.4120  data_time: 0.0127  memory: 4729  loss: 13.1664  loss_prob: 9.5651  loss_thr: 2.6333  loss_db: 0.9680
2023/04/03 12:40:19 - mmengine - INFO - Epoch(train)   [1][  46/2146]  lr: 2.0000e-06  eta: 1 day, 11:23:56  time: 0.4194  data_time: 0.0117  memory: 4729  loss: 12.9890  loss_prob: 9.4148  loss_thr: 2.6083  loss_db: 0.9658
2023/04/03 12:40:19 - mmengine - INFO - Epoch(train)   [1][  47/2146]  lr: 2.0000e-06  eta: 1 day, 11:19:57  time: 0.4193  data_time: 0.0030  memory: 4729  loss: 12.9129  loss_prob: 9.3325  loss_thr: 2.6157  loss_db: 0.9647
2023/04/03 12:40:20 - mmengine - INFO - Epoch(train)   [1][  48/2146]  lr: 2.0000e-06  eta: 1 day, 11:15:23  time: 0.4212  data_time: 0.0069  memory: 4729  loss: 12.8005  loss_prob: 9.2502  loss_thr: 2.5854  loss_db: 0.9649
2023/04/03 12:40:20 - mmengine - INFO - Epoch(train)   [1][  49/2146]  lr: 2.0000e-06  eta: 1 day, 11:07:15  time: 0.4180  data_time: 0.0070  memory: 4729  loss: 12.9727  loss_prob: 9.3810  loss_thr: 2.6244  loss_db: 0.9672
2023/04/03 12:40:20 - mmengine - INFO - Epoch(train)   [1][  50/2146]  lr: 2.0000e-06  eta: 1 day, 11:01:37  time: 0.4162  data_time: 0.0103  memory: 4729  loss: 12.8787  loss_prob: 9.2965  loss_thr: 2.6164  loss_db: 0.9658
2023/04/03 12:40:21 - mmengine - INFO - Epoch(train)   [1][  51/2146]  lr: 2.0000e-06  eta: 1 day, 11:00:43  time: 0.4254  data_time: 0.0104  memory: 4729  loss: 12.8984  loss_prob: 9.3177  loss_thr: 2.6130  loss_db: 0.9678
2023/04/03 12:40:21 - mmengine - INFO - Epoch(train)   [1][  52/2146]  lr: 2.0000e-06  eta: 1 day, 10:52:33  time: 0.4253  data_time: 0.0113  memory: 4729  loss: 12.9383  loss_prob: 9.3330  loss_thr: 2.6362  loss_db: 0.9690
2023/04/03 12:40:22 - mmengine - INFO - Epoch(train)   [1][  53/2146]  lr: 2.0000e-06  eta: 1 day, 10:48:16  time: 0.4292  data_time: 0.0200  memory: 4729  loss: 12.8859  loss_prob: 9.3131  loss_thr: 2.6031  loss_db: 0.9698
2023/04/03 12:40:22 - mmengine - INFO - Epoch(train)   [1][  54/2146]  lr: 2.0000e-06  eta: 1 day, 10:40:57  time: 0.4280  data_time: 0.0199  memory: 4729  loss: 12.8684  loss_prob: 9.2851  loss_thr: 2.6124  loss_db: 0.9708
2023/04/03 12:40:22 - mmengine - INFO - Epoch(train)   [1][  55/2146]  lr: 2.0000e-06  eta: 1 day, 10:33:16  time: 0.4252  data_time: 0.0199  memory: 4729  loss: 12.7432  loss_prob: 9.1805  loss_thr: 2.5938  loss_db: 0.9689
2023/04/03 12:40:23 - mmengine - INFO - Epoch(train)   [1][  56/2146]  lr: 2.0000e-06  eta: 1 day, 10:31:06  time: 0.4261  data_time: 0.0199  memory: 4729  loss: 12.7754  loss_prob: 9.2021  loss_thr: 2.6025  loss_db: 0.9708
2023/04/03 12:40:23 - mmengine - INFO - Epoch(train)   [1][  57/2146]  lr: 2.0000e-06  eta: 1 day, 10:24:11  time: 0.4200  data_time: 0.0198  memory: 4729  loss: 12.6641  loss_prob: 9.1277  loss_thr: 2.5653  loss_db: 0.9711
2023/04/03 12:40:24 - mmengine - INFO - Epoch(train)   [1][  58/2146]  lr: 2.0000e-06  eta: 1 day, 10:15:21  time: 0.4119  data_time: 0.0159  memory: 4729  loss: 12.6623  loss_prob: 9.0972  loss_thr: 2.5939  loss_db: 0.9712
2023/04/03 12:40:24 - mmengine - INFO - Epoch(train)   [1][  59/2146]  lr: 2.0000e-06  eta: 1 day, 10:09:20  time: 0.4115  data_time: 0.0158  memory: 4729  loss: 12.4441  loss_prob: 8.9363  loss_thr: 2.5378  loss_db: 0.9699
2023/04/03 12:40:24 - mmengine - INFO - Epoch(train)   [1][  60/2146]  lr: 2.0000e-06  eta: 1 day, 10:04:18  time: 0.4097  data_time: 0.0124  memory: 4729  loss: 12.2178  loss_prob: 8.7688  loss_thr: 2.4842  loss_db: 0.9647
2023/04/03 12:40:25 - mmengine - INFO - Epoch(train)   [1][  61/2146]  lr: 2.0000e-06  eta: 1 day, 10:00:40  time: 0.4043  data_time: 0.0123  memory: 4729  loss: 12.1250  loss_prob: 8.6989  loss_thr: 2.4620  loss_db: 0.9640
2023/04/03 12:40:25 - mmengine - INFO - Epoch(train)   [1][  62/2146]  lr: 2.0000e-06  eta: 1 day, 9:56:36  time: 0.4069  data_time: 0.0113  memory: 4729  loss: 11.9947  loss_prob: 8.5927  loss_thr: 2.4392  loss_db: 0.9628
2023/04/03 12:40:26 - mmengine - INFO - Epoch(train)   [1][  63/2146]  lr: 2.0000e-06  eta: 1 day, 9:49:33  time: 0.4006  data_time: 0.0025  memory: 4729  loss: 11.9564  loss_prob: 8.5315  loss_thr: 2.4628  loss_db: 0.9621
2023/04/03 12:40:26 - mmengine - INFO - Epoch(train)   [1][  64/2146]  lr: 2.0000e-06  eta: 1 day, 9:46:49  time: 0.4043  data_time: 0.0055  memory: 4729  loss: 11.8296  loss_prob: 8.4270  loss_thr: 2.4445  loss_db: 0.9581
2023/04/03 12:40:27 - mmengine - INFO - Epoch(train)   [1][  65/2146]  lr: 2.0000e-06  eta: 1 day, 9:43:10  time: 0.4074  data_time: 0.0055  memory: 4729  loss: 11.7791  loss_prob: 8.3991  loss_thr: 2.4231  loss_db: 0.9569
2023/04/03 12:40:27 - mmengine - INFO - Epoch(train)   [1][  66/2146]  lr: 2.0000e-06  eta: 1 day, 9:39:06  time: 0.4028  data_time: 0.0089  memory: 4729  loss: 11.6379  loss_prob: 8.3104  loss_thr: 2.3714  loss_db: 0.9562
2023/04/03 12:40:27 - mmengine - INFO - Epoch(train)   [1][  67/2146]  lr: 2.0000e-06  eta: 1 day, 9:36:01  time: 0.4060  data_time: 0.0088  memory: 4729  loss: 11.5585  loss_prob: 8.2572  loss_thr: 2.3486  loss_db: 0.9527
2023/04/03 12:40:28 - mmengine - INFO - Epoch(train)   [1][  68/2146]  lr: 2.0000e-06  eta: 1 day, 9:31:32  time: 0.4097  data_time: 0.0098  memory: 4729  loss: 11.3556  loss_prob: 8.1077  loss_thr: 2.2979  loss_db: 0.9499
2023/04/03 12:40:28 - mmengine - INFO - Epoch(train)   [1][  69/2146]  lr: 2.0000e-06  eta: 1 day, 9:30:49  time: 0.4158  data_time: 0.0184  memory: 4729  loss: 11.2567  loss_prob: 8.0119  loss_thr: 2.2958  loss_db: 0.9489
2023/04/03 12:40:29 - mmengine - INFO - Epoch(train)   [1][  70/2146]  lr: 2.0000e-06  eta: 1 day, 9:27:18  time: 0.4162  data_time: 0.0184  memory: 4729  loss: 11.3519  loss_prob: 8.1033  loss_thr: 2.2933  loss_db: 0.9553
2023/04/03 12:40:29 - mmengine - INFO - Epoch(train)   [1][  71/2146]  lr: 2.0000e-06  eta: 1 day, 9:24:33  time: 0.4159  data_time: 0.0183  memory: 4729  loss: 11.2773  loss_prob: 8.0407  loss_thr: 2.2814  loss_db: 0.9552
2023/04/03 12:40:29 - mmengine - INFO - Epoch(train)   [1][  72/2146]  lr: 2.0000e-06  eta: 1 day, 9:21:26  time: 0.4158  data_time: 0.0185  memory: 4729  loss: 11.1929  loss_prob: 8.0003  loss_thr: 2.2385  loss_db: 0.9540
2023/04/03 12:40:30 - mmengine - INFO - Epoch(train)   [1][  73/2146]  lr: 2.0000e-06  eta: 1 day, 9:17:41  time: 0.4189  data_time: 0.0186  memory: 4729  loss: 11.1063  loss_prob: 7.9512  loss_thr: 2.2014  loss_db: 0.9536
2023/04/03 12:40:30 - mmengine - INFO - Epoch(train)   [1][  74/2146]  lr: 2.0000e-06  eta: 1 day, 9:13:02  time: 0.4142  data_time: 0.0155  memory: 4729  loss: 11.1547  loss_prob: 8.0131  loss_thr: 2.1848  loss_db: 0.9568
2023/04/03 12:40:31 - mmengine - INFO - Epoch(train)   [1][  75/2146]  lr: 2.0000e-06  eta: 1 day, 9:10:37  time: 0.4148  data_time: 0.0155  memory: 4729  loss: 11.1381  loss_prob: 7.9965  loss_thr: 2.1822  loss_db: 0.9595
2023/04/03 12:40:31 - mmengine - INFO - Epoch(train)   [1][  76/2146]  lr: 2.0000e-06  eta: 1 day, 9:08:01  time: 0.4156  data_time: 0.0122  memory: 4729  loss: 11.1209  loss_prob: 7.9685  loss_thr: 2.1923  loss_db: 0.9601
2023/04/03 12:40:31 - mmengine - INFO - Epoch(train)   [1][  77/2146]  lr: 2.0000e-06  eta: 1 day, 9:04:51  time: 0.4141  data_time: 0.0122  memory: 4729  loss: 11.0592  loss_prob: 7.9276  loss_thr: 2.1688  loss_db: 0.9627
2023/04/03 12:40:32 - mmengine - INFO - Epoch(train)   [1][  78/2146]  lr: 2.0000e-06  eta: 1 day, 9:02:50  time: 0.4167  data_time: 0.0113  memory: 4729  loss: 11.0494  loss_prob: 7.9486  loss_thr: 2.1376  loss_db: 0.9633
2023/04/03 12:40:32 - mmengine - INFO - Epoch(train)   [1][  79/2146]  lr: 2.0000e-06  eta: 1 day, 9:00:14  time: 0.4125  data_time: 0.0028  memory: 4729  loss: 10.9998  loss_prob: 7.8973  loss_thr: 2.1390  loss_db: 0.9635
2023/04/03 12:40:33 - mmengine - INFO - Epoch(train)   [1][  80/2146]  lr: 2.0000e-06  eta: 1 day, 9:00:31  time: 0.4180  data_time: 0.0059  memory: 4729  loss: 11.0148  loss_prob: 7.8958  loss_thr: 2.1544  loss_db: 0.9646
2023/04/03 12:40:33 - mmengine - INFO - Epoch(train)   [1][  81/2146]  lr: 2.0000e-06  eta: 1 day, 8:56:30  time: 0.4144  data_time: 0.0059  memory: 4729  loss: 10.9803  loss_prob: 7.8710  loss_thr: 2.1453  loss_db: 0.9640
2023/04/03 12:40:34 - mmengine - INFO - Epoch(train)   [1][  82/2146]  lr: 2.0000e-06  eta: 1 day, 8:54:06  time: 0.4143  data_time: 0.0087  memory: 4729  loss: 10.9563  loss_prob: 7.8802  loss_thr: 2.1103  loss_db: 0.9658
2023/04/03 12:40:34 - mmengine - INFO - Epoch(train)   [1][  83/2146]  lr: 2.0000e-06  eta: 1 day, 8:53:12  time: 0.4184  data_time: 0.0086  memory: 4729  loss: 10.9564  loss_prob: 7.8956  loss_thr: 2.0945  loss_db: 0.9663
2023/04/03 12:40:34 - mmengine - INFO - Epoch(train)   [1][  84/2146]  lr: 2.0000e-06  eta: 1 day, 8:51:02  time: 0.4215  data_time: 0.0097  memory: 4729  loss: 10.9008  loss_prob: 7.8309  loss_thr: 2.1034  loss_db: 0.9665
2023/04/03 12:40:35 - mmengine - INFO - Epoch(train)   [1][  85/2146]  lr: 2.0000e-06  eta: 1 day, 8:49:16  time: 0.4218  data_time: 0.0188  memory: 4729  loss: 10.9355  loss_prob: 7.8729  loss_thr: 2.0936  loss_db: 0.9689
2023/04/03 12:40:35 - mmengine - INFO - Epoch(train)   [1][  86/2146]  lr: 2.0000e-06  eta: 1 day, 8:46:08  time: 0.4196  data_time: 0.0189  memory: 4729  loss: 10.9433  loss_prob: 7.8846  loss_thr: 2.0896  loss_db: 0.9690
2023/04/03 12:40:36 - mmengine - INFO - Epoch(train)   [1][  87/2146]  lr: 2.0000e-06  eta: 1 day, 8:46:10  time: 0.4248  data_time: 0.0190  memory: 4729  loss: 10.9661  loss_prob: 7.8864  loss_thr: 2.1098  loss_db: 0.9699
2023/04/03 12:40:36 - mmengine - INFO - Epoch(train)   [1][  88/2146]  lr: 2.0000e-06  eta: 1 day, 8:42:54  time: 0.4214  data_time: 0.0189  memory: 4729  loss: 10.9301  loss_prob: 7.8477  loss_thr: 2.1111  loss_db: 0.9714
2023/04/03 12:40:37 - mmengine - INFO - Epoch(train)   [1][  89/2146]  lr: 2.0000e-06  eta: 1 day, 8:39:10  time: 0.4180  data_time: 0.0190  memory: 4729  loss: 10.9093  loss_prob: 7.8581  loss_thr: 2.0784  loss_db: 0.9728
2023/04/03 12:40:37 - mmengine - INFO - Epoch(train)   [1][  90/2146]  lr: 2.0000e-06  eta: 1 day, 8:35:38  time: 0.4096  data_time: 0.0160  memory: 4729  loss: 10.8432  loss_prob: 7.8419  loss_thr: 2.0309  loss_db: 0.9704
2023/04/03 12:40:37 - mmengine - INFO - Epoch(train)   [1][  91/2146]  lr: 2.0000e-06  eta: 1 day, 8:33:05  time: 0.4112  data_time: 0.0160  memory: 4729  loss: 10.7790  loss_prob: 7.8067  loss_thr: 2.0023  loss_db: 0.9700
2023/04/03 12:40:38 - mmengine - INFO - Epoch(train)   [1][  92/2146]  lr: 2.0000e-06  eta: 1 day, 8:32:03  time: 0.4130  data_time: 0.0131  memory: 4729  loss: 10.7232  loss_prob: 7.7358  loss_thr: 2.0183  loss_db: 0.9691
2023/04/03 12:40:38 - mmengine - INFO - Epoch(train)   [1][  93/2146]  lr: 2.0000e-06  eta: 1 day, 8:28:42  time: 0.4069  data_time: 0.0132  memory: 4729  loss: 10.6728  loss_prob: 7.6975  loss_thr: 2.0050  loss_db: 0.9703
2023/04/03 12:40:39 - mmengine - INFO - Epoch(train)   [1][  94/2146]  lr: 2.0000e-06  eta: 1 day, 8:27:29  time: 0.4080  data_time: 0.0123  memory: 4729  loss: 10.5795  loss_prob: 7.6398  loss_thr: 1.9703  loss_db: 0.9695
2023/04/03 12:40:39 - mmengine - INFO - Epoch(train)   [1][  95/2146]  lr: 2.0000e-06  eta: 1 day, 8:25:54  time: 0.4074  data_time: 0.0032  memory: 4729  loss: 10.5540  loss_prob: 7.6395  loss_thr: 1.9465  loss_db: 0.9680
2023/04/03 12:40:39 - mmengine - INFO - Epoch(train)   [1][  96/2146]  lr: 2.0000e-06  eta: 1 day, 8:26:36  time: 0.4147  data_time: 0.0063  memory: 4729  loss: 10.4830  loss_prob: 7.5942  loss_thr: 1.9205  loss_db: 0.9683
2023/04/03 12:40:40 - mmengine - INFO - Epoch(train)   [1][  97/2146]  lr: 2.0000e-06  eta: 1 day, 8:23:56  time: 0.4082  data_time: 0.0063  memory: 4729  loss: 10.4198  loss_prob: 7.5419  loss_thr: 1.9110  loss_db: 0.9669
2023/04/03 12:40:40 - mmengine - INFO - Epoch(train)   [1][  98/2146]  lr: 2.0000e-06  eta: 1 day, 8:24:11  time: 0.4149  data_time: 0.0094  memory: 4729  loss: 10.3274  loss_prob: 7.4778  loss_thr: 1.8859  loss_db: 0.9637
2023/04/03 12:40:41 - mmengine - INFO - Epoch(train)   [1][  99/2146]  lr: 2.0000e-06  eta: 1 day, 8:23:05  time: 0.4197  data_time: 0.0094  memory: 4729  loss: 10.2970  loss_prob: 7.4469  loss_thr: 1.8870  loss_db: 0.9631
2023/04/03 12:40:41 - mmengine - INFO - Epoch(train)   [1][ 100/2146]  lr: 2.0000e-06  eta: 1 day, 8:20:46  time: 0.4213  data_time: 0.0106  memory: 4729  loss: 10.1804  loss_prob: 7.3332  loss_thr: 1.8839  loss_db: 0.9633
2023/04/03 12:40:42 - mmengine - INFO - Epoch(train)   [1][ 101/2146]  lr: 2.0000e-06  eta: 1 day, 8:20:59  time: 0.4269  data_time: 0.0193  memory: 4729  loss: 10.2295  loss_prob: 7.3564  loss_thr: 1.9093  loss_db: 0.9638
2023/04/03 12:40:42 - mmengine - INFO - Epoch(train)   [1][ 102/2146]  lr: 2.0000e-06  eta: 1 day, 8:19:57  time: 0.4264  data_time: 0.0194  memory: 4729  loss: 10.2731  loss_prob: 7.3994  loss_thr: 1.9082  loss_db: 0.9655
2023/04/03 12:40:42 - mmengine - INFO - Epoch(train)   [1][ 103/2146]  lr: 2.0000e-06  eta: 1 day, 8:19:04  time: 0.4313  data_time: 0.0193  memory: 4729  loss: 10.2373  loss_prob: 7.3787  loss_thr: 1.8930  loss_db: 0.9657
2023/04/03 12:40:43 - mmengine - INFO - Epoch(train)   [1][ 104/2146]  lr: 2.0000e-06  eta: 1 day, 8:17:52  time: 0.4308  data_time: 0.0192  memory: 4729  loss: 10.2699  loss_prob: 7.3944  loss_thr: 1.9083  loss_db: 0.9671
2023/04/03 12:40:43 - mmengine - INFO - Epoch(train)   [1][ 105/2146]  lr: 2.0000e-06  eta: 1 day, 8:15:03  time: 0.4272  data_time: 0.0193  memory: 4729  loss: 10.1564  loss_prob: 7.2926  loss_thr: 1.8974  loss_db: 0.9663
2023/04/03 12:40:44 - mmengine - INFO - Epoch(train)   [1][ 106/2146]  lr: 2.0000e-06  eta: 1 day, 8:11:59  time: 0.4178  data_time: 0.0162  memory: 4729  loss: 10.1113  loss_prob: 7.2528  loss_thr: 1.8934  loss_db: 0.9651
2023/04/03 12:40:44 - mmengine - INFO - Epoch(train)   [1][ 107/2146]  lr: 2.0000e-06  eta: 1 day, 8:09:40  time: 0.4177  data_time: 0.0163  memory: 4729  loss: 10.1137  loss_prob: 7.2795  loss_thr: 1.8670  loss_db: 0.9672
2023/04/03 12:40:44 - mmengine - INFO - Epoch(train)   [1][ 108/2146]  lr: 2.0000e-06  eta: 1 day, 8:09:22  time: 0.4160  data_time: 0.0131  memory: 4729  loss: 10.1562  loss_prob: 7.2960  loss_thr: 1.8900  loss_db: 0.9702
2023/04/03 12:40:45 - mmengine - INFO - Epoch(train)   [1][ 109/2146]  lr: 2.0000e-06  eta: 1 day, 8:08:25  time: 0.4158  data_time: 0.0131  memory: 4729  loss: 10.1491  loss_prob: 7.2953  loss_thr: 1.8825  loss_db: 0.9713
2023/04/03 12:40:45 - mmengine - INFO - Epoch(train)   [1][ 110/2146]  lr: 2.0000e-06  eta: 1 day, 8:06:57  time: 0.4171  data_time: 0.0120  memory: 4729  loss: 10.1426  loss_prob: 7.2886  loss_thr: 1.8824  loss_db: 0.9716
2023/04/03 12:40:46 - mmengine - INFO - Epoch(train)   [1][ 111/2146]  lr: 2.0000e-06  eta: 1 day, 8:06:16  time: 0.4145  data_time: 0.0032  memory: 4729  loss: 10.0196  loss_prob: 7.1944  loss_thr: 1.8541  loss_db: 0.9712
2023/04/03 12:40:46 - mmengine - INFO - Epoch(train)   [1][ 112/2146]  lr: 2.0000e-06  eta: 1 day, 8:05:03  time: 0.4135  data_time: 0.0064  memory: 4729  loss: 9.8994  loss_prob: 7.1003  loss_thr: 1.8299  loss_db: 0.9692
2023/04/03 12:40:47 - mmengine - INFO - Epoch(train)   [1][ 113/2146]  lr: 2.0000e-06  eta: 1 day, 8:03:43  time: 0.4117  data_time: 0.0064  memory: 4729  loss: 9.8417  loss_prob: 7.0491  loss_thr: 1.8278  loss_db: 0.9649
2023/04/03 12:40:47 - mmengine - INFO - Epoch(train)   [1][ 114/2146]  lr: 2.0000e-06  eta: 1 day, 8:03:58  time: 0.4150  data_time: 0.0098  memory: 4729  loss: 9.7635  loss_prob: 6.9991  loss_thr: 1.8012  loss_db: 0.9632
2023/04/03 12:40:47 - mmengine - INFO - Epoch(train)   [1][ 115/2146]  lr: 2.0000e-06  eta: 1 day, 8:04:47  time: 0.4237  data_time: 0.0097  memory: 4729  loss: 9.7603  loss_prob: 6.9842  loss_thr: 1.8111  loss_db: 0.9649
2023/04/03 12:40:48 - mmengine - INFO - Epoch(train)   [1][ 116/2146]  lr: 2.0000e-06  eta: 1 day, 8:03:47  time: 0.4283  data_time: 0.0108  memory: 4729  loss: 9.7419  loss_prob: 6.9702  loss_thr: 1.8059  loss_db: 0.9658
2023/04/03 12:40:48 - mmengine - INFO - Epoch(train)   [1][ 117/2146]  lr: 2.0000e-06  eta: 1 day, 8:04:12  time: 0.4351  data_time: 0.0203  memory: 4729  loss: 9.6580  loss_prob: 6.8948  loss_thr: 1.7998  loss_db: 0.9634
2023/04/03 12:40:49 - mmengine - INFO - Epoch(train)   [1][ 118/2146]  lr: 2.0000e-06  eta: 1 day, 8:02:41  time: 0.4316  data_time: 0.0205  memory: 4729  loss: 9.5742  loss_prob: 6.8412  loss_thr: 1.7708  loss_db: 0.9622
2023/04/03 12:40:49 - mmengine - INFO - Epoch(train)   [1][ 119/2146]  lr: 2.0000e-06  eta: 1 day, 8:00:46  time: 0.4285  data_time: 0.0207  memory: 4729  loss: 9.5284  loss_prob: 6.8022  loss_thr: 1.7642  loss_db: 0.9621
2023/04/03 12:40:50 - mmengine - INFO - Epoch(train)   [1][ 120/2146]  lr: 2.0000e-06  eta: 1 day, 8:00:34  time: 0.4315  data_time: 0.0206  memory: 4729  loss: 9.5000  loss_prob: 6.7777  loss_thr: 1.7604  loss_db: 0.9619
2023/04/03 12:40:50 - mmengine - INFO - Epoch(train)   [1][ 121/2146]  lr: 2.0000e-06  eta: 1 day, 8:00:42  time: 0.4335  data_time: 0.0206  memory: 4729  loss: 9.4826  loss_prob: 6.7794  loss_thr: 1.7405  loss_db: 0.9626
2023/04/03 12:40:50 - mmengine - INFO - Epoch(train)   [1][ 122/2146]  lr: 2.0000e-06  eta: 1 day, 7:59:44  time: 0.4338  data_time: 0.0174  memory: 4729  loss: 9.4809  loss_prob: 6.7689  loss_thr: 1.7487  loss_db: 0.9634
2023/04/03 12:40:51 - mmengine - INFO - Epoch(train)   [1][ 123/2146]  lr: 2.0000e-06  eta: 1 day, 7:58:29  time: 0.4336  data_time: 0.0174  memory: 4729  loss: 9.4759  loss_prob: 6.7664  loss_thr: 1.7433  loss_db: 0.9662
2023/04/03 12:40:51 - mmengine - INFO - Epoch(train)   [1][ 124/2146]  lr: 2.0000e-06  eta: 1 day, 7:59:01  time: 0.4344  data_time: 0.0141  memory: 4729  loss: 9.4177  loss_prob: 6.7158  loss_thr: 1.7353  loss_db: 0.9666
2023/04/03 12:40:52 - mmengine - INFO - Epoch(train)   [1][ 125/2146]  lr: 2.0000e-06  eta: 1 day, 7:57:02  time: 0.4263  data_time: 0.0141  memory: 4729  loss: 9.3776  loss_prob: 6.6948  loss_thr: 1.7181  loss_db: 0.9647
2023/04/03 12:40:52 - mmengine - INFO - Epoch(train)   [1][ 126/2146]  lr: 2.0000e-06  eta: 1 day, 7:54:57  time: 0.4227  data_time: 0.0129  memory: 4729  loss: 9.4035  loss_prob: 6.7016  loss_thr: 1.7361  loss_db: 0.9659
2023/04/03 12:40:53 - mmengine - INFO - Epoch(train)   [1][ 127/2146]  lr: 2.0000e-06  eta: 1 day, 7:53:22  time: 0.4167  data_time: 0.0034  memory: 4729  loss: 9.4632  loss_prob: 6.7387  loss_thr: 1.7557  loss_db: 0.9687
2023/04/03 12:40:53 - mmengine - INFO - Epoch(train)   [1][ 128/2146]  lr: 2.0000e-06  eta: 1 day, 7:53:15  time: 0.4202  data_time: 0.0062  memory: 4729  loss: 9.4293  loss_prob: 6.7043  loss_thr: 1.7567  loss_db: 0.9683
2023/04/03 12:40:53 - mmengine - INFO - Epoch(train)   [1][ 129/2146]  lr: 2.0000e-06  eta: 1 day, 7:51:32  time: 0.4202  data_time: 0.0060  memory: 4729  loss: 9.3665  loss_prob: 6.6585  loss_thr: 1.7415  loss_db: 0.9664
2023/04/03 12:40:54 - mmengine - INFO - Epoch(train)   [1][ 130/2146]  lr: 2.0000e-06  eta: 1 day, 7:51:44  time: 0.4211  data_time: 0.0094  memory: 4729  loss: 9.4149  loss_prob: 6.6903  loss_thr: 1.7557  loss_db: 0.9688
2023/04/03 12:40:54 - mmengine - INFO - Epoch(train)   [1][ 131/2146]  lr: 2.0000e-06  eta: 1 day, 7:51:25  time: 0.4196  data_time: 0.0094  memory: 4729  loss: 9.4259  loss_prob: 6.6916  loss_thr: 1.7645  loss_db: 0.9699
2023/04/03 12:40:55 - mmengine - INFO - Epoch(train)   [1][ 132/2146]  lr: 2.0000e-06  eta: 1 day, 7:49:30  time: 0.4162  data_time: 0.0104  memory: 4729  loss: 9.4076  loss_prob: 6.6850  loss_thr: 1.7520  loss_db: 0.9706
2023/04/03 12:40:55 - mmengine - INFO - Epoch(train)   [1][ 133/2146]  lr: 2.0000e-06  eta: 1 day, 7:48:45  time: 0.4173  data_time: 0.0193  memory: 4729  loss: 9.4262  loss_prob: 6.6918  loss_thr: 1.7619  loss_db: 0.9725
2023/04/03 12:40:55 - mmengine - INFO - Epoch(train)   [1][ 134/2146]  lr: 2.0000e-06  eta: 1 day, 7:46:43  time: 0.4092  data_time: 0.0192  memory: 4729  loss: 9.4225  loss_prob: 6.6883  loss_thr: 1.7619  loss_db: 0.9723
2023/04/03 12:40:56 - mmengine - INFO - Epoch(train)   [1][ 135/2146]  lr: 2.0000e-06  eta: 1 day, 7:44:53  time: 0.4089  data_time: 0.0193  memory: 4729  loss: 9.4250  loss_prob: 6.6858  loss_thr: 1.7646  loss_db: 0.9745
2023/04/03 12:40:56 - mmengine - INFO - Epoch(train)   [1][ 136/2146]  lr: 2.0000e-06  eta: 1 day, 7:43:58  time: 0.4118  data_time: 0.0194  memory: 4729  loss: 9.3692  loss_prob: 6.6500  loss_thr: 1.7452  loss_db: 0.9741
2023/04/03 12:40:57 - mmengine - INFO - Epoch(train)   [1][ 137/2146]  lr: 2.0000e-06  eta: 1 day, 7:42:00  time: 0.4100  data_time: 0.0194  memory: 4729  loss: 9.3628  loss_prob: 6.6459  loss_thr: 1.7436  loss_db: 0.9734
2023/04/03 12:40:57 - mmengine - INFO - Epoch(train)   [1][ 138/2146]  lr: 2.0000e-06  eta: 1 day, 7:39:51  time: 0.4031  data_time: 0.0163  memory: 4729  loss: 9.3663  loss_prob: 6.6669  loss_thr: 1.7259  loss_db: 0.9736
2023/04/03 12:40:57 - mmengine - INFO - Epoch(train)   [1][ 139/2146]  lr: 2.0000e-06  eta: 1 day, 7:38:07  time: 0.4023  data_time: 0.0163  memory: 4729  loss: 9.3507  loss_prob: 6.6666  loss_thr: 1.7125  loss_db: 0.9716
2023/04/03 12:40:58 - mmengine - INFO - Epoch(train)   [1][ 140/2146]  lr: 2.0000e-06  eta: 1 day, 7:39:08  time: 0.4048  data_time: 0.0130  memory: 4729  loss: 9.2661  loss_prob: 6.6038  loss_thr: 1.6946  loss_db: 0.9676
2023/04/03 12:40:58 - mmengine - INFO - Epoch(train)   [1][ 141/2146]  lr: 2.0000e-06  eta: 1 day, 7:37:24  time: 0.3997  data_time: 0.0130  memory: 4729  loss: 9.2862  loss_prob: 6.6207  loss_thr: 1.6981  loss_db: 0.9674
2023/04/03 12:40:59 - mmengine - INFO - Epoch(train)   [1][ 142/2146]  lr: 2.0000e-06  eta: 1 day, 7:36:48  time: 0.4033  data_time: 0.0121  memory: 4729  loss: 9.3188  loss_prob: 6.6611  loss_thr: 1.6891  loss_db: 0.9686
2023/04/03 12:40:59 - mmengine - INFO - Epoch(train)   [1][ 143/2146]  lr: 2.0000e-06  eta: 1 day, 7:37:52  time: 0.4089  data_time: 0.0031  memory: 4729  loss: 9.2572  loss_prob: 6.6089  loss_thr: 1.6805  loss_db: 0.9679
2023/04/03 12:41:00 - mmengine - INFO - Epoch(train)   [1][ 144/2146]  lr: 2.0000e-06  eta: 1 day, 7:38:16  time: 0.4163  data_time: 0.0063  memory: 4729  loss: 9.2416  loss_prob: 6.6038  loss_thr: 1.6699  loss_db: 0.9679
2023/04/03 12:41:00 - mmengine - INFO - Epoch(train)   [1][ 145/2146]  lr: 2.0000e-06  eta: 1 day, 7:37:20  time: 0.4187  data_time: 0.0063  memory: 4729  loss: 9.1938  loss_prob: 6.5624  loss_thr: 1.6649  loss_db: 0.9665
2023/04/03 12:41:00 - mmengine - INFO - Epoch(train)   [1][ 146/2146]  lr: 2.0000e-06  eta: 1 day, 7:36:58  time: 0.4203  data_time: 0.0090  memory: 4729  loss: 9.1709  loss_prob: 6.5440  loss_thr: 1.6605  loss_db: 0.9664
2023/04/03 12:41:01 - mmengine - INFO - Epoch(train)   [1][ 147/2146]  lr: 2.0000e-06  eta: 1 day, 7:37:09  time: 0.4270  data_time: 0.0090  memory: 4729  loss: 9.1146  loss_prob: 6.5122  loss_thr: 1.6356  loss_db: 0.9669
2023/04/03 12:41:01 - mmengine - INFO - Epoch(train)   [1][ 148/2146]  lr: 2.0000e-06  eta: 1 day, 7:36:15  time: 0.4307  data_time: 0.0103  memory: 4729  loss: 9.0995  loss_prob: 6.4870  loss_thr: 1.6447  loss_db: 0.9679
2023/04/03 12:41:02 - mmengine - INFO - Epoch(train)   [1][ 149/2146]  lr: 2.0000e-06  eta: 1 day, 7:37:09  time: 0.4394  data_time: 0.0200  memory: 4729  loss: 9.1005  loss_prob: 6.4714  loss_thr: 1.6586  loss_db: 0.9706
2023/04/03 12:41:02 - mmengine - INFO - Epoch(train)   [1][ 150/2146]  lr: 2.0000e-06  eta: 1 day, 7:35:58  time: 0.4319  data_time: 0.0201  memory: 4729  loss: 9.1144  loss_prob: 6.4851  loss_thr: 1.6555  loss_db: 0.9737
2023/04/03 12:41:03 - mmengine - INFO - Epoch(train)   [1][ 151/2146]  lr: 2.0000e-06  eta: 1 day, 7:34:56  time: 0.4339  data_time: 0.0200  memory: 4729  loss: 9.0403  loss_prob: 6.4323  loss_thr: 1.6362  loss_db: 0.9717
2023/04/03 12:41:03 - mmengine - INFO - Epoch(train)   [1][ 152/2146]  lr: 2.0000e-06  eta: 1 day, 7:34:14  time: 0.4333  data_time: 0.0199  memory: 4729  loss: 8.9559  loss_prob: 6.3408  loss_thr: 1.6461  loss_db: 0.9690
2023/04/03 12:41:03 - mmengine - INFO - Epoch(train)   [1][ 153/2146]  lr: 2.0000e-06  eta: 1 day, 7:33:02  time: 0.4254  data_time: 0.0199  memory: 4729  loss: 8.9449  loss_prob: 6.3393  loss_thr: 1.6385  loss_db: 0.9670
2023/04/03 12:41:04 - mmengine - INFO - Epoch(train)   [1][ 154/2146]  lr: 2.0000e-06  eta: 1 day, 7:31:07  time: 0.4172  data_time: 0.0167  memory: 4729  loss: 8.9735  loss_prob: 6.3574  loss_thr: 1.6484  loss_db: 0.9677
2023/04/03 12:41:04 - mmengine - INFO - Epoch(train)   [1][ 155/2146]  lr: 2.0000e-06  eta: 1 day, 7:28:39  time: 0.4113  data_time: 0.0166  memory: 4729  loss: 8.9884  loss_prob: 6.3638  loss_thr: 1.6559  loss_db: 0.9687
2023/04/03 12:41:05 - mmengine - INFO - Epoch(train)   [1][ 156/2146]  lr: 2.0000e-06  eta: 1 day, 7:28:04  time: 0.4101  data_time: 0.0138  memory: 4729  loss: 8.9511  loss_prob: 6.3352  loss_thr: 1.6487  loss_db: 0.9672
2023/04/03 12:41:05 - mmengine - INFO - Epoch(train)   [1][ 157/2146]  lr: 2.0000e-06  eta: 1 day, 7:26:08  time: 0.4022  data_time: 0.0139  memory: 4729  loss: 8.9695  loss_prob: 6.3537  loss_thr: 1.6480  loss_db: 0.9678
2023/04/03 12:41:05 - mmengine - INFO - Epoch(train)   [1][ 158/2146]  lr: 2.0000e-06  eta: 1 day, 7:24:53  time: 0.4005  data_time: 0.0126  memory: 4729  loss: 8.9181  loss_prob: 6.3210  loss_thr: 1.6324  loss_db: 0.9647
2023/04/03 12:41:06 - mmengine - INFO - Epoch(train)   [1][ 159/2146]  lr: 2.0000e-06  eta: 1 day, 7:25:04  time: 0.3978  data_time: 0.0029  memory: 4729  loss: 8.8569  loss_prob: 6.2827  loss_thr: 1.6111  loss_db: 0.9631
2023/04/03 12:41:06 - mmengine - INFO - Epoch(train)   [1][ 160/2146]  lr: 2.0000e-06  eta: 1 day, 7:24:36  time: 0.3999  data_time: 0.0062  memory: 4729  loss: 8.8093  loss_prob: 6.2442  loss_thr: 1.6040  loss_db: 0.9612
2023/04/03 12:41:07 - mmengine - INFO - Epoch(train)   [1][ 161/2146]  lr: 2.0000e-06  eta: 1 day, 7:24:05  time: 0.4013  data_time: 0.0062  memory: 4729  loss: 8.7648  loss_prob: 6.2092  loss_thr: 1.5942  loss_db: 0.9614
2023/04/03 12:41:07 - mmengine - INFO - Epoch(train)   [1][ 162/2146]  lr: 2.0000e-06  eta: 1 day, 7:22:58  time: 0.3993  data_time: 0.0096  memory: 4729  loss: 8.7570  loss_prob: 6.2151  loss_thr: 1.5791  loss_db: 0.9627
2023/04/03 12:41:07 - mmengine - INFO - Epoch(train)   [1][ 163/2146]  lr: 2.0000e-06  eta: 1 day, 7:22:46  time: 0.4026  data_time: 0.0095  memory: 4729  loss: 8.7302  loss_prob: 6.1945  loss_thr: 1.5707  loss_db: 0.9650
2023/04/03 12:41:08 - mmengine - INFO - Epoch(train)   [1][ 164/2146]  lr: 2.0000e-06  eta: 1 day, 7:23:09  time: 0.4107  data_time: 0.0109  memory: 4729  loss: 8.7112  loss_prob: 6.1786  loss_thr: 1.5661  loss_db: 0.9665
2023/04/03 12:41:08 - mmengine - INFO - Epoch(train)   [1][ 165/2146]  lr: 2.0000e-06  eta: 1 day, 7:22:57  time: 0.4186  data_time: 0.0206  memory: 4729  loss: 8.7201  loss_prob: 6.1832  loss_thr: 1.5706  loss_db: 0.9662
2023/04/03 12:41:09 - mmengine - INFO - Epoch(train)   [1][ 166/2146]  lr: 2.0000e-06  eta: 1 day, 7:21:48  time: 0.4162  data_time: 0.0205  memory: 4729  loss: 8.7041  loss_prob: 6.1803  loss_thr: 1.5592  loss_db: 0.9646
2023/04/03 12:41:09 - mmengine - INFO - Epoch(train)   [1][ 167/2146]  lr: 2.0000e-06  eta: 1 day, 7:21:23  time: 0.4215  data_time: 0.0205  memory: 4729  loss: 8.6293  loss_prob: 6.1121  loss_thr: 1.5547  loss_db: 0.9625
2023/04/03 12:41:10 - mmengine - INFO - Epoch(train)   [1][ 168/2146]  lr: 2.0000e-06  eta: 1 day, 7:21:48  time: 0.4276  data_time: 0.0206  memory: 4729  loss: 8.6508  loss_prob: 6.1242  loss_thr: 1.5605  loss_db: 0.9661
2023/04/03 12:41:10 - mmengine - INFO - Epoch(train)   [1][ 169/2146]  lr: 2.0000e-06  eta: 1 day, 7:20:59  time: 0.4236  data_time: 0.0206  memory: 4729  loss: 8.6461  loss_prob: 6.1175  loss_thr: 1.5623  loss_db: 0.9663
2023/04/03 12:41:10 - mmengine - INFO - Epoch(train)   [1][ 170/2146]  lr: 2.0000e-06  eta: 1 day, 7:20:09  time: 0.4220  data_time: 0.0172  memory: 4729  loss: 8.6377  loss_prob: 6.1142  loss_thr: 1.5576  loss_db: 0.9659
2023/04/03 12:41:11 - mmengine - INFO - Epoch(train)   [1][ 171/2146]  lr: 2.0000e-06  eta: 1 day, 7:18:51  time: 0.4187  data_time: 0.0172  memory: 4729  loss: 8.6535  loss_prob: 6.1173  loss_thr: 1.5693  loss_db: 0.9670
2023/04/03 12:41:11 - mmengine - INFO - Epoch(train)   [1][ 172/2146]  lr: 2.0000e-06  eta: 1 day, 7:18:01  time: 0.4195  data_time: 0.0138  memory: 4729  loss: 8.6183  loss_prob: 6.0906  loss_thr: 1.5640  loss_db: 0.9638
2023/04/03 12:41:12 - mmengine - INFO - Epoch(train)   [1][ 173/2146]  lr: 2.0000e-06  eta: 1 day, 7:17:05  time: 0.4163  data_time: 0.0138  memory: 4729  loss: 8.5988  loss_prob: 6.0791  loss_thr: 1.5567  loss_db: 0.9631
2023/04/03 12:41:12 - mmengine - INFO - Epoch(train)   [1][ 174/2146]  lr: 2.0000e-06  eta: 1 day, 7:17:13  time: 0.4152  data_time: 0.0125  memory: 4729  loss: 8.5560  loss_prob: 6.0533  loss_thr: 1.5400  loss_db: 0.9628
2023/04/03 12:41:12 - mmengine - INFO - Epoch(train)   [1][ 175/2146]  lr: 2.0000e-06  eta: 1 day, 7:16:27  time: 0.4128  data_time: 0.0028  memory: 4729  loss: 8.5022  loss_prob: 6.0315  loss_thr: 1.5068  loss_db: 0.9639
2023/04/03 12:41:13 - mmengine - INFO - Epoch(train)   [1][ 176/2146]  lr: 2.0000e-06  eta: 1 day, 7:16:58  time: 0.4192  data_time: 0.0065  memory: 4729  loss: 8.6022  loss_prob: 6.1161  loss_thr: 1.5184  loss_db: 0.9677
2023/04/03 12:41:13 - mmengine - INFO - Epoch(train)   [1][ 177/2146]  lr: 2.0000e-06  eta: 1 day, 7:15:50  time: 0.4160  data_time: 0.0065  memory: 4729  loss: 8.5943  loss_prob: 6.1160  loss_thr: 1.5116  loss_db: 0.9668
2023/04/03 12:41:14 - mmengine - INFO - Epoch(train)   [1][ 178/2146]  lr: 2.0000e-06  eta: 1 day, 7:15:57  time: 0.4148  data_time: 0.0099  memory: 4729  loss: 8.6682  loss_prob: 6.1755  loss_thr: 1.5253  loss_db: 0.9674
2023/04/03 12:41:14 - mmengine - INFO - Epoch(train)   [1][ 179/2146]  lr: 2.0000e-06  eta: 1 day, 7:14:40  time: 0.4125  data_time: 0.0099  memory: 4729  loss: 8.6627  loss_prob: 6.1783  loss_thr: 1.5159  loss_db: 0.9685
2023/04/03 12:41:14 - mmengine - INFO - Epoch(train)   [1][ 180/2146]  lr: 2.0000e-06  eta: 1 day, 7:13:06  time: 0.4091  data_time: 0.0110  memory: 4729  loss: 8.6732  loss_prob: 6.1890  loss_thr: 1.5153  loss_db: 0.9689
2023/04/03 12:41:15 - mmengine - INFO - Epoch(train)   [1][ 181/2146]  lr: 2.0000e-06  eta: 1 day, 7:13:36  time: 0.4161  data_time: 0.0201  memory: 4729  loss: 8.6802  loss_prob: 6.1989  loss_thr: 1.5117  loss_db: 0.9696
2023/04/03 12:41:15 - mmengine - INFO - Epoch(train)   [1][ 182/2146]  lr: 2.0000e-06  eta: 1 day, 7:12:47  time: 0.4159  data_time: 0.0201  memory: 4729  loss: 8.7047  loss_prob: 6.2095  loss_thr: 1.5223  loss_db: 0.9729
2023/04/03 12:41:16 - mmengine - INFO - Epoch(train)   [1][ 183/2146]  lr: 2.0000e-06  eta: 1 day, 7:12:55  time: 0.4202  data_time: 0.0202  memory: 4729  loss: 8.6953  loss_prob: 6.2105  loss_thr: 1.5139  loss_db: 0.9709
2023/04/03 12:4

... ...

2023/04/03 13:18:42 - mmengine - INFO - Epoch(val)   [1][379/379]    eta: 0:00:00  time: 3.6606  data_time: 0.0070  memory: 3077  
2023/04/03 13:19:05 - mmengine - INFO - Evaluating hmean-iou...
2023/04/03 13:19:06 - mmengine - INFO - prediction score threshold: 0.30, recall: 0.0000, precision: 0.0000, hmean: 0.0000

2023/04/03 13:19:06 - mmengine - INFO - prediction score threshold: 0.40, recall: 0.0000, precision: 0.0000, hmean: 0.0000

2023/04/03 13:19:06 - mmengine - INFO - prediction score threshold: 0.50, recall: 0.0000, precision: 0.0000, hmean: 0.0000

2023/04/03 13:19:06 - mmengine - INFO - prediction score threshold: 0.60, recall: 0.0000, precision: 0.0000, hmean: 0.0000

2023/04/03 13:19:06 - mmengine - INFO - prediction score threshold: 0.70, recall: 0.0000, precision: 0.0000, hmean: 0.0000

2023/04/03 13:19:06 - mmengine - INFO - prediction score threshold: 0.80, recall: 0.0000, precision: 0.0000, hmean: 0.0000

2023/04/03 13:19:06 - mmengine - INFO - prediction score threshold: 0.90, recall: 0.0000, precision: 0.0000, hmean: 0.0000

2023/04/03 13:19:06 - mmengine - INFO - Epoch(val) [1][379/379]  icdar/precision: 0.0000  icdar/recall: 0.0000  icdar/hmean: 0.0000data_time: 0.0070  time: 3.6606  
2023/04/03 13:19:07 - mmengine - INFO - Epoch(train)   [2][   1/2146]  lr: 1.2040e-05  eta: 1 day, 5:20:09  time: 0.4551  data_time: 0.0813  memory: 4713  loss: 5.4965  loss_prob: 3.3455  loss_thr: 1.1780  loss_db: 0.9730
2023/04/03 13:19:07 - mmengine - INFO - Epoch(train)   [2][   2/2146]  lr: 1.2040e-05  eta: 1 day, 5:20:09  time: 0.4568  data_time: 0.0834  memory: 4714  loss: 5.4838  loss_prob: 3.3388  loss_thr: 1.1753  loss_db: 0.9698
2023/04/03 13:19:08 - mmengine - INFO - Epoch(train)   [2][   3/2146]  lr: 1.2040e-05  eta: 1 day, 5:20:07  time: 0.4579  data_time: 0.0834  memory: 4714  loss: 5.4796  loss_prob: 3.3375  loss_thr: 1.1724  loss_db: 0.9698
2023/04/03 13:19:08 - mmengine - INFO - Epoch(train)   [2][   4/2146]  lr: 1.2040e-05  eta: 1 day, 5:20:01  time: 0.4574  data_time: 0.0834  memory: 4714  loss: 5.5357  loss_prob: 3.3867  loss_thr: 1.1779  loss_db: 0.9711
2023/04/03 13:19:09 - mmengine - INFO - Epoch(train)   [2][   5/2146]  lr: 1.2040e-05  eta: 1 day, 5:20:07  time: 0.4657  data_time: 0.0858  memory: 4714  loss: 5.5565  loss_prob: 3.4017  loss_thr: 1.1798  

... ...

2023/04/04 00:41:52 - mmengine - INFO - Epoch(val)  [42][376/379]    eta: 0:00:00  time: 0.2575  data_time: 0.0168  memory: 3969  
2023/04/04 00:41:53 - mmengine - INFO - Epoch(val)  [42][377/379]    eta: 0:00:00  time: 0.2586  data_time: 0.0180  memory: 3969  
2023/04/04 00:41:53 - mmengine - INFO - Epoch(val)  [42][378/379]    eta: 0:00:00  time: 0.2558  data_time: 0.0153  memory: 3969  
2023/04/04 00:41:53 - mmengine - INFO - Epoch(val)  [42][379/379]    eta: 0:00:00  time: 0.2424  data_time: 0.0071  memory: 3077  
2023/04/04 00:41:53 - mmengine - INFO - Evaluating hmean-iou...
2023/04/04 00:41:54 - mmengine - INFO - prediction score threshold: 0.30, recall: 0.0000, precision: 0.0000, hmean: 0.0000

2023/04/04 00:41:54 - mmengine - INFO - prediction score threshold: 0.40, recall: 0.0000, precision: 0.0000, hmean: 0.0000

2023/04/04 00:41:54 - mmengine - INFO - prediction score threshold: 0.50, recall: 0.0000, precision: 0.0000, hmean: 0.0000

2023/04/04 00:41:54 - mmengine - INFO - prediction score threshold: 0.60, recall: 0.0000, precision: 0.0000, hmean: 0.0000

2023/04/04 00:41:54 - mmengine - INFO - prediction score threshold: 0.70, recall: 0.0000, precision: 0.0000, hmean: 0.0000

2023/04/04 00:41:54 - mmengine - INFO - prediction score threshold: 0.80, recall: 0.0000, precision: 0.0000, hmean: 0.0000

2023/04/04 00:41:54 - mmengine - INFO - prediction score threshold: 0.90, recall: 0.0000, precision: 0.0000, hmean: 0.0000

2023/04/04 00:41:54 - mmengine - INFO - Epoch(val) [42][379/379]  icdar/precision: 0.0000  icdar/recall: 0.0000  icdar/hmean: 0.0000data_time: 0.0071  time: 0.2424  
2023/04/04 00:41:55 - mmengine - INFO - Epoch(train)  [43][   1/2146]  lr: 4.2369e-04  eta: 19:21:58  time: 0.4715  data_time: 0.0841  memory: 4714  loss: 4.8211  loss_prob: 2.8363  loss_thr: 1.1119  loss_db: 0.8729
2023/04/04 00:41:56 - mmengine - INFO - Epoch(train)  [43][   2/2146]  lr: 4.2369e-04  eta: 19:21:58  time: 0.4790  data_time: 0.0841  memory: 4714  loss: 4.8279  loss_prob: 2.8348  loss_thr: 1.1182  loss_db: 0.8748
2023/04/04 00:41:56 - mmengine - INFO - Epoch(train)  [43][   3/2146]  lr: 4.2369e-04  eta: 19:21:57  time: 0.4796  data_time: 0.0839  memory: 4714  loss: 4.8142  loss_prob: 2.8237  loss_thr: 1.1149  loss_db: 0.8755
2023/04/04 00:41:57 - mmengine - INFO - Epoch(train)  [43][   4/2146]  lr: 4.2369e-04  eta: 19:21:57  time: 0.4845  data_time: 0.0839  memory: 4714  loss: 4.8804  loss_prob: 2.8405  loss_thr: 1.1279  loss_db: 0.9120
2023/04/04 00:41:57 - mmengine - INFO - Epoch(train)  [43][   5/2146]  lr: 4.2369e-04  eta: 19:21:57  time: 0.4907  data_time: 0.0839  memory: 4714  loss: 4.9374  loss_prob: 2.8516  loss_thr: 1.1418  loss_db: 0.9440

Why my recall, precision and hmean are always 0, my training data and test data are in the same format.

gaotongxiao · 2023-04-04T05:05:17Z

gaotongxiao
Apr 4, 2023
Maintainer

Everything seems to be right, and the loss keeps decreasing over time but way too slowly. A general advice would be to try a larger learning rate.

3 replies

Yangget Apr 4, 2023
Author

thanks.
I'll try a larger learning rate first.

Yangget Apr 4, 2023
Author

If I want to train for 100 epochs, is this learning rate change appropriate?

_base_.optim_wrapper.optimizer.lr = 0.002
param_scheduler = [
    # learning rate scheduler
    # During the first 10 epochs, learning rate increases from 0 to lr * 10
    # during the next 90 epochs, learning rate decreases from lr * 10 to lr * 1e-4
    dict(
        type='CosineAnnealingLR',
        T_max=10,
        eta_min=_base_.optim_wrapper.optimizer.lr * 10,
        begin=0,
        end=10,
        by_epoch=True,
        convert_to_iter_based=True),
    dict(
        type='CosineAnnealingLR',
        T_max=90,
        eta_min=_base_.optim_wrapper.optimizer.lr * 1e-4,
        begin=10,
        end=100,
        by_epoch=True,
        convert_to_iter_based=True),
    # momentum scheduler
    # During the first 10 epochs, momentum increases from 0 to 0.85 / 0.95
    # during the next 90 epochs, momentum increases from 0.85 / 0.95 to 1
    dict(
        type='CosineAnnealingMomentum',
        T_max=10,
        eta_min=0.85 / 0.95,
        begin=0,
        end=10,
        by_epoch=True,
        convert_to_iter_based=True),
    dict(
        type='CosineAnnealingMomentum',
        T_max=90,
        eta_min=1,
        begin=10,
        end=100,
        by_epoch=True,
        convert_to_iter_based=True)
]

gaotongxiao Apr 4, 2023
Maintainer

Nobody can tell whether a hyperparameter is "appropriate" without experiment results... But if you want to know whether such a schedule works as expected, you can adapt this script to mmocr for the visualization.

Yangget · 2023-04-04T06:37:02Z

Yangget
Apr 4, 2023
Author

I have another question

The bbox in the figure refers to [x, y, w, h] (x, y are the coordinates of the upper left corner, w, h are the width and height of the box), or [x1, y1, x2, y2] (x1, y1 is the coordinates of the upper left corner, x1, y1 are the coordinates of the lower right corner), which one?

1 reply

gaotongxiao Apr 4, 2023
Maintainer

The second one

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Problems in training text detection models #1833

{{title}}

Replies: 2 comments 4 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

Select a reply

Problems in training text detection models #1833

Yangget Apr 4, 2023

Replies: 2 comments · 4 replies

gaotongxiao Apr 4, 2023 Maintainer

Yangget Apr 4, 2023 Author

Yangget Apr 4, 2023 Author

gaotongxiao Apr 4, 2023 Maintainer

Yangget Apr 4, 2023 Author

gaotongxiao Apr 4, 2023 Maintainer

Yangget
Apr 4, 2023

Replies: 2 comments 4 replies

gaotongxiao
Apr 4, 2023
Maintainer

Yangget Apr 4, 2023
Author

Yangget Apr 4, 2023
Author

gaotongxiao Apr 4, 2023
Maintainer

Yangget
Apr 4, 2023
Author

gaotongxiao Apr 4, 2023
Maintainer