paddleOCR 识别，内存一直上涨，只要进程不杀死，内存就不会下降

[Yins11]

🔎 Search before asking

• I have searched the PaddleOCR Docs and found no similar bug report.
• I have searched the PaddleOCR Issues and found no similar bug report.
• I have searched the PaddleOCR Discussions and found no similar bug report.

🐛 Bug (问题描述)

python tools/infer/predict_rec.py
--rec_model_dir=./inference/rec_ppocr_v4/
--image_dir="xxxxxxxxxxxxxx"
--rec_algorithm="SVTR_LCNet"
--rec_image_shape="3, 48, 320"
--rec_batch_num=1
--rec_char_dict_path="./ppocr/utils/en_dict.txt"
--benchmark=True
--use_gpu=True
--use_tensorrt=True
--warmup=True
运行命令如上，服务器运行 内存一直涨，直到挂掉,
paddlecor: v2.8.1
tensorrt: 8.6.1.6
python: 3.10
paddle: v2.6.1
这个问题只出现在使用tensorrt推理时。请问到底是什么原因

🏃‍♂️ Environment (运行环境)

linux

🌰 Minimal Reproducible Example (最小可复现问题的Demo)

使用paddleocr v2.8.1代码训练-导出-推理，使用tensorrt推理出现上述问题，推理参数设置：
def init_args():
parser = argparse.ArgumentParser()
# params for prediction engine
parser.add_argument("--use_gpu", type=str2bool, default=True)
parser.add_argument("--use_xpu", type=str2bool, default=False)
parser.add_argument("--use_npu", type=str2bool, default=False)
parser.add_argument("--use_mlu", type=str2bool, default=False)
parser.add_argument("--ir_optim", type=str2bool, default=True)
parser.add_argument("--use_tensorrt", type=str2bool, default=True)
parser.add_argument("--min_subgraph_size", type=int, default=15)
parser.add_argument("--precision", type=str, default="fp32")
parser.add_argument("--gpu_mem", type=int, default=500)
parser.add_argument("--gpu_id", type=int, default=0)
parser.add_argument("--use_openvino", type=str2bool, default=False)

# params for text detector
parser.add_argument("--image_dir", type=str)
parser.add_argument("--page_num", type=int, default=0)
parser.add_argument("--det_algorithm", type=str, default="DB")
parser.add_argument("--det_model_dir", type=str)
parser.add_argument("--det_limit_side_len", type=float, default=960)
parser.add_argument("--det_limit_type", type=str, default="max")
parser.add_argument("--det_box_type", type=str, default="quad")

# DB parmas
parser.add_argument("--det_db_thresh", type=float, default=0.3)
parser.add_argument("--det_db_box_thresh", type=float, default=0.6)
parser.add_argument("--det_db_unclip_ratio", type=float, default=1.5)
parser.add_argument("--max_batch_size", type=int, default=10)
parser.add_argument("--use_dilation", type=str2bool, default=False)
parser.add_argument("--det_db_score_mode", type=str, default="fast")

# EAST parmas
parser.add_argument("--det_east_score_thresh", type=float, default=0.8)
parser.add_argument("--det_east_cover_thresh", type=float, default=0.1)
parser.add_argument("--det_east_nms_thresh", type=float, default=0.2)

# SAST parmas
parser.add_argument("--det_sast_score_thresh", type=float, default=0.5)
parser.add_argument("--det_sast_nms_thresh", type=float, default=0.2)

# PSE parmas
parser.add_argument("--det_pse_thresh", type=float, default=0)
parser.add_argument("--det_pse_box_thresh", type=float, default=0.85)
parser.add_argument("--det_pse_min_area", type=float, default=16)
parser.add_argument("--det_pse_scale", type=int, default=1)

# FCE parmas
parser.add_argument("--scales", type=list, default=[8, 16, 32])
parser.add_argument("--alpha", type=float, default=1.0)
parser.add_argument("--beta", type=float, default=1.0)
parser.add_argument("--fourier_degree", type=int, default=5)

# params for text recognizer
parser.add_argument("--rec_algorithm", type=str, default="SVTR_LCNet")#文本识别算法名称，目前支持CRNN, SRN, RARE, NETR, SAR, ViTSTR, ABINet, VisionLAN, SPIN, RobustScanner, SVTR, SVTR_LCNet
parser.add_argument("--rec_model_dir", type=str,default ="PaddleOCR/inference/rec_ppocr_v3/")#识别inference模型路径
parser.add_argument("--rec_image_inverse", type=str2bool, default=True)
parser.add_argument("--rec_image_shape", type=str, default="3, 48, 320")#识别时的图像尺寸
parser.add_argument("--rec_batch_num", type=int, default=6) #识别的batch size
parser.add_argument("--max_text_length", type=int, default=25)#识别结果最大长度，在SRN中有效
parser.add_argument(
    "--rec_char_dict_path", type=str, default="./ppocr/utils/ppocr_keys_v1.txt"
)#识别的字符字典文件
parser.add_argument("--use_space_char", type=str2bool, default=True)#是否包含空格，如果为True，则会在最后字符字典中补充空格字符
parser.add_argument("--vis_font_path", type=str, default="./doc/fonts/simfang.ttf")
parser.add_argument("--drop_score", type=float, default=0.5)

# params for e2e
parser.add_argument("--e2e_algorithm", type=str, default="PGNet")#端到端算法名称，目前支持PGNet
parser.add_argument("--e2e_model_dir", type=str)
parser.add_argument("--e2e_limit_side_len", type=float, default=768)
parser.add_argument("--e2e_limit_type", type=str, default="max")

# PGNet parmas
parser.add_argument("--e2e_pgnet_score_thresh", type=float, default=0.5)
parser.add_argument(
    "--e2e_char_dict_path", type=str, default="./ppocr/utils/ic15_dict.txt"
)
parser.add_argument("--e2e_pgnet_valid_set", type=str, default="totaltext")
parser.add_argument("--e2e_pgnet_mode", type=str, default="fast")

# params for text classifier
parser.add_argument("--use_angle_cls", type=str2bool, default=False)
parser.add_argument("--cls_model_dir", type=str)
parser.add_argument("--cls_image_shape", type=str, default="3, 48, 192")
parser.add_argument("--label_list", type=list, default=["0", "180"])
parser.add_argument("--cls_batch_num", type=int, default=6)
parser.add_argument("--cls_thresh", type=float, default=0.9)

parser.add_argument("--enable_mkldnn", type=str2bool, default=False)
parser.add_argument("--cpu_threads", type=int, default=10)
parser.add_argument("--use_pdserving", type=str2bool, default=False)
parser.add_argument("--warmup", type=str2bool, default=False)

# SR parmas
parser.add_argument("--sr_model_dir", type=str)
parser.add_argument("--sr_image_shape", type=str, default="3, 32, 128")
parser.add_argument("--sr_batch_num", type=int, default=1)

#
parser.add_argument("--draw_img_save_dir", type=str, default="./inference_results")
parser.add_argument("--save_crop_res", type=str2bool, default=False)
parser.add_argument("--crop_res_save_dir", type=str, default="./output")

# multi-process
parser.add_argument("--use_mp", type=str2bool, default=False)
parser.add_argument("--total_process_num", type=int, default=1)
parser.add_argument("--process_id", type=int, default=0)

parser.add_argument("--benchmark", type=str2bool, default=False)
parser.add_argument("--save_log_path", type=str, default="./log_output/")

parser.add_argument("--show_log", type=str2bool, default=True)
parser.add_argument("--use_onnx", type=str2bool, default=False)

[liuhongen1234567]

您好，请问在paddle3.0b1上也会内存上涨吗？您那边可以试试，paddle2.6我这边也同步看一下

[Yins11]

您好，请问在paddle3.0b1上也会内存上涨吗？您那边可以试试，paddle2.6我这边也同步看一下

未测试paddle3.0b1，不过我刚把paddle版本降级到2.4.2，就不上涨了，正常执行，不知道是不是paddle版本问题，或者是和tensorrt版本不匹配？？工作需要把paddle版本升级到2.6.1，ocr版本升级为2.8.1，所以还是要定位原因。麻烦了

[KyleWang-Hunter]

到底哪些版本会存在内存泄漏问题，官方能不能具体给个说明？

[liuhongen1234567]

您好，您那边可以提供一下导出的配置的文件和官方权重的位置吗？

[Yins11]

您好，您那边可以提供一下导出的配置的文件和官方权重的位置吗？

model : https://github.com/PaddlePaddle/PaddleOCR/blob/v2.8.1/doc/doc_ch/models_list.md
v3 config:
`Global:
debug: false
epoch_num: 100
log_smooth_window: 20
print_batch_step: 10
save_model_dir: ./output/v3_en_mobile
save_epoch_step: 3
eval_batch_step:

• 0
• 100
  cal_metric_during_train: true
  pretrained_model: xxxxxxxxxxxxx/pretrain_model/en_PP-OCRv3_rec_train/best_accuracy.pdparams
  checkpoints: null
  save_inference_dir: null
  use_visualdl: false
  infer_img: doc/imgs_words/ch/word_1.jpg
  character_dict_path: ppocr/utils/en_dict.txt
  max_text_length: 25
  infer_mode: false
  use_space_char: true
  distributed: true
  save_res_path: ./output/rec/predicts_ppocrv3_en.txt
  Optimizer:
  name: Adam
  beta1: 0.9
  beta2: 0.999
  lr:
  name: Cosine
  learning_rate: 0.001
  warmup_epoch: 5
  regularizer:
  name: L2
  factor: 3.0e-05
  Architecture:
  model_type: rec
  algorithm: SVTR_LCNet
  Transform: null
  Backbone:
  name: MobileNetV1Enhance
  scale: 0.5
  last_conv_stride:
  • 1
  • 2
    last_pool_type: avg
    last_pool_kernel_size:
  • 2
  • 2
    Head:
    name: MultiHead
    head_list:
  • CTCHead:
    Neck:
    name: svtr
    dims: 64
    depth: 2
    hidden_dims: 120
    use_guide: true
    Head:
    fc_decay: 1.0e-05
  • SARHead:
    enc_dim: 512
    max_text_length: 25
    Loss:
    name: MultiLoss
    loss_config_list:
• CTCLoss: null
• SARLoss: null
  PostProcess:
  name: CTCLabelDecode
  Metric:
  name: RecMetric
  main_indicator: acc
  ignore_space: false
  Train:
  dataset:
  name: SimpleDataSet
  data_dir: xxxxxxxxxxxxxxxxxxxxxxxxxx
  ext_op_transform_idx: 1
  label_file_list:
  • xxxxxxxxxxxxxxxxxxxxxxx/train_list.txt
    transforms:
  • DecodeImage:
    img_mode: BGR
    channel_first: false
  • RecConAug:
    prob: 0.5
    ext_data_num: 2
    image_shape:
    • 48
    • 320
    • 3
      max_text_length: 25
  • RecAug: null
  • MultiLabelEncode: null
  • RecResizeImg:
    image_shape:
    • 3
    • 48
    • 320
  • KeepKeys:
    keep_keys:
    • image
    • label_ctc
    • label_sar
    • length
    • valid_ratio
      loader:
      shuffle: true
      batch_size_per_card: 128
      drop_last: true
      num_workers: 4
      Eval:
      dataset:
      name: SimpleDataSet
      data_dir: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
      label_file_list:
  • xxxxxxxxxxxxxxxxxxxxxxx/val_list.txt
    transforms:
  • DecodeImage:
    img_mode: BGR
    channel_first: false
  • MultiLabelEncode: null
  • RecResizeImg:
    image_shape:
    • 3
    • 48
    • 320
  • KeepKeys:
    keep_keys:
    • image
    • label_ctc
    • label_sar
    • length
    • valid_ratio
      loader:
      shuffle: false
      drop_last: false
      batch_size_per_card: 128
      num_workers: 4
      profiler_options: null
      v4 config:Global:
      debug: false
      use_gpu: true
      epoch_num: 2
      amp_level: O1
      log_smooth_window: 20
      print_batch_step: 10
      save_model_dir: xxxxxxxxxxxxxxxxxxxxxxxx
      save_epoch_step: 10
      eval_batch_step:
• 0
• 80
  cal_metric_during_train: true
  pretrained_model: xxxxxxxxxxxxxxxxxxx/pretrain_model/en_PP-OCRv4_rec_train/best_accuracy.pdparams
  checkpoints: null
  save_inference_dir: null
  use_visualdl: true
  use_wandb: true
  infer_img: doc/imgs_words/ch/word_1.jpg
  character_dict_path: PaddleOCR/ppocr/utils/en_dict.txt
  max_text_length: 25
  infer_mode: false
  use_space_char: true
  distributed: true
  save_res_path: PaddleOCR/output/rec/predicts_ppocrv3.txt
  Optimizer:
  name: Adam
  beta1: 0.9
  beta2: 0.999
  lr:
  name: Cosine
  learning_rate: 0.0005
  warmup_epoch: 5
  regularizer:
  name: L2
  factor: 3.0e-05
  Architecture:
  model_type: rec
  algorithm: SVTR_LCNet
  Transform: null
  Backbone:
  name: PPLCNetV3
  scale: 0.95
  Head:
  name: MultiHead
  head_list:
  • CTCHead:
    Neck:
    name: svtr
    dims: 120
    depth: 2
    hidden_dims: 120
    kernel_size:
    - 1
    - 3
    use_guide: true
    Head:
    fc_decay: 1.0e-05
  • NRTRHead:
    nrtr_dim: 384
    max_text_length: 25
    Loss:
    name: MultiLoss
    loss_config_list:
• CTCLoss: null
• NRTRLoss: null
  PostProcess:
  name: CTCLabelDecode
  Metric:
  name: RecMetric
  main_indicator: acc
  ignore_space: false
  Train:
  dataset:
  name: MultiScaleDataSet
  ds_width: false
  data_dir: xxxxxxxxxxxxxxxxxxxxxxxxxxxx
  ext_op_transform_idx: 1
  label_file_list:
  • xxxxxxxxxxxxxxxxxxxxxxxxxxxx/train_list.txt
    transforms:
  • DecodeImage:
    img_mode: BGR
    channel_first: false
  • RecConAug:
    prob: 0.5
    ext_data_num: 2
    image_shape:
    • 48
    • 320
    • 3
      max_text_length: 25
  • RecAug: null
  • MultiLabelEncode:
    gtc_encode: NRTRLabelEncode
  • KeepKeys:
    keep_keys:
    • image
    • label_ctc
    • label_gtc
    • length
    • valid_ratio
      sampler:
      name: MultiScaleSampler
      scales:
  • • 320
    • 32
  • • 320
    • 48
  • • 320
    • 64
      first_bs: 96
      fix_bs: false
      divided_factor:
  • 8
  • 16
    is_training: true
    loader:
    shuffle: true
    batch_size_per_card: 96
    drop_last: true
    num_workers: 8
    Eval:
    dataset:
    name: SimpleDataSet
    data_dir: xxxxxxxxxxxxxxxxxxxxxx
    label_file_list:
  • xxxxxxxxxxxxxxxxxxxxx/val_list.txt
    transforms:
  • DecodeImage:
    img_mode: BGR
    channel_first: false
  • MultiLabelEncode:
    gtc_encode: NRTRLabelEncode
  • RecResizeImg:
    image_shape:
    • 3
    • 48
    • 320
  • KeepKeys:
    keep_keys:
    • image
    • label_ctc
    • label_gtc
    • length
    • valid_ratio
      loader:
      shuffle: false
      drop_last: false
      batch_size_per_card: 128
      num_workers: 4
      profiler_options: null
      `

[liuhongen1234567]

您好，请问您那边测试的图片数量是多少？有测试过官方模型是否会内存泄漏吗？我用官方模型开tensorrt测了一下，并用memory_profiler 可视化了一下内存，发现它确实会涨一下，但是也没有持续的上涨。

[liuhongen1234567]

我这边一张图重复了10万次

[liuhongen1234567]

可视化内存的代码如下predict_rec.py.zip

[JoshonSmith]

我这边一张图重复了10万次

每次将图做点修改后，执行，重复10万次呢

[Yins11]

可视化内存的代码如下predict_rec.py.zip

感谢解答！ 已定位到原因，是环境问题，现在可以正常推理了。