Wildreceipt dataset KIE training with LayoutXLM on Google Colab - various Errors (loss: nanxxx)

[sandorkonya]

Hi there,

i am trying to train the Wildreceipt dataset (dataset here) on Google colab.

Env:
paddlepaddle-gpu==2.3.2 (see issue here)

Full training script:

Corresponding config settings (full settings within colab notebook):

  Backbone:
    name: LayoutXLMForSer
    pretrained: True
    checkpoints:
    # one of base or vi
    mode: vi
    num_classes: &num_classes 51

Architecture:
  model_type: kie
  algorithm: &algorithm "LayoutXLM"
  Transform:
  Backbone:
    name: LayoutXLMForSer
    pretrained: True
    checkpoints:
    # one of base or vi
    mode: vi
    num_classes: &num_classes 51

Loss:
  name: VQASerTokenLayoutLMLoss
  num_classes: *num_classes
  key: "backbone_out"

Optimizer:
  name: Adam
  beta1: 0.9
  beta2: 0.999
  lr:
    name: Linear
    learning_rate: 0.000005
    epochs: *epoch_num
    warmup_epoch: 2
  regularizer:
    name: L2
    factor: 0.00000

I get however error:

[2022/12/05 20:00:31] ppocr INFO: epoch: [1/100], global_step: 10, lr: 0.000000, loss: 3.848741, avg_reader_cost: 0.04087 s, avg_batch_cost: 0.94356 s, avg_samples: 8.0, ips: 8.47852 samples/s, eta: 4:09:53
eval model:: 100% 59/59 [00:20<00:00,  2.94it/s]
[2022/12/05 20:00:59] ppocr INFO: cur metric, precision: 0.0, recall: 0.0, hmean: 0.0, fps: 26.290234026876494
[2022/12/05 20:01:13] ppocr INFO: save best model is to ./output/ser_vi_layoutxlm_wildreceipt/best_accuracy
[2022/12/05 20:01:13] ppocr INFO: best metric, hmean: 0.0, precision: 0.0, recall: 0.0, fps: 26.290234026876494, best_epoch: 1
[2022/12/05 20:01:14] ppocr INFO: epoch: [1/100], global_step: 20, lr: 0.000000, loss: nanxxx, avg_reader_cost: 0.00025 s, avg_batch_cost: 0.56954 s, avg_samples: 8.0, ips: 14.04639 samples/s, eta: 3:20:14
[2022/12/05 20:01:22] ppocr INFO: epoch: [1/100], global_step: 30, lr: 0.000000, loss: nanxxx, avg_reader_cost: 0.00024 s, avg_batch_cost: 0.53082 s, avg_samples: 8.0, ips: 15.07106 samples/s, eta: 3:00:12

What i tried so far:

• lowered learning rate

• lowered class number

• changed optimizer SDG / Adam

• changed Architecture from "vi" to "base" and got NO ERROR, but the training stucked at the first epoch, and got no response even after waiting for long time.

Anyone ideas?