Training from scratch & Inference Problem

[SanghoonKang503]

Hi, thank you for great application Monailabel.

I was trying to train the segmentation model from scratch and inference the data.

I put 3-segment elements including background at segment session and trained with them, but the inference result only showed one segmentation result.

When I checked the label that I created, It only has one mask. I am worried about that the label made correctly.

• Is that the problem of insufficient training? or is there another option to make label?

• Additional Question

1. What is epistemic option for?
2. The training with Monailabel only support the batchsize=1?
   When I checked the log, the iteration was 1/1. That means the model only takes 1 nifti file? not every slices for training?
3. Is there any options for training whole data at once without active learning?

[tangy5]

Hi @SanghoonKang503 , you could check following compoenet to debug your workflow. It looks like a code issue instead of insufficient training.

1. You could check the post transform on your Python or bundle config files, for example, how the activation and asdiscrete are used.
2. are the label in the lib/config/ defined correctly?
3. are the output channels defined correctly in the lib/config/.

For additional questions:

1. The epistemic is for "active learning", which means the algorithm will rank all unlabeled images and each time you fetch an image, it will based on the ranks. And ranks will be based on samples that will be most useful.
2. You could adjust the batchsize, But you need to deep into the code to change here:

   MONAILabel/monailabel/tasks/train/basic_train.py

   Line 164 in 1d03472

   "train_batch_size": 1,

   
   Or you can pass the train batch size as an argument and change it.
   3.Yes, as you could see the annotated data are saved in the "finished" folder in your dataset path. You could put all label in the folder. And the datastore will recognize that all data are annotated and ready for training. In this case, the monailabel and 3D Slicer are just a UI for triggering training.

[SanghoonKang503]

Thank you for your explanation @tangy5 !!

For basic question:
In my case, I do not have the lib/config path in my MonaiLabel Deepedit apps.
In my experience using 3D-slicer with MonaiLabel, I knew that the *.nii.gz file is automatically created at labels/final located in 'studies root' after manually annotation made and click 'Submit Label'.
Also in the apps/deepedit/logs/app.log shows that the labels are read from there.

[2023-08-10 13:02:17.765][ INFO](monailabel.deepedit.transforms) - Label 029-2012_WIP_3D_GE_Quads_NFS_1401.nii.gz has more than one mask - taking SINGLE mask ...
[2023-08-10 13:02:17.897][ INFO](monailabel.deepedit.transforms) - Image 074-2014_3D_GRE_40.nii has more than one modality - taking FIRST modality ...
[2023-08-10 13:02:17.897][ INFO](monailabel.deepedit.transforms) - Label 074-2014_3D_GRE_40.nii.gz has more than one mask - taking SINGLE mask ...
[2023-08-10 13:02:17.928][ INFO](monailabel.deepedit.transforms) - Image 084-2013_3D_GRE_FS_55.nii has more than one modality - taking FIRST modality ...
[2023-08-10 13:02:17.929][ INFO](monailabel.deepedit.transforms) - Label 084-2013_3D_GRE_FS_55.nii.gz has more than one mask - taking SINGLE mask ...
[2023-08-10 13:02:17.932][ INFO](monailabel.deepedit.transforms) - Image 029-2014_WIP_3D_GE_Quads_NFS_TE2.3_1401.nii has more than one modality - taking FIRST modality ...
[2023-08-10 13:02:17.933][ INFO](monailabel.deepedit.transforms) - Label 029-2014_WIP_3D_GE_Quads_NFS_TE2.3_1401.nii.gz has more than one mask - taking SINGLE mask ...
[2023-08-10 13:02:17.938][ INFO](monailabel.deepedit.transforms) - Image 029-2011_WIP_3D_GE_Quads_NFS_1501.nii has more than one modality - taking FIRST modality ...
[2023-08-10 13:02:17.939][ INFO](monailabel.deepedit.transforms) - Label 029-2011_WIP_3D_GE_Quads_NFS_1501.nii.gz has more than one mask - taking SINGLE mask ...
Loading dataset:  17%|█▋        | 1/6 [00:00<00:02,  1.72it/s][2023-08-10 13:02:18.453][ INFO](monailabel.deepedit.transforms) - Image 032-2017_WIP_3D_GE_NFS_TE2.3_UL_1401.nii has more than one modality - taking FIRST modality ...
[2023-08-10 13:02:18.455][ INFO](monailabel.deepedit.transforms) - Label 032-2017_WIP_3D_GE_NFS_TE2.3_UL_1401.nii.gz has more than one mask - taking SINGLE mask ...

For additional question:
1,3. Thank you for your explanation. I understand the "epistemic" mean finally:) and also it is helpful for me how to train model at once.
2. I tried to change the batchsize at 3D-slicer UI, but it occurs error of mismatching tensor size like below logs. Is it occurred from pytorch DataLoader?

[2023-08-10 13:01:20.606][CRITICAL](dev_collate) - >> E: stack expects each tensor to be equal size, but got [24] at entry 0 and [33] at entry 1, shape [torch.Size([24]), torch.Size([33])] in collate([tensor([135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148,
        149, 150, 151, 152, 153, 154, 155, 156, 157, 158]), tensor([134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147,
        148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161,
        162, 163, 164, 165, 166])])
[2023-08-10 13:01:20.606][CRITICAL](dev_collate) - > collate dict key "guidance" out of 9 keys
Exception in thread Thread-267:
Traceback (most recent call last):
  File "/opt/monai/monai/data/utils.py", line 340, in list_data_collate
    ret[key] = default_collate([d[key] for d in data])
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 64, in default_collate
    return default_collate([torch.as_tensor(b) for b in batch])
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 56, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [24] at entry 0 and [33] at entry 1
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/opt/conda/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/monai/monai/data/thread_buffer.py", line 45, in enqueue_values
    for src_val in self.src:
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 561, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
  File "/opt/conda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    return self.collate_fn(data)
  File "/opt/monai/monai/data/utils.py", line 354, in list_data_collate
    raise RuntimeError(re_str) from re
RuntimeError: stack expects each tensor to be equal size, but got [24] at entry 0 and [33] at entry 1
Collate error on the key 'sids' of dictionary data.
MONAI hint: if your transforms intentionally create images of different shapes, creating your `DataLoader` with `collate_fn=pad_list_data_collate` might solve this problem (check its documentation).
Data iterator can not provide data anymore but required total number of iterations to run is not reached. Current iteration: 0 vs Total iterations to run : 300
[2023-08-10 13:01:20.665][ERROR](ignite.engine.engine.SupervisedTrainer) - Engine run is terminating due to exception: object of type 'NoneType' has no len()

Thank you for your help:)

[SanghoonKang503]

HI, @diazandr3s

I followed the instruction of the uf-tutorials version that has no config files. Finally, I solved the probelem with installing the monailabel v0.6.0 and it appears.

For training, do you suggest batch size of 1 even with the whole dataset?
I tried to make whole labels at once and click 'trian' to train model with whole dataset, not active learning.

[diazandr3s]

Hi @SanghoonKang503,

Good to hear you managed to solve this.
If using DeepEdit, yes. I'd recommend using a batch size of 1. If you have enough GPU memory, you can use SmartCacheDataset (default option) or CacheDataset for faster training.

Hope this helps,

[SanghoonKang503]

Thanks @diazandr3s !
Would you give me a short explanation about why the batch size of 1 is suitable for training?

[diazandr3s]

It is essentially the click-generation mechanism that is currently implemented in DeepEdit model. This mechanism is a combination of transforms that does not support batch sizes bigger than one. Especially for the inner iteration: https://github.com/Project-MONAI/MONAILabel/wiki/DeepEdit#training-schema

It is important to highlight that this only applies to the DeepEdit model. The segmentation model shouldn't have a problem with a batch size bigger than one.

[SanghoonKang503]

@diazandr3s
I understand! Thank you for your help:)