Problems with Auto Segmentation

[ErlendAQK]

Hi,
I have some troubles with the Auto Segmentation of my 2D DICOM images of fish. Nothing happens when I click run Auto Segmentation.

Training my model
I started out with the radiology app, made a new copy (segmentation_fisk2.py) from the file segmentation.py in the config-folder. I edited the labels to (norwegian)

self.labels = {
            "Fisk": 1,    #Fisk=Fish
            "Ryggrad": 2, #Ryggrad=spine
            "Skade": 3,   #Skade=damage
        } 

and set pretrain to false: strtobool(self.conf.get("use_pretrained_model", "False"))

I have annotated and submitted labels for (only) 9 images similar to this image. The annotation work is not very precise. I just want to get some results. I have not used the annotation label 3 "Skade/damage". I ran "train" and the model was trained through 50 epochs as expected with decent results. Or is it something wrong already at this point?

From train_stats.json:

{
  "rank": 0,
  "current_epoch": 50,
  "current_iteration": 350,
  "total_epochs": 50,
  "total_iterations": 7,
  "epoch": 50,
  "start_ts": 1724841657,
  "total_time": "1:12:05",
  "best_metric": 0.7286776304244995,
  "train": {
    "metrics": {
      "train_mean_dice": 0.7298908233642578,
      "train_Fisk_mean_dice": 0.8380375504493713,
      "train_Ryggrad_mean_dice": 0.7253682613372803,
      "train_Skade_mean_dice": 0.0
    },
    "key_metric_name": "train_mean_dice",
    "best_metric": 0.7567542195320129,
    "best_metric_epoch": 46
  },
  "eval": {
    "metrics": {
      "val_mean_dice": 0.7286776304244995,
      "val_Fisk_mean_dice": 0.7747041583061218,
      "val_Ryggrad_mean_dice": 0.682651162147522,
      "val_Skade_mean_dice": 0.0
    },
    "key_metric_name": "val_mean_dice",
    "best_metric": 0.7286776304244995,
    "best_metric_epoch": 50
  }
}

The problem
Now I have loaded a new image, but when I press run autosegmentation, nothing happens. And I don't know where to troubleshoot. I assume the problem lies in the AutoSegmentation and not the model itself? But I don't know where to look for the error. I hope you are able to help me out.

From monailabel error log:

log1

Current Selection Options Section: infer
Current Selection Options Name: segmentation_fisk2
Invalidate: models => segmentation_fisk2 => largest_cc => False => <class 'bool'>
Invalidate: models => segmentation_fisk2 => device => ['cpu'] => <class 'list'>

log2

Preparing input data for segmentation
GET http://127.0.0.1:8000/auth/
URI Path: /auth/
HTTP Response Code: 404
HTTP Response Message: Not Found
HTTP Response Headers: [('date', 'Wed, 28 Aug 2024 12:04:21 GMT'), ('server', 'uvicorn'), ('content-length', '22'), ('content-type', 'application/json')]
HTTP Response Content-Type: application/json
Reading status/content from simple response!
POST http://127.0.0.1:8000/infer/segmentation_fisk2?image=F1_Sinkaberg-Hansen_Svaberget__9.dcm
URI Path: /infer/segmentation_fisk2?image=F1_Sinkaberg-Hansen_Svaberget__9.dcm

log3

HTTP Response Code: 200
HTTP Response Message: OK
HTTP Response Headers: [('date', 'Wed, 28 Aug 2024 12:04:21 GMT'), ('server', 'uvicorn'), ('content-length', '1989873'), ('content-type', 'multipart/form-data; boundary=8c2415e876524c6d8a3736c22738f65a')]
HTTP Response Content-Type: multipart/form-data; boundary=8c2415e876524c6d8a3736c22738f65a
FILE-NAME: None; NAME: params; SIZE: 508
FILE-NAME: tmpmngofc_n.nrrd; NAME: image; SIZE: 1989053
Response FORM: {'params': '{"label_names": {"Fisk": 1, "Ryggrad": 2, "Skade": 3}, "latencies": {"pre": 0.06, "infer": 3.65, "invert": 0.18, "post": 0.3, "write": 0.01, "total": 4.19, "transform": {"pre": {"LoadImaged": 0.0321, "EnsureTyped": 0.0, "EnsureChannelFirstd": 0.0, "Orientationd": 0.004, "Spacingd": 0.016, "NormalizeIntensityd": 0.001, "GaussianSmoothd": 0.002, "ScaleIntensityd": 0.0}, "post": {"EnsureTyped": 0.0, "Activationsd": 0.009, "AsDiscreted": 0.245, "Restored": 0.001, "GetCentroidsd": 0.0375}}}, "centroids": {}}'}
Response FILES: dict_keys(['tmpmngofc_n.nrrd'])
Saving tmpmngofc_n.nrrd to C:/Users/ErlendKallelid/AppData/Local/Temp/Slicer/slicer-monai-label2024-08-28_12+38+02.553\tmpmngofc_n.nrrd; Size: 1989053
Image Response: C:/Users/ErlendKallelid/AppData/Local/Temp/Slicer/slicer-monai-label2024-08-28_12+38+02.553\tmpmngofc_n.nrrd
JSON  Response: {'label_names': {'Fisk': 1, 'Ryggrad': 2, 'Skade': 3}, 'latencies': {'pre': 0.06, 'infer': 3.65, 'invert': 0.18, 'post': 0.3, 'write': 0.01, 'total': 4.19, 'transform': {'pre': {'LoadImaged': 0.0321, 'EnsureTyped': 0.0, 'EnsureChannelFirstd': 0.0, 'Orientationd': 0.004, 'Spacingd': 0.016, 'NormalizeIntensityd': 0.001, 'GaussianSmoothd': 0.002, 'ScaleIntensityd': 0.0}, 'post': {'EnsureTyped': 0.0, 'Activationsd': 0.009, 'AsDiscreted': 0.245, 'Restored': 0.001, 'GetCentroidsd': 0.0375}}}, 'centroids': {}}

log4
Result Params for Segmentation: {'label_names': {'Fisk': 1, 'Ryggrad': 2, 'Skade': 3}, 'latencies': {'pre': 0.06, 'infer': 3.65, 'invert': 0.18, 'post': 0.3, 'write': 0.01, 'total': 4.19, 'transform': {'pre': {'LoadImaged': 0.0321, 'EnsureTyped': 0.0, 'EnsureChannelFirstd': 0.0, 'Orientationd': 0.004, 'Spacingd': 0.016, 'NormalizeIntensityd': 0.001, 'GaussianSmoothd': 0.002, 'ScaleIntensityd': 0.0}, 'post': {'EnsureTyped': 0.0, 'Activationsd': 0.009, 'AsDiscreted': 0.245, 'Restored': 0.001, 'GetCentroidsd': 0.0375}}}, 'centroids': {}}

log5
Update Segmentation Mask from: C:/Users/ErlendK/AppData/Local/Temp/Slicer/slicer-monai-label2024-08-28_12+38+02.553\tmpmngofc_n.nrrd

log6

Update Segmentation Mask using Labels: ['Fisk', 'Ryggrad', 'Skade']
Time consumed by Import LabelMask: 0.0
Import only Freezed label: None
Time consumed by updateSegmentationMask: 0.0

log7

Current SegmentID: ; Segment: None
Current SegmentID: ; Segment: None
Current SegmentID: Fisk; Segment: vtkSegment (000001D80CCF4AC0)
  Name: Fisk
  Color: (0.501961, 0.682353, 0.501961)
  NameAutoGenerated: true
  ColorAutoGenerated: true
  Debug: Off
  Modified Time: 15170982
  Representations:
    Binary labelmap:
      ClassName: vtkOrientedImageData
      Origin: 0 0 0
      Spacing: 0.125 0.125 1
      Extent: 0 2156 0 921 0 0
      Scalar type: unsigned char
      Number of components: 1
      IJKToRASDirections:
        -1 0 0 
        0 -1 0 
        0 0 1 
  Tags:
      TerminologyEntry: Segmentation category and type - 3D Slicer General Anatomy list~SCT^85756007^Tissue~SCT^85756007^Tissue~^^~Anatomic codes - DICOM master list~^^~^^


Fisk => Fisk Control points are: 
Current SegmentID: Fisk; Segment: vtkSegment (000001D80CCF4AC0)
  Name: Fisk
  Color: (0.501961, 0.682353, 0.501961)
  NameAutoGenerated: true
  ColorAutoGenerated: true
  Debug: Off
  Modified Time: 15170982
  Representations:
    Binary labelmap:
      ClassName: vtkOrientedImageData
      Origin: 0 0 0
      Spacing: 0.125 0.125 1
      Extent: 0 2156 0 921 0 0
      Scalar type: unsigned char
      Number of components: 1
      IJKToRASDirections:
        -1 0 0 
        0 -1 0 
        0 0 1 
  Tags:
      TerminologyEntry: Segmentation category and type - 3D Slicer General Anatomy list~SCT^85756007^Tissue~SCT^85756007^Tissue~^^~Anatomic codes - DICOM master list~^^~^^


Fisk => Fisk Control points are: 
GET http://127.0.0.1:8000/auth/
URI Path: /auth/
HTTP Response Code: 404
HTTP Response Message: Not Found
HTTP Response Headers: [('date', 'Wed, 28 Aug 2024 12:04:21 GMT'), ('server', 'uvicorn'), ('content-length', '22'), ('content-type', 'application/json')]
HTTP Response Content-Type: application/json
Reading status/content from simple response!
GET http://127.0.0.1:8000/train/?check_if_running=true
URI Path: /train/?check_if_running=true
HTTP Response Code: 404
HTTP Response Message: Not Found
HTTP Response Headers: [('date', 'Wed, 28 Aug 2024 12:04:21 GMT'), ('server', 'uvicorn'), ('content-length', '49'), ('content-type', 'application/json')]
HTTP Response Content-Type: application/json
Reading status/content from simple response!

log8
Time consumed by segmentation: 4.3

Finally, the model is run with roi_size (96,96,96). Is that relevant? Could I have used (96,96,1) since I have 2D images? Will that increase the speed?

[ErlendAQK]

Can the problem be as simple that I have trained a 3D-model on 2D-images and that it does not manage to place the labels on the correct layer?

Anyhow, I tried to make it a 2D model by chaning roi_size, target_spacing and spatial_dims:

        self.target_spacing = (1.5, 1.5)  # target space for image
        # Setting ROI size - This is for the image padding
        self.roi_size = (96, 96)

        # Network
        self.network = SegResNet(
            spatial_dims=2,
            in_channels=1,
            out_channels=len(self.labels)+1,  # labels plus background,
            init_filters=32,
            blocks_down=(1,2,2,4),
            blocks_up=(1,1,1),
            dropout_prob=0.2,
        )

which gave the following error in the cmd-window.

  File "C:\Users\ErlendKallelid\apps\radiology\lib\trainers\segmentation.py", line 123, in val_pre_transforms
    k_divisible=[self.roi_size[0], self.roi_size[1], self.roi_size[2]],

Do I need to change any other parameteres? Is the pipeline able to handle 2D-models at all?

[diazandr3s]

Hi @ErlendAQK,

Thanks for sharing your experience here.

Can the problem be as simple that I have trained a 3D-model on 2D-images and that it does not manage to place the labels on the correct layer?

You're correct. MONAI Label is designed for 3D image segmentation. Using the Radiology segmentation model for 2D images requires significant changes to pre-transforms, network architecture, and hyperparameters. For 2D segmentation, consider using MONAI Label's Endoscopy application, which works with the CVAT viewer.

Hope this helps to clarity scope

[ErlendAQK]

Thanks for your reply.
I will definitely look into it. Do you know how it performs on images such as mine? Should I run it from scratch or with a pretrained model? A pretrained model is perhaps built-in the app?

[diazandr3s]

That's right. There is already a pretrained model but that's for endoscopy videos. I'd suggest starting from scratch.
Some experience with CVAT viewer is highly recommended.

[ErlendAQK]

Thank you! Learning step by step :-)