DeepEdit click based refinement seems to be adding segmentation on wrong side

[VishalJ99]

Ive taken the default radiology app and used it to train a deepEdit model to do volumetric MRI pancreas segmentation.

The auto segmentation works well however I am experiencing a strange issue when I try and make click based refinements.

Here is an example. This is the auto segmentation generated by the model:

You can see the right side could use further filling in so i add a foreground click in the desired region and the segmentation is updated to give the following:

The refinement seems to be based on a click that was defined on the left side when I placed it on the right...

Here is the output when I purposefully define the click on the wrong side:

i get the desired output.

I do not experience this issue when making click based refinements on a model trained on the CT spleen dataset downloaded from monai labels example datasets. I have not made any changes to any files other than renaming the labels.

The only explanation i can think of is that my input data is of orientation LAS whereas the CT spleen data is of orientation RAS.
However, i noticed that in the training and infer pre transforms, one of the transforms defined is the orientationD transform which re orientates the volumes to RAS, so this should not cause any issue...

[lassoan]

DICOM and most (if not all) medical imaging software uses right-handed coordinate system. How did you end up with a left-handed one (LAS)? You may need to resample your image to be in a left-handed coordinate system.

[VishalJ99]

DICOM and most (if not all) medical imaging software uses right-handed coordinate system. How did you end up with a left-handed one (LAS)? You may need to resample your image to be in a left-handed coordinate system.

Not sure, however it shouldn't matter because of the orientationD transform defined in the infer and training pre transforms re orientating all volumes to RAS anyway?

[lassoan]

It should not matter which right-handed coordinate system you use. If you use a left-handed coordinate system in the image then the image to physical coordinate system transformation matrix will have a negative determinant, which may cause problems in some algorithms. It is also possible that the matrix is orthogonalized somewhere along the way by computing the third axis from the cross product of two other axes, which works perfectly for any right-handed coordinate system but will flip an image axis for images that have left-handed coordinate system.

I would recommend to try with images in LPS and RAS coordinate systems and if they all work well then most likely the issue is the left-handed coordinate system. If that's the case then MONAILabel should at least fail with an error instead of silently doing something wrong.

[VishalJ99]

I called nib.as_closest_canonical(img) on each pancreas volume and corresponding mask to re orientate my dataset into RAS prior to training however I am noticing the same issue.

[tangy5]

In addition, you could check the meta information, whether it's normal. MONAI Label takes the meta info and do reorientation/resample transform...etc. If the meta is not in correctly set, it may result in the wrong reorientation...

[VishalJ99]

sorry can you point to me which file / function this step is done? Still getting used to the framework. Though im beginning to suspect its more to do with the infer process rather than the training process. If the clicks were being incorrectly simulated the training loss should not decrease. I think this is to do with how the clicks im defining via slicers GUI are being assigned to a specific voxel.

[VishalJ99]

update:
Confirmed that error is due to guidance signal being flipped
Here is the input click on slicer:

Saved foreground guidance signal being passed as input to model along with input volume, here is the result:

[SachidanandAlle]

@diazandr3s can you fix the pre-transform to take care of click points if you are doing a flip (orientation) on the original image..

[VishalJ99]

Why would the original image be getting flipped if its already been re orientated to RAS by the nibabel.as_closest_canonical() function?

[diazandr3s]

@diazandr3s can you fix the pre-transform to take care of click points if you are doing a flip (orientation) on the original image..

Sure! More than happy to help with this.

I just wanted to ensure we are "collecting" and sending the clicks with the correct orientation. Unless the issue is what @lassoan mentioned.

@VishalJ99 I think this is what you were asking: https://github.com/Project-MONAI/MONAILabel/blob/main/plugins/slicer/MONAILabel/MONAILabel.py#L936-L958

That's the function that processes the clicks. Specifically these two lines: https://github.com/Project-MONAI/MONAILabel/blob/main/plugins/slicer/MONAILabel/MONAILabel.py#L936-L958

@VishalJ99 is this happening with all CTs?

[VishalJ99]

@diazandr3s can you fix the pre-transform to take care of click points if you are doing a flip (orientation) on the original image..

Sure! More than happy to help with this.

I just wanted to ensure we are "collecting" and sending the clicks with the correct orientation. Unless the issue is what @lassoan mentioned.

@VishalJ99 I think this is what you were asking: https://github.com/Project-MONAI/MONAILabel/blob/main/plugins/slicer/MONAILabel/MONAILabel.py#L936-L958

That's the function that processes the clicks. Specifically these two lines: https://github.com/Project-MONAI/MONAILabel/blob/main/plugins/slicer/MONAILabel/MONAILabel.py#L936-L958

@VishalJ99 is this happening with all CTs?

No, for the spleen CT dataset downloaded with the example radiology app, there were no such issues. This issue arises when I tried to train a deepedit model with the same training and infer configs (except for modifying label names and choosing to not use a pre trained model) to do MRI pancreas segmentation.

thanks for the links! Will compare how the points are being recorded for a click made on the spleen CT volumes vs the pancreas.

[VishalJ99]

Another update:
There definetly seems to be difference in how the coordinates of the clicks are being recorded on slicer:

Here is a spleen CT, you can see a click placed on the top left has the following coordinate info:

Here is the same for the pancreas MRI:

It seems for the spleen the top left corner is where L=0 whereas for the pancreas its where roughly R=250
Further in terms of the actual array index, the top left corner seems to have an index of (img.shape[0],img.shape[1]) for the spleen but of (0,img.shape[1]) for the pancreas

What is the difference between the two different coordinates (for given case), i get that the bottom one refers to the actual voxel index the cursor is over, but im not sure what the coordinates shown on top with the L/R axis codes refer to.

EDIT:
Since this probably indicates that there is ultimately some difference in the header info, ive tried to investigate using a sample case from the spleen CT and the pancreas MRI datasets:

>>> import nibabel as nib
>>> pancreas_mri = nib.load("/home/monai/datasets/Pancreas_dataset/imagesTr/101-0001.nii.gz")
>>> spleen_ct = nib.load("/home/monai/datasets/Task09_Spleen/imagesTr/spleen_2.nii.gz")
>>> pancreas_mri.affine
array([[   1.97920001,    0.        ,    0.        , -169.02656555],
       [   0.        ,    1.97920001,    0.        , -190.72224426],
       [   0.        ,    0.        ,    4.        , -383.73284912],
       [   0.        ,    0.        ,    0.        ,    1.        ]])
>>> spleen_ct.affine
array([[   0.79492199,    0.        ,    0.        , -406.20513916],
       [   0.        ,    0.79492199,    0.        , -406.20513916],
       [   0.        ,    0.        ,    5.        ,    0.        ],
       [   0.        ,    0.        ,    0.        ,    1.        ]])
>>> nib.aff2axcodes(pancreas_mri.affine)
('R', 'A', 'S')
>>> nib.aff2axcodes(spleen_ct.affine)
('R', 'A', 'S')
>>> pancreas_mri.header['qform_code']
array(0, dtype=int16)
>>> spleen_ct.header['qform_code']
array(1, dtype=int16)
>>> pancreas_mri.header['sform_code']
array(2, dtype=int16)
>>> spleen_ct.header['sform_code']
array(1, dtype=int16)

so while the orientation appears to be the same, due to the differences in the qform and sform codes the relationship between the voxel coordinates and the real-world anatomical coordinate system maybe different for the two volumes. This could be causing part of the issue.

[VishalJ99]

2nd update:
For what its worth, i edited the header info of the pancreas mri volume so the qform and sform codes match that of the CT spleen volume's. When manually loading the modified mri volume into slicer, the top left corner of an axial slice does indeed match the spleen CT coordinate, in that voxel coordinates (i,j,k) for the top left corner has i = image.shape[0] instead of 0 as it is currently.

However, when loading this same volume with the modified header in via monai labels slicer plugin (clicking next sample), for some reason this change is reset such that the top left corner again has i = 0 (as shown in the picture attached in the previous post). Currently checking why loading in the volume via the monai label plug in does this...

[VishalJ99]

2nd update: For what its worth, i edited the header info of the pancreas mri volume so the qform and sform codes match that of the CT spleen volume's. When manually loading the modified mri volume into slicer, the top left corner of an axial slice does indeed match the spleen CT coordinate, in that voxel coordinates (i,j,k) for the top left corner has i = image.shape[0] instead of 0 as it is currently.

However, when loading this same volume with the modified header in via monai labels slicer plugin (clicking next sample), for some reason this change is reset such that the top left corner again has i = 0 (as shown in the picture attached in the previous post). Currently checking why loading in the volume via the monai label plug in does this...

Ok so turns out the reason for this was that the volume was not actually being downloaded from the datastore and loaded when next sample was clicked, it was being loaded from cache - so the volumes header was unchanged... Deleting the cache, the new volume with the updated header is being read in the same was as the spleen CT. Making clicks on this volume work as intended.

Im not sure id put this as a solution but more a work around? Maybe add requirements on the sform and qform codes required?

[SachidanandAlle]

Meanwhile you can try deep grow 2d and 3d models.. if that helps

[diazandr3s]

Thanks for all the updates, @VishalJ99. I'm planning to work on this early next week.
We could also meet and further discuss if that's OK for you. We can then report back on this thread.

[VishalJ99]

@diazandr3s sounds good, happy to help in any capacity.

[diazandr3s]

Hi @VishalJ99,

I checked this issue and it seems it is more about the combination of the image header and the reader you use.

Can you please try removing or changing the reader's argument in this line? https://github.com/Project-MONAI/MONAILabel/blob/main/sample-apps/radiology/lib/infers/deepedit.py#L77

You could try something like this:

LoadImaged(keys="image"),

Or this

LoadImaged(keys="image", reader="NibabelReader"),

Hope this helps,

[diazandr3s]

@diazandr3s sounds good, happy to help in any capacity.

Hi @VishalJ99,

I've opened a PR to solve the click orientation issue: #1537
Would you please help us to check if that works for you?

Looking forward to hearing from you,

[VishalJ99]

@diazandr3s sounds good, happy to help in any capacity.

Hi @VishalJ99,

I've opened a PR to solve the click orientation issue: #1537 Would you please help us to check if that works for you?

Looking forward to hearing from you,

Sure will check tomorrow so just to make sure i understand what it is I am checking can you confirm the following?

loading in a pancreas case with an orientation of RAS and Qform code of 0 and sform code of 2 I should expect the clicks to map onto the original volume correctly. My current solution is to modify the Q form and S form codes to 1 which has worked for me so far.

[diazandr3s]

Thanks, @VishalJ99.
I was thinking more about testing this PR on images with a different orientation that RAS.
If the images have corrupted headers, that's something a bit out of scope for MONAI Label. It is more a data conversion kind of thing.
But please try it anyway and let us know.

[lassoan]

If you work with NIFTI file format then unfortunately you'll always run into image orientation issues, because this file format uses a complex and redundant way of representing image orientation. People who develop and use NIFTI has been unable to clean up the mess - they cannot agree how to resolve ambiguities and some of them don't even acknowledge that there is an problem. I would recommend to use DICOM if you want to preserve all metadata in a very structured and standard way; or use NRRD if you only need simple image storage (with some basic optional custom metadata).