[Question]INFO [email protected]:0.5:0.05: -0.470, [email protected]: -0.433, [email protected]: -0.497.

[NGYLK]

❓ Question

Hello, first of all, thank you for your contribution. I encountered the following issues during the detection of prostate cancer lesions. My input consists of prostate images that have been segmented using nnUNet, along with the corresponding lesions. After alignment, I passed them through preprocessing. During the nnDetection training process, my results show:

INFO [email protected]:0.5:0.05: -0.470, [email protected]: -0.433, [email protected]: -0.497.

These negative values are obviously unreasonable. However, the Train Loss and Val Loss are gradually decreasing, indicating that the model is converging and optimizing during training. But so far, the values remain negative, which suggests an issue.

I have also followed the dataset and procedures mentioned in the following guide for training: https://github.com/MIC-DKFZ/nnDetection/blob/main/projects/Task021_ProstateX/README.md, but I encountered the same problem as I do now.

Could it be that prostate lesion detection is too complex and requires many more epochs before significant improvements are seen? Does your prostate data have the same issue as mine? Below is the current image from my training. I believe the warning information does not affect the accuracy. I hope you can resolve my confusion. Once again, thank you for your contribution.

[mibaumgartner]

Dear @NGYLK ,

the negative metrics indicate a problem in your dataset.json - as the warning is already indicating ;)

It seems like you are missing lesions of class 0 which will be added as -1 during the metric computation. We formulated ProstateX as a two class problem: benign lesions are class 0 and malignant lesions are class 1. If your dataset only encompass lesions of a single class you need to remove the other one.

Best,
Michael

[NGYLK]

Dear @NGYLK ,

the negative metrics indicate a problem in your dataset.json - as the warning is already indicating ;)

It seems like you are missing lesions of class 0 which will be added as -1 during the metric computation. We formulated ProstateX as a two class problem: benign lesions are class 0 and malignant lesions are class 1. If your dataset only encompass lesions of a single class you need to remove the other one.

Best, Michael

Hello, I’m glad to receive your reply. In fact, all of my label files have lesions labeled as 1, and the background is labeled as 0, as shown in the image. Currently, I’m not considering whether the lesion is clinically significant or not; I only care about whether the patient is affected by the disease. Therefore, my json file is as follows:

json
{
"task": "Task001_ProstateX",
"name": "Prostate",
"target_class": 1,
"test_labels": true,
"labels": {
"0": "background",
"1": "cancer"
},
"modalities": {
"0": "T2",
"1": "ADC",
"2": "DWI",
"3": "Ktrans"
},
"dim": 3
}
The "target_class": 1 indicates that I am only interested in class 1 (i.e., cancer lesions). Could you please advise on how I should modify my dataset or the json file to achieve my goal? Once again, thank you for your valuable contributions.

[NGYLK]

Dear @mibaumgartner
I apologize for disturbing you again. I attempted to map the lesion to label 0 using the following configuration:

json

{
"task": "Task001_ProstateX",
"name": "Prostate",
"target_class": null,
"test_labels": true,
"labels": {
"0": "cancer"
},
"modalities": {
"0": "T2",
"1": "ADC",
"2": "DWI",
"3": "Ktrans"
},
"dim": 3
}
Additionally, I used this instance mapping:

json

{
"instances": {
"1": 0
}
}
I mapped label 1 to 0. I noticed that when handling binary classification problems, this approach is often used, but since I only have one class, I followed the same idea.

My questions are:

1. I found that in my dataset, as well as in the example dataset you provided, when visualizing in ITK-SNAP, the background is labeled as 0, and the lesions are labeled as 1 or 2. Will directly mapping the lesions to 0 in the instances cause confusion or affect the performance?
2. Is this method reasonable for my case?
   Also, I noticed that when running nndet_unpack ${det_data}/Task000D3_Example/preprocessed/D3V001_3d/imagesTr 6Task000D3_Example, it only processes the images. Should I also process the labels accordingly? When processing the labels (including for the example dataset ), I encounter errors.
   I look forward to your help in resolving this issue

[mibaumgartner]

Dear @NGYLK ,

1. yes, your changes are correct. The json file should map 1 (the ID/number in the segmentation file) to class 0 (the class defined in the field in the "labels" json). There is a very distinct difference between the number in the segmentation masks (these only enumerate things in the image, and 0 is always background) and the class of objects (these are encoded in the labels field of the json).

2. You only need to unpack the images in the imagesTr folder.

[NGYLK]

Dear @mibaumgartner
Thank you for your guidance; you've helped solve many issues for me. I've also achieved good results in my project, but I noticed that in my project, the loss function often becomes negative after around epoch 5. After the loss becomes negative, there are occasional improvements in accuracy, but the appearance of negative values happens too early. I suspect there might be overfitting, though my dataset is large enough, with over 1,000 samples, so it's unlikely to overfit after just 5 epochs.

Have you encountered similar situations in your training for prostate cancer detection? What solutions did you find effective? Is changing the loss function a possible solution? If so, do you have any recommended loss functions?

Thank you again for your reply and your contributions.

[mibaumgartner]

Dear @NGYLK ,

it is completely fine that the loss becomes negative, nothing to worry about.
We use the GIoU loss which needs to be minimized, usually defined as 1 - GIoU to keep it positive. Since the +1 part does not actually add any value to the training we skip this part -> as a result the GIoU loss ranges between [-1,0] instead of [0,1].

The improvement of longer schedules reduces over time, especially in the online validation. There is a distribution shift between the online validation (balances foreground and background patches) and the final validation (many background patches) which also needs to be considered, e.g. some improvements only come apparent when evaluating the full patients. (since this is usually quite costly we only do this once at the end of the training, but we ran multiple experiments during the development where we observed this behaviour)

Best,
Michael

[github-actions]

This issue is stale because it has been open for 30 days with no activity.