Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DSIFN accuracy #5

Closed
WesleyZhang1991 opened this issue Feb 16, 2022 · 8 comments
Closed

DSIFN accuracy #5

WesleyZhang1991 opened this issue Feb 16, 2022 · 8 comments
Labels
question Further information is requested

Comments

@WesleyZhang1991
Copy link

WesleyZhang1991 commented Feb 16, 2022

Hi wgcban,
I notice that DSIFN-CD dataset has much higher accuracy than BIT[4] (IoU from BIT 52.97% to ChangeFormer 76.48%). On another dataset, LEVIR-CD, the difference is not as large as DSIFN-CD. Could you please explain the main source of the large improvement on DSIFN-CD? e.g. training strategy, data augmentation, model structure...
Thanks
Wesley

@wgcban
Copy link
Owner

wgcban commented Feb 16, 2022

Hi @WesleyZhang1991, yes, we do change the training strategy when training on the DSIFN dataset. As mentioned in the paper, instead of random cropping 256x256 images from 512x512 images during the training, we observed much-improved performance and stable training when the network is training on the training dataset which is created by making non-overlapping patches of 256x256 from original 512x512 images. So, for the DSIFN dataset, first, we created 256x256 non-overlapping patches from the original training data and then trained the network as we did it for the LEVIR-CD dataset. You can find the dataset processing file for DSIFN and LEVIR inside the data_preparation folder: ChangeFormer/data_preparation/

@wgcban wgcban closed this as completed Feb 17, 2022
@wgcban wgcban pinned this issue Feb 20, 2022
@WesleyZhang1991
Copy link
Author

WesleyZhang1991 commented Feb 21, 2022

In the experiments of our own models, we suffer from severe overfitting for DSIFN-CD. The foreground IoU could achieve a IoU of 53% (which is similar to BIT) and then drops to about 50%(or even worse) with more training. The non-overlapping 256x256 patches strategy did not work for our cases.

I notice that the training log of your code does not suffer from overfitting. The test accuracy(e.g. iou_1) improves stably until 200 epochs. I have two questions:
(1) How does ChangeFormer avoid the overfitting problem in DSIFN-CD?
(2) How much improvement did each component of your method make over BIT(or other methods which get about 53% iou_1 for DSIFN-CD)?(e.g. the non-overlapping 256x256 strategy improves 10% iou_1?model structure improves 10% iou_1?)
(3) Why pre-computed non-overlapping 256x256 patches are better than "random cropping 256x256 images from 512x512 images during the training"?

Looking forward to your reply
Wesley

@wgcban
Copy link
Owner

wgcban commented Feb 21, 2022

Hi @WesleyZhang1991,
Yes, I agree with your observation that, we haven't seen any overfitting with our ChangeFormer for the DSIFN dataset. As I remember, I also observed this overfitting scenario at the beginning of my project, and the following modifications are helped me to avoid the problem:

  1. Model architecture matters. Incorporating multi-scale hidden feature difference maps helped to improve the performance and stability of the training as it is able to capture multi-scale changes. I suspect one of the reasons for getting low IoU for BIT is that they do not have this multi-scale feature integration. (As you can see in the codebase, I have multiple versions of ChangeFormer architectures, because I play around with the architecture a lot).
  2. Play around with the optimizer. I observed that changing from adam and SGD to adamW helped to improve the stability of the training process and resulted in better performance metrics.
  3. Finally, moving from random crops to non-overlapping patches also helped. I don't have the exact numbers for these cases, but I remember these changes gradually improved the testing accuracy and stability of training. I will add ablation study later - when I get some free time.

I hope, this answers your question.

Best,
Chaminda.

@wgcban wgcban unpinned this issue Feb 23, 2022
@wgcban wgcban pinned this issue Feb 24, 2022
@WesleyZhang1991
Copy link
Author

WesleyZhang1991 commented Mar 3, 2022

Hi @wgcban
As suggested here, I use ChangeFormer repo for reproducing. Here is my observation:
(1) For testing, the provided DSIFN model gives about 76% iou_1 on DSIFN
(2) For training DSIFN with LEVIR pretrained model, I did not achieve results as the logs in CD_ChangeFormerV6_DSIFN_b16_lr0.00006_adamw_train_test_200_linear_ce_multi_train_True_multi_infer_False_shuffle_AB_False_embed_dim_256. Only around iou_1=45% is achieved

Please check whether DSIFN-CD test data is merged into training data and is used for training. Since they share the same naming format, e.g., '1_0.png' for both train and test.
Sincrely

@wgcban wgcban added the question Further information is requested label Mar 3, 2022
@yuwanting828
Copy link

Hello, I am trying to reproduce your model code for BIT-CD on my side, but it looks incorrect when using WHU-CD dataset as well as DSIFN-CD dataset, what is the situation that the result appears too high than your result? But it looks normal when using the LEVID-CD dataset. Do I need to change the pre-training?

@YangFanghan
Copy link

Hi wgcban.
I have questions about the split part, does train:trainval, val:test, or train:train, val:val have any effect on the final experimental results?
Thanks

@wgcban
Copy link
Owner

wgcban commented Sep 30, 2022

Yes @YangFanghan
Yes.
The LEVIR-CD provides its own train/val/test splits, whereas DSIFN-CD does not. Therefore, in my experiments, I divided the DSIFN-CD dataset into train/val/test splits randomly. Hence, results in different results.

@shenqi0423
Copy link

The images in test set also exist in the train set in your DSIFN dataset, like 6_3. This is the reason for high accuracy。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

5 participants