Using ZegCLIP on my custom data

[aliman80]

Hi , First of all extremely grateful for sharing this wonderful work. I want to use this model on my custom data which comprised of 4 classes and dataset comprised of images and masks with json file . How can i use this model for my problem. Will be great help,

Thank you.

Regards

[aliman80]

I have tried to run it on coco stuff dataset but i am getting mIoU near to zero and values of all the classes as nan . Can you give a hint what am i messing up.
Thanks again

[thapaliya19]

Any idea on how to use custom data for binary segmentation?

[ZiqinZhou66]

Any idea on how to use custom data for binary segmentation?

We sincerely appreciate your interest in our work.
These issues might be helpful. The number of classes in the config file could be 2 for binary segmentation or only 1 with setting a threshold according to your method, task, and objective function.

#8 (comment)
#6 (comment)

[ZiqinZhou66]

I have tried to run it on coco stuff dataset but i am getting mIoU near to zero and values of all the classes as nan . Can you give a hint what am i messing up. Thanks again

If I understand correctly, your experimentation with our supplied model on the coco dataset has yielded results of zero or NaN values? Could you kindly furnish us with additional insights into your experimental process and the encountered errors? We have outlined comprehensive steps detailing the preparation of custom datasets, and we have a successful use case that serves as a reproducing reference:
#6 (comment)
#8 (comment)
#7 (comment)

Your additional information will greatly assist us in addressing and resolving the situation effectively.

[aliman80]

Hi many thanks for your quick response, I did the following before the experiment;

In configs/base/datasets/cocostuff_512x512.py on line no 3 i updated the path data_root = 'Path/to/data/coco_stuff164k'
then in configs/coco/vpt_seg_fully_vit-b_512x512_80k_12_100_multi.py i updated the path pretrained = 'Path/to/pretrained/ViT-B-16.pt' with your pretrained model.
Then in https://github.com/ZiqinZhou66/ZegCLIP/blob/main/configs/_base_/models/zegclip.py line 9 pretrained='Path/to/pre trained/RN50.pt' I downloaded weights from the internet and updated this path
then on https://github.com/ZiqinZhou66/ZegCLIP/blob/main/configs/_base_/datasets/dataloader/coco_stuff.py line 108 i changed the suffix to png in place of _labelTrainIds.png.
Then i ran the validation experiment

[aliman80]

2023-08-29 08:03:03,870 - mmseg - INFO - Loaded 5000 images
load checkpoint from local path: /l/users/muhammad.ali/zigclip/ZegCLIP/modd/coco_fully_512_vit_base.pth
[ ] 0/5000, elapsed: 0s, ETA:/home/muhammad.ali/.conda/envs/myenv/lib/python3.8/site-packages/torch/nn/functional.py:3631: UserWarning: Default upsampling behavior when mode=bilinear is changed to align_corners=False since 0.4.0. Please specify align_corners=True if the old behavior is desired. See the documentation of nn.Upsample for details.
warnings.warn(
[>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>] 5000/5000, 11.1 task/s, elapsed: 452s, ETA: 0stest done
/l/users/muhammad.ali/zigclip/ZegCLIP/configs/base/datasets/dataloader/coco_stuff.py:194: RuntimeWarning: Mean of empty slice
unseen_ret_metric: np.round(np.nanmean(unseen_ret_metric_value) * 100, 2)

+++++++++++ Total classes +++++++++++++
per class results:
+------------------+-------+-------+
| Class | IoU | Acc |
+------------------+-------+-------+
| person | 83.15 | 94.6 |
| bicycle | 61.93 | 79.27 |
| car | 59.29 | 80.84 |
| motorcycle | 78.34 | 89.45 |
| airplane | 77.76 | 91.26 |
| bus | 77.47 | 86.93 |
| train | 77.88 | 91.08 |
| truck | 56.91 | 73.98 |
| boat | 57.77 | 78.09 |
| traffic light | 57.84 | 73.57 |
| fire hydrant | 83.27 | 93.39 |
| stop sign | 0.0 | nan |
| parking meter | 0.0 | 0.0 |
| bench | 0.0 | 0.01 |
| bird | 0.0 | 0.0 |
| cat | 0.01 | 0.09 |
| dog | 1.23 | 2.17 |
| horse | 0.38 | 0.61 |
| sheep | 0.0 | 0.0 |
| cow | 0.39 | 0.82 |
| elephant | 0.78 | 2.22 |
| bear | 0.0 | 0.0 |
| zebra | 0.0 | 0.0 |
| giraffe | 0.63 | 1.07 |
| backpack | 0.0 | 0.0 |
| umbrella | 0.0 | nan |
| handbag | 3.21 | 6.24 |
| tie | 0.0 | 0.0 |
| suitcase | 0.0 | nan |
| frisbee | 0.0 | nan |
| skis | 0.0 | 0.0 |
| snowboard | 0.0 | 0.0 |
| sports ball | 0.0 | 0.0 |
| kite | 0.01 | 0.05 |
| baseball bat | 0.0 | 0.0 |
| baseball glove | 0.0 | 0.0 |
| skateboard | 0.01 | 0.06 |
| surfboard | 0.06 | 0.17 |
| tennis racket | 0.4 | 2.5 |
| bottle | 0.0 | 0.0 |
| wine glass | 0.0 | 0.0 |
| cup | 0.0 | 0.0 |
| fork | 0.0 | 0.0 |
| knife | 0.0 | 0.0 |
| spoon | 0.0 | nan |
| bowl | 0.02 | 0.19 |
| banana | 0.0 | 0.0 |
| apple | 0.0 | 0.0 |
| sandwich | 0.0 | 0.01 |
| orange | 0.01 | 0.1 |
| broccoli | 2.91 | 3.52 |
| carrot | 0.02 | 0.02 |
| hot dog | 0.0 | 0.0 |
| pizza | 1.17 | 4.63 |
| donut | 0.0 | 0.0 |
| cake | 0.0 | 0.0 |
| chair | 0.0 | 0.0 |
| couch | 0.0 | 0.0 |
| potted plant | 0.0 | 0.0 |
| bed | 0.0 | 0.0 |
| dining table | 0.7 | 9.64 |
| toilet | 0.03 | 0.05 |
| tv | 0.02 | 0.03 |
| laptop | 0.0 | 0.01 |
| mouse | 0.0 | 0.0 |
| remote | 0.0 | nan |
| keyboard | 0.0 | 0.0 |
| cell phone | 0.0 | nan |
| microwave | 0.0 | nan |
| oven | 0.0 | 0.0 |
| toaster | 0.0 | nan |
| sink | 0.0 | 0.0 |
| refrigerator | 0.0 | 0.0 |
| book | 0.0 | 0.03 |
| clock | 0.37 | 1.46 |
| vase | 0.0 | 0.0 |
| scissors | 0.0 | 0.0 |
| teddy bear | 0.0 | 0.0 |
| hair drier | 0.0 | 0.0 |
| toothbrush | 0.0 | 0.0 |
| banner | 0.0 | 0.0 |
| blanket | 0.0 | 0.0 |
| branch | 0.0 | nan |
| bridge | 0.0 | 0.0 |
| building-other | 0.26 | 6.52 |
| bush | 0.0 | 0.0 |
| cabinet | 0.0 | 0.02 |
| cage | 0.0 | 0.0 |
| cardboard | 0.0 | 0.0 |
| carpet | 0.0 | 0.0 |
| ceiling-other | 0.0 | nan |
| ceiling-tile | 0.0 | 0.0 |
| cloth | 0.0 | 0.0 |
| clothes | 0.0 | 0.0 |
| clouds | 0.03 | 0.66 |
| counter | 0.14 | 0.16 |
| cupboard | 0.0 | 0.0 |
| curtain | 0.08 | 0.14 |
| desk-stuff | 0.0 | 0.0 |
| dirt | 0.0 | 0.02 |
| door-stuff | 0.04 | 0.1 |
| fence | 0.01 | 0.03 |
| floor-marble | 0.0 | 0.0 |
| floor-other | 0.09 | 0.6 |
| floor-stone | 0.0 | 0.0 |
| floor-tile | 0.0 | 0.0 |
| floor-wood | 0.03 | 0.11 |
| flower | 0.0 | 0.0 |
| fog | 0.0 | 0.0 |
| food-other | 0.01 | 0.01 |
| fruit | 0.02 | 0.02 |
| furniture-other | 0.46 | 0.79 |
| grass | 0.7 | 4.18 |
| gravel | 0.0 | 0.0 |
| ground-other | 0.14 | 0.15 |
| hill | 0.0 | 0.0 |
| house | 0.0 | 0.0 |
| leaves | 0.0 | 0.0 |
| light | 0.01 | 0.02 |
| mat | 0.0 | 0.0 |
| metal | 0.21 | 0.54 |
| mirror-stuff | 0.01 | 0.03 |
| moss | 0.0 | 0.0 |
| mountain | 0.03 | 0.04 |
| mud | 0.0 | 0.0 |
| napkin | 0.0 | 0.0 |
| net | 0.0 | 0.0 |
| paper | 0.0 | 0.0 |
| pavement | 0.03 | 0.29 |
| pillow | 0.0 | 0.0 |
| plant-other | 0.0 | 0.0 |
| plastic | 0.77 | 0.87 |
| platform | 0.0 | 0.0 |
| playingfield | 0.0 | 0.0 |
| railing | 0.0 | 0.0 |
| railroad | 0.0 | 0.0 |
| river | 0.0 | 0.0 |
| road | 0.0 | 0.0 |
| rock | 0.0 | 0.0 |
| roof | 0.0 | 0.0 |
| rug | 0.08 | 0.38 |
| salad | 0.0 | 0.0 |
| sand | 0.01 | 0.03 |
| sea | 0.05 | 0.61 |
| shelf | 0.0 | 0.0 |
| sky-other | 0.0 | 0.25 |
| skyscraper | 0.0 | 0.0 |
| snow | 0.04 | 0.19 |
| solid-other | 0.0 | 0.0 |
| stairs | 0.02 | 0.03 |
| stone | 0.0 | 0.0 |
| straw | 0.0 | 0.0 |
| structural-other | 0.0 | 0.0 |
| table | 0.0 | 0.0 |
| tent | 0.0 | 0.0 |
| textile-other | 0.04 | 0.08 |
| towel | 0.0 | 0.0 |
| tree | 0.05 | 1.19 |
| vegetable | 0.0 | 0.0 |
| wall-brick | 0.0 | 0.01 |
| wall-concrete | 0.1 | 6.52 |
| wall-other | 0.0 | 0.0 |
| wall-panel | 0.0 | 0.0 |
| wall-stone | 0.0 | 0.0 |
| wall-tile | 0.08 | 0.15 |
| wall-wood | 0.0 | 0.0 |
| water-other | 0.0 | 0.0 |
| waterdrops | 0.0 | 0.0 |
| window-blind | 0.0 | 0.0 |
| window-other | 0.0 | 0.03 |
| wood | 0.02 | 0.03 |
+------------------+-------+-------+
Summary:
+-------+------+------+
| aAcc | mIoU | mAcc |
+-------+------+------+
| 13.44 | 4.61 | 6.21 |
+-------+------+------+

+++++++++++ Seen classes +++++++++++++
seen per class results:
+------------------+-------+-------+
| Class | IoU | Acc |
+------------------+-------+-------+
| person | 83.15 | 94.6 |
| bicycle | 61.93 | 79.27 |
| car | 59.29 | 80.84 |
| motorcycle | 78.34 | 89.45 |
| airplane | 77.76 | 91.26 |
| bus | 77.47 | 86.93 |
| train | 77.88 | 91.08 |
| truck | 56.91 | 73.98 |
| boat | 57.77 | 78.09 |
| traffic light | 57.84 | 73.57 |
| fire hydrant | 83.27 | 93.39 |
| stop sign | 0.0 | nan |
| parking meter | 0.0 | 0.0 |
| bench | 0.0 | 0.01 |
| bird | 0.0 | 0.0 |
| cat | 0.01 | 0.09 |
| dog | 1.23 | 2.17 |
| horse | 0.38 | 0.61 |
| sheep | 0.0 | 0.0 |
| cow | 0.39 | 0.82 |
| elephant | 0.78 | 2.22 |
| bear | 0.0 | 0.0 |
| zebra | 0.0 | 0.0 |
| giraffe | 0.63 | 1.07 |
| backpack | 0.0 | 0.0 |
| umbrella | 0.0 | nan |
| handbag | 3.21 | 6.24 |
| tie | 0.0 | 0.0 |
| suitcase | 0.0 | nan |
| frisbee | 0.0 | nan |
| skis | 0.0 | 0.0 |
| snowboard | 0.0 | 0.0 |
| sports ball | 0.0 | 0.0 |
| kite | 0.01 | 0.05 |
| baseball bat | 0.0 | 0.0 |
| baseball glove | 0.0 | 0.0 |
| skateboard | 0.01 | 0.06 |
| surfboard | 0.06 | 0.17 |
| tennis racket | 0.4 | 2.5 |
| bottle | 0.0 | 0.0 |
| wine glass | 0.0 | 0.0 |
| cup | 0.0 | 0.0 |
| fork | 0.0 | 0.0 |
| knife | 0.0 | 0.0 |
| spoon | 0.0 | nan |
| bowl | 0.02 | 0.19 |
| banana | 0.0 | 0.0 |
| apple | 0.0 | 0.0 |
| sandwich | 0.0 | 0.01 |
| orange | 0.01 | 0.1 |
| broccoli | 2.91 | 3.52 |
| carrot | 0.02 | 0.02 |
| hot dog | 0.0 | 0.0 |
| pizza | 1.17 | 4.63 |
| donut | 0.0 | 0.0 |
| cake | 0.0 | 0.0 |
| chair | 0.0 | 0.0 |
| couch | 0.0 | 0.0 |
| potted plant | 0.0 | 0.0 |
| bed | 0.0 | 0.0 |
| dining table | 0.7 | 9.64 |
| toilet | 0.03 | 0.05 |
| tv | 0.02 | 0.03 |
| laptop | 0.0 | 0.01 |
| mouse | 0.0 | 0.0 |
| remote | 0.0 | nan |
| keyboard | 0.0 | 0.0 |
| cell phone | 0.0 | nan |
| microwave | 0.0 | nan |
| oven | 0.0 | 0.0 |
| toaster | 0.0 | nan |
| sink | 0.0 | 0.0 |
| refrigerator | 0.0 | 0.0 |
| book | 0.0 | 0.03 |
| clock | 0.37 | 1.46 |
| vase | 0.0 | 0.0 |
| scissors | 0.0 | 0.0 |
| teddy bear | 0.0 | 0.0 |
| hair drier | 0.0 | 0.0 |
| toothbrush | 0.0 | 0.0 |
| banner | 0.0 | 0.0 |
| blanket | 0.0 | 0.0 |
| branch | 0.0 | nan |
| bridge | 0.0 | 0.0 |
| building-other | 0.26 | 6.52 |
| bush | 0.0 | 0.0 |
| cabinet | 0.0 | 0.02 |
| cage | 0.0 | 0.0 |
| cardboard | 0.0 | 0.0 |
| carpet | 0.0 | 0.0 |
| ceiling-other | 0.0 | nan |
| ceiling-tile | 0.0 | 0.0 |
| cloth | 0.0 | 0.0 |
| clothes | 0.0 | 0.0 |
| clouds | 0.03 | 0.66 |
| counter | 0.14 | 0.16 |
| cupboard | 0.0 | 0.0 |
| curtain | 0.08 | 0.14 |
| desk-stuff | 0.0 | 0.0 |
| dirt | 0.0 | 0.02 |
| door-stuff | 0.04 | 0.1 |
| fence | 0.01 | 0.03 |
| floor-marble | 0.0 | 0.0 |
| floor-other | 0.09 | 0.6 |
| floor-stone | 0.0 | 0.0 |
| floor-tile | 0.0 | 0.0 |
| floor-wood | 0.03 | 0.11 |
| flower | 0.0 | 0.0 |
| fog | 0.0 | 0.0 |
| food-other | 0.01 | 0.01 |
| fruit | 0.02 | 0.02 |
| furniture-other | 0.46 | 0.79 |
| grass | 0.7 | 4.18 |
| gravel | 0.0 | 0.0 |
| ground-other | 0.14 | 0.15 |
| hill | 0.0 | 0.0 |
| house | 0.0 | 0.0 |
| leaves | 0.0 | 0.0 |
| light | 0.01 | 0.02 |
| mat | 0.0 | 0.0 |
| metal | 0.21 | 0.54 |
| mirror-stuff | 0.01 | 0.03 |
| moss | 0.0 | 0.0 |
| mountain | 0.03 | 0.04 |
| mud | 0.0 | 0.0 |
| napkin | 0.0 | 0.0 |
| net | 0.0 | 0.0 |
| paper | 0.0 | 0.0 |
| pavement | 0.03 | 0.29 |
| pillow | 0.0 | 0.0 |
| plant-other | 0.0 | 0.0 |
| plastic | 0.77 | 0.87 |
| platform | 0.0 | 0.0 |
| playingfield | 0.0 | 0.0 |
| railing | 0.0 | 0.0 |
| railroad | 0.0 | 0.0 |
| river | 0.0 | 0.0 |
| road | 0.0 | 0.0 |
| rock | 0.0 | 0.0 |
| roof | 0.0 | 0.0 |
| rug | 0.08 | 0.38 |
| salad | 0.0 | 0.0 |
| sand | 0.01 | 0.03 |
| sea | 0.05 | 0.61 |
| shelf | 0.0 | 0.0 |
| sky-other | 0.0 | 0.25 |
| skyscraper | 0.0 | 0.0 |
| snow | 0.04 | 0.19 |
| solid-other | 0.0 | 0.0 |
| stairs | 0.02 | 0.03 |
| stone | 0.0 | 0.0 |
| straw | 0.0 | 0.0 |
| structural-other | 0.0 | 0.0 |
| table | 0.0 | 0.0 |
| tent | 0.0 | 0.0 |
| textile-other | 0.04 | 0.08 |
| towel | 0.0 | 0.0 |
| tree | 0.05 | 1.19 |
| vegetable | 0.0 | 0.0 |
| wall-brick | 0.0 | 0.01 |
| wall-concrete | 0.1 | 6.52 |
| wall-other | 0.0 | 0.0 |
| wall-panel | 0.0 | 0.0 |
| wall-stone | 0.0 | 0.0 |
| wall-tile | 0.08 | 0.15 |
| wall-wood | 0.0 | 0.0 |
| water-other | 0.0 | 0.0 |
| waterdrops | 0.0 | 0.0 |
| window-blind | 0.0 | 0.0 |
| window-other | 0.0 | 0.03 |
| wood | 0.02 | 0.03 |
+------------------+-------+-------+
Seen Summary:
+-------+------+------+
| aAcc | mIoU | mAcc |
+-------+------+------+
| 13.44 | 4.61 | 6.21 |
+-------+------+------+

+++++++++++ Unseen classes +++++++++++++
unseen per class results:
+-------+-----+-----+
| Class | IoU | Acc |
+-------+-----+-----+
+-------+-----+-----+
Unseen Summary:
+-------+------+------+
| aAcc | mIoU | mAcc |
+-------+------+------+
| 13.44 | nan | nan |