Instance segmentation with MMOCR

[apacha]

Hi, we would like to perform instance segmentation with MMOCR on a word-level like this:

We would like to find and recognize all words in the image, while at the same time also getting the instance segmentation masks for each word (i.e., all letters in "Adagio" or "Allegro" would be of the same class, thus receive the same instance segmentation color).

Is this possible with MMOCR? It seems like you are using MMDet underneath and some approaches like MaskRCN allow to obtain instance segmentation masks like these. However, for us it is currently not clear how we would be able to get those masks, because we only receive the bounding boxes / polygons and the transcription, but not the pixel masks for each detected word.

I've found #1478, but we don't need to distinguish between different classes, however we are interested in different instances!

Any help would be highly appreciated.

[gaotongxiao]

Is this possible with MMOCR? It seems like you are using MMDet underneath and some approaches like MaskRCN allow to obtain instance segmentation masks like these.

It's true. MMOCR calls MMDet's MaskRCNN via MMDetWrapper, which converts the masks into polygons/bboxes in adapt_predictions:

mmocr/mmocr/models/textdet/detectors/mmdet_wrapper.py

Line 90 in e067dde

def adapt_predictions(self, data: MMDET_SampleList,

If you just want to obtain the masks, you can dump the original data_samples of MMDet in this method. Note that MMOCR can't actually handle the masks yet, so you can't visualize or further process these masks with MMOCR's utils.