No signal from onnx from Pytorch #74

SahPet · 2023-07-04T06:22:43Z

SahPet
Jul 4, 2023

Hi!

I get no signal from an onnx file exported from pytorch with the script below (a regular timm DeeplabV3-Resnet18 model). I get great and proper signals with similar models exported from Matlab onnx export. Any idea why? I can send the onnx file and an example image if needed.

Onnx export script:

import torch
import os
import glob
from torch import nn, optim
import segmentation_models_pytorch as smp
import argparse
import datetime

parser = argparse.ArgumentParser()
parser.add_argument("--encoder", type=str, default="tu-res2net50_26w_4s", help="Encoder type for the model")
parser.add_argument("--model_type", type=str, default="DeepLabV3Plus", help="Model type")
parser.add_argument("--checkpoint_dir", type=str, default="checkpoints", help="Directory for saving checkpoints")
parser.add_argument("--num_classes", type=int, default=2, help="Number of classes")
parser.add_argument("--learning_rate", type=float, default=3e-5, help="Learning rate")
parser.add_argument("--image_size", type=int, default=2048, help="Image input size.")
parser.add_argument("--encoder_output_stride", type=int, default=16, help="encoder_output_stride")
parser.add_argument("--decoder_atrous_rates", type=str, default='12,24,36', help="decoder_atrous_rates")
args = parser.parse_args()
print(f'Image size: {args.image_size}')

decoder_atrous_rates = [int(item) for item in args.decoder_atrous_rates.split(',')]

class FocalLoss(nn.Module):
    def __init__(self, alpha=0.25, gamma=2, reduction='mean'):
        super(FocalLoss, self).__init__()
        self.alpha = alpha
        self.gamma = gamma
        self.reduction = reduction

    def forward(self, inputs, targets):
        ce_loss = nn.CrossEntropyLoss(reduction='none')(inputs, targets)
        pt = torch.exp(-ce_loss)
        focal_loss = self.alpha * (1 - pt) ** self.gamma * ce_loss

        if self.reduction == 'mean':
            return focal_loss.mean()
        elif self.reduction == 'sum':
            return focal_loss.sum()
        else:
            return focal_loss

class_names = ['Background', 'Tumor']  # Add the class names you want to use


ENCODER = args.encoder
ENCODER_WEIGHTS = 'imagenet'
CLASSES = class_names
ACTIVATION = 'sigmoid'  # could be None for logits or 'softmax2d' for multiclass segmentation
ENCODER_OUTPUT_STRIDE = args.encoder_output_stride

model_dict = {
    "Unet": smp.Unet,
    "UnetPlusPlus": smp.UnetPlusPlus,
    "MAnet": smp.MAnet,
    "Linknet": smp.Linknet,
    "FPN": smp.FPN,
    "PSPNet": smp.PSPNet,
    "PAN": smp.PAN,
    "DeepLabV3": smp.DeepLabV3,
    "DeepLabV3Plus": smp.DeepLabV3Plus,
}

model_class = model_dict.get(args.model_type)
if model_class is None:
    raise ValueError(f"Invalid model type specified: {args.model_type}")

# Creating the model...
print("Creating the model...")
if args.model_type == "DeepLabV3Plus":
    model = model_class(
        encoder_name=ENCODER,
        encoder_weights=ENCODER_WEIGHTS,
        encoder_output_stride=ENCODER_OUTPUT_STRIDE,
        decoder_atrous_rates=decoder_atrous_rates,
        classes=len(CLASSES),
        activation=ACTIVATION,
    )
else:
    model = model_class(
        encoder_name=ENCODER,
        encoder_weights=ENCODER_WEIGHTS,
        classes=len(CLASSES),
        activation=ACTIVATION,
    )
print("Model created.")

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model.to(device)

learning_rate = args.learning_rate
optimizer = optim.Adam(model.parameters(), lr=learning_rate)
checkpoint_dir = args.checkpoint_dir
os.makedirs(checkpoint_dir, exist_ok=True)
checkpoints = glob.glob(os.path.join(checkpoint_dir, "*.pth"))

criterion = FocalLoss(alpha=0.25, gamma=2)

if checkpoints:
    latest_checkpoint = max(checkpoints, key=os.path.getmtime)
else:
    latest_checkpoint = None

if latest_checkpoint and os.path.exists(latest_checkpoint):
    checkpoint = torch.load(latest_checkpoint)
    model.load_state_dict(checkpoint["model_state_dict"])
    optimizer.load_state_dict(checkpoint["optimizer_state_dict"])
    start_epoch = checkpoint["epoch"]
    print(f"Loaded checkpoint '{latest_checkpoint}' (epoch {start_epoch})")
else:
    start_epoch = 0
    print("No checkpoints found.")

# Get current date
current_date = datetime.datetime.now()
# Format the date
formatted_date = current_date.strftime("%d%m%y")  # European format day/month/year

onnx_model_path = os.path.join(checkpoint_dir, f'{args.model_type}_{args.encoder}_{formatted_date}.onnx')

# Export the model to an ONNX file
model.eval()  # set model to evaluation mode
example = torch.rand(1, 3, args.image_size, args.image_size).to(device)  # example input

# Convert args.decoder_atrous_rates to a string and replace commas with dashes
decoder_atrous_rates_str = args.decoder_atrous_rates.replace(',', '-')

# Use this string in the filename
onnx_model_path = os.path.join(checkpoint_dir, f'{args.model_type}_{args.encoder}_Stride{args.encoder_output_stride}_AtrousRates{decoder_atrous_rates_str}_ImageSize{args.image_size}_{formatted_date}.onnx')

torch.onnx.export(model,
                  example,
                  onnx_model_path,
                  export_params=True,
                  opset_version=11,
                  do_constant_folding=True,
                  input_names=['input'],
                  output_names=['output'],
                  dynamic_axes={'input' : {0 : 'batch_size'}, 'output' : {0 : 'batch_size'}})
print("Saving ONNX file.")

I get one warning when exporting the onnx, might that be a hint to what's wrong?

venv3\lib\site-packages\segmentation_models_pytorch\base\model.py:16: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if h % output_stride != 0 or w % output_stride != 0:
============= Diagnostic Run torch.onnx.export version 2.0.1+cu118 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

Saving ONNX file.

Answered by andreped

Aug 4, 2023

scale-factor should be 0.00392156862 (which corresponds to 1/255) instead of 1, as you are 0-1 normalizing the images. The scale-factor is multiplied by the image. See the SegmentationNetwork PO in your FPL file.

View full answer

andreped · 2023-07-04T06:56:48Z

andreped
Jul 4, 2023
Maintainer

Hello, @SahPet!

Sigmoid/Softmax issue

I see that you have used sigmoid as the final activation layer. For this I assume that you are doing binary segmentation (single-class detection), right?

Using sigmoid, this results in a single-channel output. AFAIK, FAST neglects the first channel by default, as FAST assumes that the first channel is the background class. This is because normally our models use the softmax activation layer instead, which in your use case results in a two channel output, where the first channel is background and the second is the class of interest.

However, it should still work if you have used sigmoid. For the HeatmapRenderer we have a mechanism to enable showing the zero-class (see here). Can't see that we have the same for the SegmentationRenderer (see here), but maybe that is not the main issue.

Simple fix

A very simple fix to this problem, as this is the export script, is to replace the sigmoid activation layer with the softmax2d layer, as commented in your code here:

ACTIVATION = 'sigmoid'  # could be None for logits or 'softmax2d' for multiclass segmentation

Preprocessing issue

If that does not resolve the issue, the preprocessing likely does not match between deployment in FastPathology and what you do in PyTorch. Quite often in PyTorch, when using pretrained backbones such as ResNet-18 which are pretrained on ImageNet, a specific type of preprocessing is performed on all image patches:

Image patches may have been resized to a different size
Intensity standardization using predefined RGB mean and standard deviations calculated from the ImageNet dataset are used.

If you could send me the code used for training, it should be easier for me to debug this further.

10 replies

SahPet Aug 4, 2023
Author

There is preprocessing of the images during training, yes. Here is the relevant code:

class NormalizeImage(A.ImageOnlyTransform):
    def __init__(self, always_apply=True, p=1):
        super(NormalizeImage, self).__init__(always_apply, p)

    def apply(self, img, **params):
        min_value = np.min(img)
        max_value = np.max(img)
        if max_value - min_value > 0:  # Avoid division by zero
            return (img - min_value) / (max_value - min_value)
        else:
            return img  # Return the original image if max_value == min_value


def get_augmentation_pipeline(args):
    return A.Compose([
        A.Resize(width=args.image_size, height=args.image_size),
        A.VerticalFlip(p=args.vertical_flip_p),
        A.HorizontalFlip(p=args.horizontal_flip_p),

[...]
        NormalizeImage(),
        ToTensorV2()
    ], additional_targets={'image': 'image', 'mask': 'mask'})


class CustomDataset(Dataset):
    def __init__(self, images_path, labels_path, transforms=None):
        self.images_path = images_path
        self.labels_path = labels_path
        self.image_files = os.listdir(images_path)
        self.transforms = transforms

    def __len__(self):
        return len(self.image_files)

    def __getitem__(self, idx):
        img_name = self.image_files[idx]
        img_path = os.path.join(self.images_path, img_name)
        img_name_base = os.path.splitext(img_name)[0]
        label_path = os.path.join(self.labels_path, f"Labels_{img_name_base}.tif")

        image = cv2.imread(img_path)
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
        label = cv2.imread(label_path, cv2.IMREAD_GRAYSCALE)

    if self.transforms:
        augmented = self.transforms(image=image, mask=label)
        image = augmented['image']
        label = augmented['mask']

    # Convert the image and label to float tensors
    image = image.float()
    label = label.float()

    return image, label

SahPet Aug 4, 2023
Author

Is it possible to reproduce this preprocessing in the pipeline file?

Here is the pipepline file:

PipelineName "Mitosis Pytorch Aug 2023"
PipelineDescription "Mitosis Pytorch Aug 2023"
PipelineInputData WSI "Whole-slide image"
PipelineOutputData segmentation stitcher 0
Attribute classes "Background;Mitosis"

### Processing chain
ProcessObject tissueSeg TissueSegmentation
Attribute filter-zeros true
Attribute threshold 100
Input 0 WSI

ProcessObject patch PatchGenerator
Attribute patch-size 1024 1024
Attribute patch-magnification 40
Attribute patch-overlap 0.05
Attribute mask-threshold 0.05
Input 0 WSI
Input 1 tissueSeg 0

ProcessObject network SegmentationNetwork
Attribute scale-factor 1.0
Attribute model "C:\Users\henri\fastpathology\models\DeepLabV3Plus_tu-resnet18_Stride16_AtrousRates12-24-36_ImageSize1024_040823.onnx"
Input 0 patch 0

ProcessObject stitcher PatchStitcher
Input 0 network 0

### Renderers
Renderer imgRenderer ImagePyramidRenderer
Input 0 WSI

Renderer segRenderer SegmentationRenderer
Attribute opacity 0.5
Attribute border-opacity 1.0
Attribute border-radius 10.0
Input 0 stitcher 0

SahPet Aug 4, 2023
Author

Here is some of the --verbose prints when running the pytorch onnx file:

INFO [58580] Running ONNX runtime ..
INFO [58580] Finished run ONNX runtime
INFO [58580] EXECUTING EmptyProcessObject because PO is modified.
INFO [58580] EXECUTING TensorToSegmentation because PO is modified.
INFO [58580] Ignoring channels: -1
INFO [58580] EXECUTING PatchStitcher because PO has new input data.
INFO [58580] Stitching 2D data 23 46
INFO [58580] EXECUTING SegmentationRenderer because PO has new input data.
INFO [58580] EXECUTING RunLambda because PO has new input data.
INFO [58580] EXECUTING SegmentationNetwork because PO has new input data.
INFO [58580] ONNXRuntime: Creating memory info..
INFO [58580] Running ONNX runtime ..
INFO [58580] Finished run ONNX runtime
INFO [58580] EXECUTING EmptyProcessObject because PO is modified.
INFO [58580] EXECUTING TensorToSegmentation because PO is modified.
INFO [58580] Ignoring channels: -1
INFO [58580] EXECUTING PatchStitcher because PO has new input data.
INFO [58580] Stitching 2D data 17 47
INFO [58580] EXECUTING SegmentationRenderer because PO has new input data.
INFO [58580] EXECUTING RunLambda because PO has new input data.
INFO [58580] EXECUTING SegmentationNetwork because PO has new input data.
INFO [58580] ONNXRuntime: Creating memory info..
INFO [58580] Running ONNX runtime ..
INFO [58580] Finished run ONNX runtime
INFO [58580] EXECUTING EmptyProcessObject because PO is modified.
INFO [58580] EXECUTING TensorToSegmentation because PO is modified.
INFO [58580] Ignoring channels: -1
INFO [58580] EXECUTING PatchStitcher because PO has new input data.
INFO [58580] Stitching 2D data 18 47
INFO [58580] EXECUTING SegmentationRenderer because PO has new input data.
INFO [58580] EXECUTING RunLambda because PO has new input data.
INFO [58580] EXECUTING SegmentationNetwork because PO has new input data.
INFO [58580] ONNXRuntime: Creating memory info..
INFO [58580] Running ONNX runtime ..
INFO [58580] Finished run ONNX runtime
INFO [58580] EXECUTING EmptyProcessObject because PO is modified.
INFO [58580] EXECUTING TensorToSegmentation because PO is modified.
INFO [58580] Ignoring channels: -1
INFO [58580] EXECUTING PatchStitcher because PO has new input data.
INFO [58580] Stitching 2D data 19 47
INFO [58580] EXECUTING SegmentationRenderer because PO has new input data.
Saving ImagePyramid data to C:/Users/henri/fastpathology/projects/mitosisTestSoftmax//results/examplefile#41/Mitosis Pytorch Aug 2023/segmentation
INFO [56880] EXECUTING EmptyProcessObject because PO is modified.
INFO [56880] EXECUTING TIFFImagePyramidExporter because PO is modified.
Writing timestamping..2023-08-04-142605
canceled..
done waiting
stopping pipeline
done
removing renderers..
done
stopping pipeline
done
removing renderers..
done
INFO [56880] Destroying ImagePyramidRenderer in THREAD: 56880
INFO [56880] Pyramid cleared
Attribute disabled false
Set attribute disabled to false for object SegmentationRenderer
Attribute opacity 0.5
Set attribute opacity to 0.5 for object SegmentationRenderer
Attribute border-opacity 1
Set attribute border-opacity to 1 for object SegmentationRenderer
Attribute border-radius 10
Set attribute border-radius to 10 for object SegmentationRenderer
Attribute label-colors "1" "green" "2" "blue" "3" "red" "4" "yellow" "5" "cyan" "6" "magenta" "7" "brown" "255" "cyan"
Set attribute label-colors to "1" "green" "2" "blue" "3" "red" "4" "yellow" "5" "cyan" "6" "magenta" "7" "brown" "255" "cyan" for object SegmentationRenderer

Background;Mitosis
Attribute disabled false
Set attribute disabled to false for object SegmentationRenderer
Attribute opacity 0
Set attribute opacity to 0 for object SegmentationRenderer
Attribute border-opacity 0
Set attribute border-opacity to 0 for object SegmentationRenderer
Attribute border-radius 3
Set attribute border-radius to 3 for object SegmentationRenderer
Attribute label-colors "1" "green" "2" "blue" "3" "red" "4" "yellow" "5" "cyan" "6" "magenta" "7" "brown" "255" "cyan"
Set attribute label-colors to "1" "green" "2" "blue" "3" "red" "4" "yellow" "5" "cyan" "6" "magenta" "7" "brown" "255" "cyan" for object SegmentationRenderer

Background;Mitosis
Attribute disabled false
Set attribute disabled to false for object SegmentationRenderer
Attribute opacity 0.5
Set attribute opacity to 0.5 for object SegmentationRenderer
Attribute border-opacity 1
Set attribute border-opacity to 1 for object SegmentationRenderer
Attribute border-radius 10
Set attribute border-radius to 10 for object SegmentationRenderer
Attribute label-colors "1" "green" "2" "blue" "3" "red" "4" "yellow" "5" "cyan" "6" "magenta" "7" "brown" "255" "cyan"
Set attribute label-colors to "1" "green" "2" "blue" "3" "red" "4" "yellow" "5" "cyan" "6" "magenta" "7" "brown" "255" "cyan" for object SegmentationRenderer

Background;Mitosis
INFO [58580] EXECUTING EmptyProcessObject because PO is modified.
INFO [58580] EXECUTING TIFFImagePyramidImporter because PO is modified.
INFO [58580] Reading WSI with TIFF
INFO [58580] Level 0 has size 117760 48128 and tile size: 912 912 channels 1
INFO [58580] Level 1 has size 58880 24064 and tile size: 912 912 channels 1
INFO [58580] Level 2 has size 29440 12032 and tile size: 912 912 channels 1
INFO [58580] Level 3 has size 14720 6016 and tile size: 912 912 channels 1
INFO [58580] Level 4 has size 7360 3008 and tile size: 912 912 channels 1
INFO [58580] Spacing from TIFF was 0.00025 0.00025
INFO [58580] EXECUTING TIFFImagePyramidImporter because PO is modified.
INFO [58580] Reading WSI with TIFF
INFO [58580] Level 0 has size 117760 48128 and tile size: 912 912 channels 1
INFO [58580] Level 1 has size 58880 24064 and tile size: 912 912 channels 1
INFO [58580] Level 2 has size 29440 12032 and tile size: 912 912 channels 1
INFO [58580] Level 3 has size 14720 6016 and tile size: 912 912 channels 1
INFO [58580] Level 4 has size 7360 3008 and tile size: 912 912 channels 1
INFO [58580] Spacing from TIFF was 0.00025 0.00025
INFO [58580] EXECUTING TIFFImagePyramidImporter because PO is modified.
INFO [58580] Reading WSI with TIFF
INFO [58580] Level 0 has size 117760 48128 and tile size: 912 912 channels 1
INFO [58580] Level 1 has size 58880 24064 and tile size: 912 912 channels 1
INFO [58580] Level 2 has size 29440 12032 and tile size: 912 912 channels 1
INFO [58580] Level 3 has size 14720 6016 and tile size: 912 912 channels 1
INFO [58580] Level 4 has size 7360 3008 and tile size: 912 912 channels 1
INFO [58580] Spacing from TIFF was 0.00025 0.00025
INFO [58580] EXECUTING ImagePyramidRenderer because PO is modified.
INFO [58580] EXECUTING SegmentationRenderer because PO is modified.
INFO [58580] EXECUTING SegmentationRenderer because PO is modified.
INFO [58580] EXECUTING SegmentationRenderer because PO is modified.
INFO [13108] EXECUTING EmptyProcessObject because PO is modified.
INFO [13108] EXECUTING ImageSharpening because PO is modified.
INFO [13108] EXECUTING EmptyProcessObject because PO is modified.
INFO [13108] EXECUTING ImageSharpening because PO is modified.
INFO [13108] EXECUTING EmptyProcessObject because PO is modified.
INFO [13108] EXECUTING ImageSharpening because PO is modified.
INFO [13108] EXECUTING EmptyProcessObject because PO is modified.
INFO [13108] EXECUTING ImageSharpening because PO is modified.
INFO [13108] EXECUTING EmptyProcessObject because PO is modified.
INFO [13108] EXECUTING ImageSharpening because PO is modified.
INFO [13108] EXECUTING EmptyProcessObject because PO is modified.
INFO [13108] EXECUTING ImageSharpening because PO is modified.
WARNING [56148] TIFF: TIFFFillTile: 702453643000: Invalid tile byte count, tile 7
WARNING [56148] TIFF: TIFFFillTile: 702453643000: Invalid tile byte count, tile 7
WARNING [56148] TIFF: TIFFFillTile: 702453643000: Invalid tile byte count, tile 7
INFO [13108] EXECUTING EmptyProcessObject because PO is modified.
INFO [13108] EXECUTING ImageSharpening because PO is modified.
WARNING [56148] TIFF: TIFFFillTile: 702453643000: Invalid tile byte count, tile 7
WARNING [56148] TIFF: TIFFFillTile: 702453643000: Invalid tile byte count, tile 7
INFO [13108] EXECUTING EmptyProcessObject because PO is modified.
INFO [13108] EXECUTING ImageSharpening because PO is modified.
WARNING [56148] TIFF: TIFFFillTile: 702453643000: Invalid tile byte count, tile 7
WARNING [56148] TIFF: TIFFFillTile: 702453643000: Invalid tile byte count, tile 7
WARNING [56148] TIFF: TIFFFillTile: 702453643000: Invalid tile byte count, tile 7
WARNING [56148] TIFF: TIFFFillTile: 702453643000: Invalid tile byte count, tile 7
WARNING [56148] TIFF: TIFFFillTile: 702453643000: Invalid tile byte count, tile 7
WARNING [58716] TIFF: TIFFFillTile: 702522847352: Invalid tile byte count, tile 15WARNING [56148] TIFF: TIFFFillTile: 702453643000: Invalid tile byte count, tile 7

WARNING [58716] TIFF: TIFFFillTile: 702522847352: Invalid tile byte count, tile 15
WARNING [56148WARNING [58716] TIFF: TIFFFillTile: ] TIFF: TIFFFillTile: 702453643000: Invalid tile byte count, tile 7702522847352: Invalid tile byte count, tile 15

andreped Aug 4, 2023
Maintainer

scale-factor should be 0.00392156862 (which corresponds to 1/255) instead of 1, as you are 0-1 normalizing the images. The scale-factor is multiplied by the image. See the SegmentationNetwork PO in your FPL file.

Answer selected by SahPet

SahPet Aug 4, 2023
Author

That seemed to work, thank you ❤️ It's a bit surprising, since the matlab/MIB models are also trained with 0-1 normalization - any ideas why that might be?

andreped Aug 4, 2023
Maintainer

I would assume MIB does the same intensity normalization as FAST, which in turn now also should be the same as in PyTorch.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

No signal from onnx from Pytorch #74

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 10 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

No signal from onnx from Pytorch #74

SahPet Jul 4, 2023

Replies: 1 comment · 10 replies

andreped Jul 4, 2023 Maintainer

Sigmoid/Softmax issue

Simple fix

Preprocessing issue

SahPet Aug 4, 2023 Author

SahPet Aug 4, 2023 Author

SahPet Aug 4, 2023 Author

andreped Aug 4, 2023 Maintainer

SahPet Aug 4, 2023 Author

andreped Aug 4, 2023 Maintainer

SahPet
Jul 4, 2023

Replies: 1 comment 10 replies

andreped
Jul 4, 2023
Maintainer

SahPet Aug 4, 2023
Author

SahPet Aug 4, 2023
Author

SahPet Aug 4, 2023
Author

andreped Aug 4, 2023
Maintainer

SahPet Aug 4, 2023
Author

andreped Aug 4, 2023
Maintainer