MSE loss calculation in adaquant optimize layer seems off for ResNet

[hy-chen]

This implies to calculate MSE between relu(conv_out) for conv1 and conv2 layers

CalibTIP/utils/adaquant.py

Line 124 in 69077c9

relu_condition = lambda layer_name: 'conv1' in layer_name or 'conv2' in layer_name

But in ResNet architecture, conv2 is not followed by a direct Relu. Instead it follows by a residual addition, then Relu.

CalibTIP/models/resnet.py

Line 71 in 69077c9

out = self.conv2(out)

CalibTIP/models/resnet.py

Line 79 in 69077c9

out += residual

CalibTIP/models/resnet.py

Line 80 in 69077c9

out = self.relu2(out)

How was this difference justified?

[itayhubara]

Depend on the ResNet model for R18 you are right (basic block) for R50 you are wrong. We aim to give an example here. A better implementation would run on the model's graph and check what is the next layer.