Practices

From https://developer.nvidia.com/blog/int8-inference-autonomous-vehicles-tensorrt.

• Create a separate calibration dataset that shouldn’t overlap with the training, validation or test datasets, in order to avoid a situation where the calibrated model only works well on the these datasets.
• The calibration batch size defines how many calibration images get processed at the same time for collecting the input distribution required to compute the correct scaling factor. The calibration batch size can be different from the max batch size parameter for inference. Using larger calibration batch size usually speeds up the calibration process and I recommend using the maximum batch size that can fit in GPU memory.