Doing multiple batch inference?

[nikkkkhil]

I have deployed the yolov3 object detection model on the TF server. I can successfully do inference on the single image now I want to test server capacity for multiple batches of images but when I try to pass multiple images I get an error as "Can not squeeze dim[0], expected a dimension of 1, got 6\n\t [[{{node yolov3/yolo_nms/Squeeze}}]]"
this line throwing an error in models.py
Does this model support multiple batch inferences?

load_imgs = load_images_from_dir("/content/yolov3-tf2/image_data/",416,6)
print(load_imgs.shape)
(6, 416, 416, 3)

request.inputs["input"].CopyFrom(
tf.make_tensor_proto(
load_imgs,
dtype= types_pb2.DT_FLOAT ,
shape=load_imgs.shape
)
)

Can I pass an arbitrary number of images to the model which is trained on different batch size? or is it hardcoded to specific batch size? or am I calling it in a wrong way?

[mauricioCS]

Hi @nikkkkhil !
I was facing the exactly same problem when I tried to predict images divided in batches with size > 1...

During my research I found this issue #92 , marked with tags "inference" and "enhancement", so I don't know if this was already implemented or not.

Now I still studying how to solve this and I found some material that I think might be useful:

• Use Keras models for batch predictions on AI Platform
• Multi-Output Model with TensorFlow Keras Functional API
• TensorFlow Tutorial 07 - Functional API + Multi-output Project

I don't know if this is the correct approach to solve the problem, but if I find the solution, I'll share here.

I hope that you can progress too!

[talenterj]

@mauricioCS have you get any idea

[mauricioCS]

Sorry about my delay @talenterj

I couldn't solve the problem at that time. I've had to run individual image predictions.

My goal was use this model in Google Cloud TPU. To do this I divided my dataset in groups of 8 images, because I was using the TPU v2-8 version with 8 cores, maintaining the 1:1 proportion.