Batch inference with multiple models in parallel?

[KDeser]

I'm gearing up for a large inference job involving millions of images that each need to be processed through multiple yolov8 models. I already have code implemented that uses multithreading to efficiently load the images into memory, placing cv2/numpy arrays into a Queue of lists where each list is a single batch of N images that have been resized to 640x640.

Now what I'm struggling with is how to feed each batch into multiple yolov8 models at the same time, rather than looping over the models one by one. Everything is happening on a single Windows computer with a 16 GB GPU, 8 core CPU, and 64 GB of memory, and the models are a mix of yolov8n, yolov8n, and yolov8s-seg.

Looking for hints or code fragments, or links to repos that implement this. Anything to get me started would help!

Thanks!

[glenn-jocher]

@KDeser hello!

Thank you for reaching out with your question. It sounds like you have an exciting and challenging project ahead! To efficiently run batch inference with multiple YOLOv8 models in parallel, you can leverage Python's threading or multiprocessing modules to handle concurrent model inference. Given your setup, here are some steps and code snippets to help you get started:

Thread-Safe Inference

First, ensure that each thread or process instantiates its own YOLO model to avoid race conditions. Here's an example using the threading module:

import threading
from queue import Queue
from ultralytics import YOLO

# Function to perform inference
def thread_safe_inference(model_path, image_batch):
    model = YOLO(model_path)
    results = model.predict(image_batch)
    # Process results
    return results

# Function to handle batches
def process_batches(model_paths, image_queue):
    while not image_queue.empty():
        image_batch = image_queue.get()
        threads = []
        for model_path in model_paths:
            thread = threading.Thread(target=thread_safe_inference, args=(model_path, image_batch))
            threads.append(thread)
            thread.start()
        
        for thread in threads:
            thread.join()

# Example usage
if __name__ == "__main__":
    # Paths to your models
    model_paths = ["yolov8n.pt", "yolov8n.pt", "yolov8s-seg.pt"]
    
    # Queue of image batches
    image_queue = Queue()
    # Assuming you have a function to load and preprocess your images into batches
    for batch in load_image_batches():
        image_queue.put(batch)
    
    # Process the batches
    process_batches(model_paths, image_queue)

Multiprocessing for Better Performance

For better performance, especially on a multi-core CPU, consider using the multiprocessing module. This avoids the Global Interpreter Lock (GIL) limitations of threading:

import multiprocessing
from queue import Queue
from ultralytics import YOLO

# Function to perform inference
def process_batch(model_path, image_batch):
    model = YOLO(model_path)
    results = model.predict(image_batch)
    # Process results
    return results

# Function to handle batches
def process_batches(model_paths, image_queue):
    while not image_queue.empty():
        image_batch = image_queue.get()
        processes = []
        for model_path in model_paths:
            process = multiprocessing.Process(target=process_batch, args=(model_path, image_batch))
            processes.append(process)
            process.start()
        
        for process in processes:
            process.join()

# Example usage
if __name__ == "__main__":
    # Paths to your models
    model_paths = ["yolov8n.pt", "yolov8n.pt", "yolov8s-seg.pt"]
    
    # Queue of image batches
    image_queue = Queue()
    # Assuming you have a function to load and preprocess your images into batches
    for batch in load_image_batches():
        image_queue.put(batch)
    
    # Process the batches
    process_batches(model_paths, image_queue)

Additional Tips

1. Ensure Thread Safety: Always instantiate the YOLO model within the thread or process to avoid shared state issues.
2. Optimize Batch Size: Experiment with different batch sizes to find the optimal balance between GPU memory usage and inference speed.
3. Monitor Resources: Keep an eye on your GPU and CPU usage to ensure you are not overloading your system.

For more detailed guidance on thread-safe inference, you can refer to our YOLO Thread-Safe Inference Guide.

Feel free to reach out if you have any more questions or need further assistance. Happy coding! 🚀

[Leminhhuy2386]

In the Multiprocessing case, the process will copy an instance of the model. So, should we load the model just once time, outside the process_batch?

[glenn-jocher]

@Leminhhuy2386 each process should instantiate its own model within process_batch to ensure thread safety and avoid shared state issues.

[Leminhhuy2386]

Thank you! However, in your sample code, the threads and processes are init, start, and join in the while loop. Should they be outside the loop?

[glenn-jocher]

@Leminhhuy2386 you're right; initializing, starting, and joining threads or processes should be outside the loop to avoid repeatedly creating them for each batch. Adjust the structure accordingly.