Complete/Fail a job from external service?

[kevinost]

Hey everyone,

Just came across the pro-version. We've been working on implementing groups ourselves now during the past week, but then I found that you've already done it in the pro version! Well done on that feature. It's a game changer for us

I've got a use case where I want to complete/fail/retry a job manually (the job itself should not become completed once it finishes processing). Basically, the situation is

1. Job is added to a grouped queue
2. The processing function prepares the processing (when these preparations complete we want the job to still be in progress)
3. Job is added to a second queue (without grouping) that works as a throttler for the actual processing that happen in a kubernetes cluster
4. After processing, the job result is sent back to the first service to be handed
5. At this point we'd like to ack the grouped queue, to allow the next job to go into the same chain of processing

I understand it's not a common pattern, but do you have any suggestions on how we can achieve this with bullmq?

[manast]

@kevinost thank you for your interest in the pro version.
I wonder if you are familiar with the "flows" feature in the standard version? https://docs.bullmq.io/guide/flows
Flows will allow you to have a parent job that is only processed when its children are processed and pass its results to the parent.
From the explanation above I could not fully understand the need for the "groups" feature. It seems as if you could have 3 queues, one standard queue where you do the preparations and then finalizes by creating a flow with one parent and one child, the child will be the job processed in the kubernetes cluster and when the child completes the parent gets the result and does whatever it needs to do.

[kevinost]

Thanks! The flows would solve that challange. After some additional research however we realized we will need the grouped queues to have a consistent distribution rather than round-robin. The use case is a one-queue-per-user system where order of jobs is guaranteed. From what I've understood it's usually solved with a hash function that distributes messages across partitions in a consistent manner. (For example a "consistent hash exchange" in rabbitmq or Kafka's built in support for consistent partitioning).
Is there any neat solution for this type of consistent ordering in bull today except for actually creating a real queue for each user?

[manast]

For distributing the job in a "fair" way among different users you will need to use the "groups" feature in the Pro version as there is no such functionality in the standard version. I recommend you start a trial and check if it fulfills your needs.