Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Is there any possible make group and batch work together? #58

Open
0xuhe opened this issue Aug 30, 2023 · 3 comments
Open

Is there any possible make group and batch work together? #58

0xuhe opened this issue Aug 30, 2023 · 3 comments

Comments

@0xuhe
Copy link

0xuhe commented Aug 30, 2023

// worker.ts
const worker = new WorkerPro(
  "aaa",
  async (job: JobPro) => {
    console.log("start");
    // await sleep(12e3);
    const jobs = job.getBatch();
    for (const job of jobs) {
      console.log(job.data);
    }
    console.log("end");
  },
  {
    connection: redis,
    concurrency: 20,
    group: {
      concurrency: 1, // Limit to max 3 parallel jobs per group
    },
    batch: { size: 10 },
  }
);
const queue = createQueue("aaa");

await queue.add(
    "bbb",
    { a: 1 },
    {
      group: {
        id: "aaa",
      },
    }
  );
await queue.add(
    "bbb",
    { a: 2 },
    {
      group: {
        id: "aaa",
      },
    }
  );

desire:

start
{a: 1}
{a: 2}
end

actually

start 
{a:1}
end
start
{a:2}
end
@manast
Copy link
Contributor

manast commented Aug 30, 2023

Currently, this is not supported. Could you please tell us a bit more about your use case and why this is important for you?

@0xuhe
Copy link
Author

0xuhe commented Aug 30, 2023

@manast

In my case, I need jobs to change host's IPs, a host only change one IP or many IPs in one process, but could not process meanwhile on the same host, each process cost 30s.
First, there are many workers to process corresponding hosts at same time.
Second, When a worker receive a job, the worker will process its job in 30s, the next jobs will wating for this job completed. There maybe many jobs waiting for first job to be completed. After first job completed, the worker could process wating jobs in batch.
eg:
I have many hosts and IPs need change.
hostA: Ip1
hostA: Ip2
hostA: Ip3
hostB: Ip1
hostB: Ip2
hostB: Ip3

My desire:
queueA: hostA
queueB: hostB
works in the meantime. It will process Ip1 immediately and it will cost 30s, and this queue receive job Ip2 at 5s and job Ip3 at 10s. I wish the worker could process Ip2 and Ip3 in batch after job Ip1 complete at 30s.

@manast
Copy link
Contributor

manast commented Aug 30, 2023

Thanks. We will make an analysis to understand how feasible it is to add batch support to groups.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants