Support max connections per server

[Cadair]

Given we can now open many connections per server (n files x n splits) it would be useful to be able to specify limits for servers which are not able (or even block) too many connections. This would mean that we could prioritise downloads from other servers to maintain the parallelism while putting less load on servers.

[Cadair]

This appears to be possible to do with aiohttp by customising the connector class used: https://docs.aiohttp.org/en/stable/client_reference.html#baseconnector

We would probably want to be a little smarter than that though so that we don't fill up the download queue with pending downloads from a host where the max number of connections is lower than the number of parallel downloads * number of splits.

[exitflynn]

Hi @Cadair!

We would probably want to be a little smarter than that though so that we don't fill up the download queue with pending downloads from a host where the max number of connections is lower than the number of parallel downloads * number of splits.

I'm a bit confused and having trouble understanding this, implementing a way to set the limit_per_host = (no. of parallel downloads) * (no. of splits) should work right? If a user requests more files than that, then they're bound to go in pending, we can't really do much about it? (assuming we only have a single source to download from)

[exitflynn]

oooh i think i get it now, if many files are enqueued and a handful of them are from a server which has less max_conn limits, we could end up in a scenario where the queue, which has other files to download from different servers, is now filled up by the pending downloads due to that particular server.

But, shouldn't this not be a problem since we're doing things asynchronously here? A lot of pending downloads due to one server shouldn't cause a hold-up for others, right?

Gah! I think i'm still having trouble understanding the issue here 😅

[Cadair]

So this issue is about having a limit on max_splits * max_conn per server. Imagine you have 100 files from 10 servers. Nine servers can handle the default number of connections, but one server can only handle one connection at a time. This issue is about limiting just that server to one connection while all the others can have many.