[Feature] Download Limits

[VasVadum]

Problem

Downloading 6 items simultaneously is not good for all networks. I am on 1.5Mbit DSL, which allows for a max speed somewhere around 180KB/s. CKAN tried to download 6 large mods at once causing failure 8 times.

Suggestions

Describe the solution you'd like
Simply add settings to limit the number of simultaneous downloads so that users can set it back to CKAN's old system of one at a time or higher if they so choose. I checked the settings when I first saw this new feature and didn't see any.

[HebaruSan]

set it back to CKAN's old system of one at a time
when I first saw this new feature

For what it's worth, and this doesn't matter much for the enhancement requested here, but there was no such "old system"; CKAN has always downloaded multiple files in parallel (it just didn't show progress bars for all of them in the old days). Here's @pjf editing that code back in 2014:

• f0bb21b

Recently we moved away from that practice somewhat, because GitHub really doesn't like multiple parallel downloads regardless of how fast your network is (see #3054). That functionality is host-based at the moment; do you happen to remember which hosts those 6 downloads were coming from? Probably SpaceDock?

[VasVadum]

I don't remember what hosts they were from. I just know that I would like to limit the downloads to one at a time so that things are handled more smoothly. I know others probably want more, which is why I'm just requesting the feature to adjust how many downloads can go at the same time.

Its the same with the browser. If I open 1 tab every 3-5 seconds with an image in it, I can usually load them all. However if I load 20 tabs with images in them all at once, several minutes later they all suddenly stop loading and either have a half loaded image or just fail to load the image completely and I have to go back and retry everything that failed.

The network seems to cancel things that take too long, or don't respond in time. So if you have a ton of network streams at once on a slow connection, some of them may not respond in time to continue with their next packet or however the network handles this. Which is why I get "unexpected end of file" a lot with CKAN.

[HebaruSan]

I have been installing big groups of mods to test #3525, and I've found that even SpaceDock does not seem to like having a lot of parallel downloads; any beyond 2 or 3 seemed to fail and fallback to archive.org or require retrying the installation. And since Mono isn't going to fix #3555, one of the workaround ideas from #3343 was to reduce the number of parallel downloads.

@DasSkelett, what do you think of applying the current GitHub download limiting behavior to all hosts, effectively limiting CKAN to one download per host at once? Am I right to think that would make sense in terms of network architecture, that SpaceDock will send the same number of bytes per second for 1 download as it would for 10, basically just filling the pipe between it and us?

@VasVadum, it would be good to know which host or hosts you've had problems with, so we can determine whether the above change would help you. If it's exclusively been too many downloads from SpaceDock, then that change would fix your problem and no new setting would be necessary.

[DasSkelett]

Yes, let's do that. It can happen that a single TCP stream (= download) doesn't manage to fill a big pipe completely, but not really for today's average residential download speeds (they aren't high enough, and those that are hit SpaceDock's limits). It might also be faster for SpaceDock to serve one zip at a time (per user) instead of fetching them all simultaneously from the storage, with lots of context switches. It would loose a bit if e.g. a "later" download is already cached in SD's proxy, where that one could already be served while the earlier one is being fetched from the storage in the meantime.

We could also do something like two downloads at a time, which should still be low enough not to trigger the bug from #3343 or stall one TCP connection completely, while also benefiting of some multitasking.

[VasVadum]

I would very much like the ability to simply max my concurrent downloads though. Not that its an issue for me anymore, I just think it'd be a good idea for the users to be able to click a setting somewhere that says "I want this many downloads going at one time, and no more", regardless of host.

Its no longer an issue for me, because I moved to a location that has high speed at like, 100MB/s. So I mean, these downloads all finish faster than I can click start. But I still believe in choice, and there may be other people out there who will want a "one at a time" approach.