Is low memory operation in scope for gitoxide?

[tanriol]

Recently I wanted to mirror several big git reporisories by cloning them (as bare) and then periodically fetching updates on a low-end NAS. Turned out neither git nor gitoxide can currently be used to clone something like https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ while having just 512MB RAM.

I've failed to find whether being usable in low-memory environments is a goal for gitoxide. Would it make sense to try adding support for such use cases or would trying to support these without degrading performance in the normal case be unwanted complexity for gitoxide that'd be really unlikely to be accepted?

[Byron]

Thanks for sharing, that's a very interesting comparison and I am somewhat glad that gitoxide isn't worse than Git here.

However, it's also not a project goal to support smaller systems, particularly not 32bit systems which will have more trouble than with Git. Most 'small' systems nowadays are 64 bit, so the greatest obstacle towards completing a task will be swapping.

When you say that cloning the linux kernel with 512MB RAM didn't work, what does that mean exactly? OOM kill? Or excruciating swapping? If the case is the latter, I would have expected it to finish eventually.

Normally, I'd say that there is no way that this is made to work with the current algorithm - it's already mindful about memory and should be a bit better than Git, but it still has to hold millions of objects in memory all at once, which is where the high demands for the linux kernel are coming from.

However, there is an issue whose resolution basically requires rewriting the way packs are resolved. This algorithm will be more compatible, but also will be less parallel. While at it, I don't think it would hurt to make it possible to work in low-memory situations. To make that work, the only thing I can imagine that it crates some on-disk memory-mappable data-structure that first has to grow while streaming the pack, to later be traversed. Interestingly I'd think that if the disk is moderately fast, that streaming of data shouldn't be a bottleneck neither for reading nor for writing, if done cleverly, (or if a sqlite database is used ;)).

That's a long-winded way of saying: A fix would be welcome if it would also fix #1025 at the same time.

[tanriol]

When you say that cloning the linux kernel with 512MB RAM didn't work, what does that mean exactly? OOM kill? Or excruciating swapping? If the case is the latter, I would have expected it to finish eventually.

So far I've seen two problems with cloning big repos. The first appeared while cloning https://github.com/gentoo/gentoo.git (5M objects). When (towards the end of cloning) RAM becomes tight, realtime updates of the in-memory index start swapping and thus slowing down the clone operation to the point where (I think) github just cuts it (manifested with gix producing an IO error like "failed to fill whole buffer" IIRC). I've seemingly succeeded at doing that once with 32-bit build, possibly a matter of luck.
git in this case had some other network-looking problem.

The second happens with cloning https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/ and is different
git: after 24 hours or so of swapping was at ~20% of "resolving deltas", interrupted manually
gix: panicked after ~20 min (around 25% objects received) with "Unable to allocate another span" (looks like coming from something tracing-related, not reproducing on a desktop)

[...]
That's a long-winded way of saying: A fix would be welcome if it would also fix #1025 at the same time.

Okay, I'll look into that, thank you.

[Byron]

Thank you for elaborating, it's very interesting to hear how low-memory situations are 'expressing' themselves. From what I hear, even if the remote wouldn't cut the connection after a day, the initial clone is still too slow to happen.

Oh, before I forget. Something simpler than writing a new pack resolution algorithm could be to download bundles. That way, the initial download is just an existing pack file along with its index… never mind, bundles are also just pack files, the index will have to be produced locally which is what's causing the problem in the first place.

So indeed, there seems to be no way around creating a low-memory way of resolving a pack if this should have a chance to work, a prospect I find very interesting to say the least. If you were to succeed, then gitoxide already served its purpose of also being a platform to experiment with novel approaches, something that should be easier in Rust to do for more people.

[NobodyXu]

Maybe using a different allocator could help?

I.e. mimalloc or jemalloc which has less fragmentation.

If it's swapping a lot...then maybe

• use zswap/zram which compresses the memory instead of swapping
• increase the swapping partition
• use a faster SSD

😅 just my two cents on how it could be speedup, it could be totally wrong for your environment