Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add cross-distribution cache hit support #47

Open
eddebc opened this issue Feb 12, 2022 · 1 comment
Open

Add cross-distribution cache hit support #47

eddebc opened this issue Feb 12, 2022 · 1 comment

Comments

@eddebc
Copy link

eddebc commented Feb 12, 2022

Packages that exist for both Manjaro and Archlinux, for example, with the same sig will be downloaded twice.

pi@raspberrypi:~ $ ls -l ext_disk1/archmirror-cache/pkgs/*/acl-2.3.1-2-x86_64.pkg.tar.zst
-rw-r--r-- 1 pi pi 138970 Feb  1 15:50 ext_disk1/archmirror-cache/pkgs/archlinux/acl-2.3.1-2-x86_64.pkg.tar.zst
-rw-r--r-- 1 pi pi 138970 Feb  2 15:01 ext_disk1/archmirror-cache/pkgs/manjaro/acl-2.3.1-2-x86_64.pkg.tar.zst

pi@raspberrypi:~ $ sha256sum ext_disk1/archmirror-cache/pkgs/*/acl-2.3.1-2-x86_64.pkg.tar.zst.sig 
00b0bcd4ac5e7e8f3308bfca5892ba46086b13b81b5aa8d523bb61905c85b16c  ext_disk1/archmirror-cache/pkgs/archlinux/acl-2.3.1-2-x86_64.pkg.tar.zst.sig
00b0bcd4ac5e7e8f3308bfca5892ba46086b13b81b5aa8d523bb61905c85b16c  ext_disk1/archmirror-cache/pkgs/manjaro/acl-2.3.1-2-x86_64.pkg.tar.zst.sig

Ideally, this should cause a cache hit rather than re-download.

@Focshole
Copy link
Contributor

Focshole commented Mar 25, 2022

Well, it would be great, but it would be a lot of work. Let me explain why:

  • Different repos may have slightly different modifications on some files, maybe without changing the version number (e.g. it may include a patch for their distro) -> they require integrity checking
  • The upgrades may diverge between repos, so keeping track of which file should be purged can be messy
  • Maybe the same package may be signed from different authors in two different repos, so if that happens, they have to be treated as separate files -> can be solved with integrity checking

To guarantee that approach, what it should be done is:

  • keeping track of shared packages between repos, by also doing integrity checking between them (if they match, they are the same package, otherwise they have to be treated as separate packages).
  • So far pacoloco does not integrity check its packages, it leaves that job to pacman (which also checks the signature), so also this should be implemented and maintained (in case something in the packages/db format/signing method changes), so this may cause many future issues.
  • Refactor the code to clearly distinguish packages from their files (files has to be treated as another table in the db, because they may overlap between repos), and keep track of each file each time the prefetching routine is being run too, by updating the code accordingly

It can clearly be useful, it is up to who wants to implement and maintain it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants