-
Notifications
You must be signed in to change notification settings - Fork 379
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix excessive memory and disk usage when sharing layers #2636
Conversation
storage/storage_dest.go
Outdated
} | ||
for _, blob := range dataBlobs.Values() { | ||
v, err := os.ReadFile(s.lockProtected.filenames[blob]) | ||
// Set up to save the cnofig as a data item. Since we only share layers, the config should be in a file. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
typo
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! Fixed.
7508283
to
880817f
Compare
I can't recall any places where we're using |
Signed-off-by: Miloslav Trmač <[email protected]>
Podman CI run: containers/podman#24629 |
In manual testing, I have confirmed both the bug hypothesis and that this fixes it. |
Can you remove Draft flag? |
Can this safely be merged? The skip F40, Go is too old commit suggests that this won't be able to be merged into podman or buildah, but it's not clear to me if it's "until F40 gets a new Golang update" or "never as long as F40 exists". |
I want to see the Podman CI ~passing first. |
That’s already the case before this PR, due to #2601 .
https://bodhi.fedoraproject.org/updates/FEDORA-2024-9b94592629 means this should be resolved shortly. In general, I‘ve been assuming that Go patch releases tend to include vulnerability fixes, and therefore tend to be included in Fedora quickly (… and given the static linking of Go binaries, our updated RPMs would be ~blocked on that Go update anyway). That’s not been the case with Go 1.22.{7,8}. Separately, there’s an effort to change sigstore so that it doesn’t add the dependency on very newest Golang so quickly, around sigstore/sigstore#1879 . |
880817f
to
c284a07
Compare
containers/podman#24629 looks reasonable, rebased and ready for review. |
LGTM |
a48541d
to
832134b
Compare
So this was a regression from #2067 ? Which was motivated by containers/storage#595 ? Wow, a lot going on there. Anyways, this code looks superficially sane to me... |
No, that’s not related at all; #2067 only moves the containers/podman#24527 (comment) points at the commit that introduced this bug. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Merge at will
Ahh, OK. Tangential: just glancing at that code one thing that seems like it'd help is if we had the canonical uncompressed size too in the spec, then some of this could just be mapping between the blob descriptor and and uncompressed descriptor instead of having separate maps for digest and size or so? |
|
createNewLayer can add new entries to s.filenames for uncompressed versions of layers; these entries don't match manifest.LayerInfos, causing the existing logic to assume they are a config and to store them as ImageBigData. That's both a potentially unlimited disk space waste (recording uncompressed versions of layers unnecessarily), and a memory pressure concern (ImageBigData is created by reading the whole blob into memory). Instead, precisely track the digest of the config, and never include more than one file in ImageBigData. Blobs added via PutBlob/TryReusingBlob but not referenced from the manifest are now ignored instead of being recorded inside the image. Signed-off-by: Miloslav Trmač <[email protected]>
832134b
to
6902e2c
Compare
Absolutely untested, filing for early review.createNewLayer
can add new entries tos.filenames
for uncompressed versions of layers; these entries don't matchmanifest.LayerInfos
, causing the existing logic to assume they are a config and to store them asImageBigData
.That's both a potentially unlimited disk space waste (recording uncompressed versions of layers unnecessarily), and a memory pressure concern (
ImageBigData
is created by reading the whole blob into memory).Instead, precisely track the digest of the config, and never include more than one file in
ImageBigData
.Blobs added via
PutBlob
/TryReusingBlob
but not referenced from the manifest are now ignored instead of being recorded inside the image.This is intended to fix containers/podman#24527 . @nalind is there anything that depends on the current behavior that I’m missing?
Cc: @giuseppe , affects performance measurements.