Reduce EVE rootfs size by removing unused files and stripping binaries #4322
Comments
rucoder
added
bug
|
use shared libraries for Go binaries |
|
Are we comfortable with shared libraries vs static binaries? I guess within a single container it's fine, although we would like to break pillar apart. Then again, does it matter, if most of it is a single pillar binary? |
There is f.e. also the vtpm binary |
containerd is 36MB, I guess it would also profit a bit from shared libraries I think if we have the same (i.e. same hash sum) shared library in different containers, then squashfs can compress it very well. |
|
It would be an interesting experiment to see what actually happens if we use shared libraries. We could be spending all of this time to discover that it makes little difference. I know that if you launch two copies of the same executable, the read only memory consumption is once. Same for shared objects. If two distinct executables, each way 5MB, use a shared object of 3MB, then total memory is 5*2+3, and not (5+3)*2, ie one copy of the shared object. What about statically compiled? Is it intelligent enough to recognize the common bits? I would think no, but @rucoder would know much better. |
|
@deitch this is exactly why shared libraries were born. Static images cannot share their code. |
I know. I didn't know if in the years since, loaders had gotten smarter about recognizing identical binary chunks in different files. Well, this thread isn't really about memory usage, is it? I was just checking that as as side question.
You mean to share libraries across containers? I don't see it. Nothing in containerd infrastructure supports it now, I haven't seen anyone else doing it. I have seen people mounting some shared libs from a host filesystem into multiple containers, but it isn't all that common. Mainly because size of root filesystem doesn't matter all that much to most? |
|
@deitch this is exactly what we need to solve: the image size. I think it is not that hard to bind mount /hosfs/lib into /lib inside the container. We can even do it manually in build.yml but LK must move all libs from individual containers to host fs and de-duplicate them |
If two containers share the same base layer, then in the overlayfs they have the same lower mount, don't they? Then they share the files. |
|
yes, if SHA is the same but all our containers are |
Right, not memory usage. Agreed.
Doable? Yes. I am really concerned about the management and synchronization overhead. We would need to have something that makes it easier to use. Like we have our As a general rule, containers that cannot run on their own, that depend on some volume mount or bind mount just to get executables to run, goes against the grain. In any case, are we really sure that shared libraries across containers is what will make the difference? As it is, pillar - the biggest one - is mostly a single binary anyways. We should take a good hard look at a built eve and see where the sizes are big. |
|
FWIW, I just built eve from latest master commit: $ du -s * | sort -n
4 config
4 home
4 mnt
4 persist
4 proc
4 root
4 run
4 srv
4 sys
4 tmp
8 opt
12 dev
16 media
84 var
1292 EFI
1372 init
3844 etc
4180 sbin
4236 bin
22580 boot
127140 usr
234784 lib
585720 containersSo the only things that really matter:
Starting with the smallest: $ du -s -h boot/*
4.0K boot/cmdline
14M boot/kernel
7.8M boot/ucode.img
1.2M boot/xen.gzKernel and code, not much else there. $ du -s lib/* | sort -n
0 lib/libc.musl-x86_64.so.1
0 lib/libcom_err.so.2
0 lib/libfdisk.so.1
0 lib/libmount.so.1
0 lib/libpam.so.0
0 lib/libpam_misc.so.0
0 lib/libpamc.so.0
0 lib/libsmartcols.so.1
0 lib/libuuid.so.1
0 lib/libz.so
4 lib/mdev
4 lib/modules-load.d
8 lib/sysctl.d
16 lib/libcom_err.so.2.1
16 lib/libpam_misc.so.0.82.1
16 lib/pkgconfig
20 lib/libpamc.so.0.82.1
32 lib/libuuid.so.1.3.0
52 lib/resolvconf
60 lib/libpam.so.0.85.1
88 lib/libkmod.so.2
100 lib/libz.so.1
100 lib/libz.so.1.2.12
100 lib/libz.so.1.2.13
128 lib/libudev.so.1
180 lib/libapk.so.3.12.0
200 lib/libsmartcols.so.1.1.0
308 lib/libblkid.so.1
308 lib/libblkid.so.1.1.0
332 lib/libmount.so.1.1.0
400 lib/libfdisk.so.1.1.0
512 lib/libssl.so.1.1
556 lib/udev
592 lib/ld-musl-x86_64.so.1
592 lib/libssl.so.3
740 lib/security
996 lib/apk
2552 lib/libcrypto.so.1.1
3792 lib/libcrypto.so.3
65556 lib/modules
156420 lib/firmware
$ du -s lib/firmware/* | sort -n
...
4236 lib/firmware/intel
6232 lib/firmware/cypress
8876 lib/firmware/display-t234-dce.bin
8996 lib/firmware/mrvl
9452 lib/firmware/ti-connectivity
13732 lib/firmware/brcm
19960 lib/firmware/ath10kDo we need all of those on every build? The only other really big one in lib was: $ du -s lib/modules/6.1.106-linuxkit-06a737b0f212/* | sort -n
0 lib/modules/6.1.106-linuxkit-06a737b0f212/build
12 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.order
28 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.builtin
28 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.dep
36 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.builtin.bin
40 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.dep.bin
164 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.symbols
192 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.builtin.modinfo
196 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.symbols.bin
316 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.alias.bin
320 lib/modules/6.1.106-linuxkit-06a737b0f212/modules.alias
8288 lib/modules/6.1.106-linuxkit-06a737b0f212/extra
55928 lib/modules/6.1.106-linuxkit-06a737b0f212/kernelMoving on to usr: $ du -s -h usr/*
104M usr/bin
3.8M usr/include
14M usr/lib
388K usr/libexec
20K usr/local
2.0M usr/sbin
720K usr/shareOK, mostly $ du -s usr/bin/* | sort -n
...
11648 usr/bin/containerd-shim-runc-v2
13168 usr/bin/runc
14252 usr/bin/service
20324 usr/bin/ctr
37128 usr/bin/containerdOnly really big things there are containerd, runc and service. Last $ du -s containers/*/* | sort -n
28 containers/services/pillar
1444 containers/onboot/000-rngd
1884 containers/onboot/001-sysctl
6764 containers/services/watchdog
8272 containers/onboot/006-measure-config
8612 containers/onboot/003-kdump
11460 containers/services/wlan
13140 containers/services/memory-monitor
18592 containers/services/newlogd
18648 containers/onboot/005-apparmor
21460 containers/services/guacd
24936 containers/services/edgeview
28708 containers/onboot/002-storage-init
45268 containers/services/wwan
53160 containers/services/vtpm
88840 containers/services/debug
97104 containers/services/xen-tools
137388 containers/onboot/004-pillar-onbootMost of them are pretty small. The last bunch start getting really big. $ du -s containers/onboot/004-pillar-onboot/lower/* | sort -n
0 containers/onboot/004-pillar-onboot/lower/containers
4 containers/onboot/004-pillar-onboot/lower/dhcpcd.conf
4 containers/onboot/004-pillar-onboot/lower/fscrypt.conf
4 containers/onboot/004-pillar-onboot/lower/home
4 containers/onboot/004-pillar-onboot/lower/init.sh
4 containers/onboot/004-pillar-onboot/lower/mnt
4 containers/onboot/004-pillar-onboot/lower/proc
4 containers/onboot/004-pillar-onboot/lower/root
4 containers/onboot/004-pillar-onboot/lower/run
4 containers/onboot/004-pillar-onboot/lower/srv
4 containers/onboot/004-pillar-onboot/lower/sys
4 containers/onboot/004-pillar-onboot/lower/tmp
12 containers/onboot/004-pillar-onboot/lower/dev
16 containers/onboot/004-pillar-onboot/lower/media
104 containers/onboot/004-pillar-onboot/lower/var
1604 containers/onboot/004-pillar-onboot/lower/etc
2320 containers/onboot/004-pillar-onboot/lower/bin
5068 containers/onboot/004-pillar-onboot/lower/sbin
7076 containers/onboot/004-pillar-onboot/lower/lib
52460 containers/onboot/004-pillar-onboot/lower/usr
68652 containers/onboot/004-pillar-onboot/lower/optSo $ du -s containers/onboot/004-pillar-onboot/lower/usr/bin/* | sort -n
...
1032 containers/onboot/004-pillar-onboot/lower/usr/bin/sgdisk
1100 containers/onboot/004-pillar-onboot/lower/usr/bin/coreutils
2044 containers/onboot/004-pillar-onboot/lower/usr/bin/qemu-io
2100 containers/onboot/004-pillar-onboot/lower/usr/bin/qemu-img
2220 containers/onboot/004-pillar-onboot/lower/usr/bin/qemu-nbd
3064 containers/onboot/004-pillar-onboot/lower/usr/bin/qemu-storage-daemon
$ du -s containers/onboot/004-pillar-onboot/lower/usr/lib/* | sort -n
...
852 containers/onboot/004-pillar-onboot/lower/usr/lib/libisoburn.so.1.111.0
1064 containers/onboot/004-pillar-onboot/lower/usr/lib/libglib-2.0.so.0.7200.4
1084 containers/onboot/004-pillar-onboot/lower/usr/lib/libp11-kit.so.0.3.0
1660 containers/onboot/004-pillar-onboot/lower/usr/lib/libunistring.so.2.2.0
1732 containers/onboot/004-pillar-onboot/lower/usr/lib/libgio-2.0.so.0.7200.4
1892 containers/onboot/004-pillar-onboot/lower/usr/lib/libgnutls.so.30.34.1
2008 containers/onboot/004-pillar-onboot/lower/usr/lib/xtables
4164 containers/onboot/004-pillar-onboot/lower/usr/lib/libzpool.so.5.0.0and: $ du -s containers/onboot/004-pillar-onboot/lower/opt/zededa/bin/* | sort -n
...
8 containers/onboot/004-pillar-onboot/lower/opt/zededa/bin/onboot.sh
20 containers/onboot/004-pillar-onboot/lower/opt/zededa/bin/device-steps.sh
392 containers/onboot/004-pillar-onboot/lower/opt/zededa/bin/dnsmasq
5288 containers/onboot/004-pillar-onboot/lower/opt/zededa/bin/fscrypt
62916 containers/onboot/004-pillar-onboot/lower/opt/zededa/bin/zedboxzedbox is big. You can keep going for the other big ones, xen-tools and debug. |
|
FWIW, CDI can also be used to provide a common software stack base to all containers, we could have a directory with the whole common Alpine base files and mount it inside all containers through a CDI spec.... but considering our ecosystem, I still don't see any "easy/medium implementation efforts" for this kind of approach.... |
|
Yeah, it gets pretty complicated. We don't want solutions that make it even harder to build and run EVE and its components. In pillar, zedbox is big, but so are usr and lib. Do we need everything there? |
|
Why is onboot container so large? For instance, do we need qemu* in onboot? And why does onboot need zedbox?? |
|
It's the same container image filesystem as services. It's just reused. |
|
Breakdown of zedbox done with https://github.com/Zxilly/go-size-analyzer |
|
Have we tried building pillar while removing the $ grep -r -w '"reflect"' * | grep -v vendor
cipher/cipher.go: "reflect"
cmd/usbmanager/subscriptions.go: "reflect"
cmd/tpmmgr/tpmmgr.go: "reflect"
cmd/zedagent/attesttask.go: "reflect"
cmd/domainmgr/domainmgr_test.go: "reflect"
cmd/zedmanager/handlezedrouter.go: "reflect"
containerd/oci_test.go: "reflect"
dpcmanager/dpc_test.go: "reflect"
dpcreconciler/genericitems/resolvconf.go: "reflect"
dpcreconciler/linuxitems/wlan.go: "reflect"
dpcreconciler/linuxitems/bond.go: "reflect"
evetpm/tpm_test.go: "reflect"
hypervisor/hypervisor_test.go: "reflect"
netmonitor/mock.go: "reflect"
objtonum/map.go: "reflect"
pubsub/large_test.go: "reflect"
pubsub/subscribe.go: "reflect"
pubsub/util.go: "reflect"
pubsub/pubsub.go: "reflect"
pubsub/publish.go: "reflect"
types/dns.go: "reflect"
types/assignableadapters.go: "reflect"
types/errortime_test.go: "reflect"
types/dpc.go: "reflect"
types/errortime.go: "reflect"
utils/deserialize_test.go: "reflect"
utils/deserialize.go: "reflect" |
|
Actually, that would be an interesting strategy. We should check for |
|
@mikem-zed @rene @milan-zededa any thoughts on this comment, especially the part about halfway through it:
|
Doesn't json marshalling depend on Good, that I just added more use of it ... (#4295 ) |
|
Maybe I'll try next week to build pillar without any reflect and see what happens. Not sure about json marshalling. I don't think it depends on it, but not sure. |
For |
|
I just checked the source code for golang/go, same thing. I wish I could remember where it was I had the conversation about reflect and removing unused, and if it is all uses of reflect, or only certain ones. |
|
Here is the part that strips things out unless reflect is used. |
|
I spent quite some time digging more deeply. There is an excellent presentation - and pretty good tool - given by the author of whydeadcode (links to slides and talk at the repo). Short form, it is not all usage of
Here is one example from our own code, package zedpac
import (
"fmt"
"github.com/jackwakefield/gopac"
)
func Find_proxy_sync(pac, url, host string) (string, error) {
parser := new(gopac.Parser)
if err := parser.ParseBytes([]byte(pac)); err != nil {
return "", fmt.Errorf("invalid proxy auto-configuration file: %v", err)
}
return parser.FindProxy(url, host)
}And the dependency chain: That should be replaceable. I tried building without that dependency - not replacing the functionality, just to test. The problem is that it is not the only place we use those methods, and here it gets harder: That is a long chain, but not only containerd, everything that uses There is a huge amount of installed code which depends on these reflect methods, directly or indirectly. I suspect if we could, the size of the binary would drop a lot, although I couldn't begin to guess how much. But it is far bigger than us. It even goes into core classes. And I checked latest mainline branch for |
|
If I comment out the content of New output starts with: |
Correct, but that isn't because of any general deadcode detection. It is because of anything directly referenced by what we filtered out, and its dependencies. There might be some deadcode in there as well, that only was included because of the reflect issues, but those are minor. To make a real difference, we would need to get rid of the deadcode inclusion-causing calls ( |
EVE rootfs is close t 250Mb in size. The obvious option is to increase a partition size but first let's try to get rid of unused files/tools/packages and strip debug information from binaries. We cannot increase rootfs size > 300 MB
The text was updated successfully, but these errors were encountered: