Upgrade Multus to v4.x

[chomatdam]

Context

v4 is doing a split between thin (as v3) and thick (adding metrics)

Migrate to thick and resolve any issues encountered

Notes for the migration:

Notes gathered when trying to upgrade to 4.0.2, some items may have been resolved since

• They moved from a python container image on 3.x to a distroless image on 4.x (cat not available anymore)
• Their thick container image uses a slim Debian where we would still have cat for the startupProbe
• Add -thick to the overridden container tag to not have their too loose versioning
• --multus-conf-file not available on their thick version, being a flag only for their thin entrypoint

[chomatdam]

@scoquelin:
Was able to confirm some of those points locally after installing the thick version :

# /usr/src/multus-cni/bin/multus --version        
multus: version:(clean), commit:e1a0d2a3fd4f490861ff1872840fdf0a69200747, date:2024-10-15T16:06:23+00:00

# /usr/src/multus-cni/bin/multus-daemon --help
Usage of /usr/src/multus-cni/bin/multus-daemon:
  -config string
        Specify the path to the multus-daemon configuration (default "/etc/cni/net.d/multus.d/daemon-config.json")
  -version
        Show version
# cat /etc/cni/net.d/multus.d/daemon-config.json
{
    "chrootDir": "/hostroot",
    "cniVersion": "0.3.1",
    "logLevel": "verbose",
    "logToStderr": true,
    "cniConfigDir": "/host/etc/cni/net.d",
    "multusAutoconfigDir": "/host/etc/cni/net.d",
    "multusConfigFile": "auto",
    "socketDir": "/host/run/multus/"
}

# whereis cat
cat: /usr/bin/cat

# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

The yaml definitions for v4.1.3 point to an image that is tagged with snapshot-thick yet the v4.1.3-thick tag is available from here so I guess we have to override it explicitly with kustomize

[chomatdam]

@scoquelin:
Here is some documentation attempt on how to configure multusMasterCNI with the thick version : k8snetworkplumbingwg/multus-cni#1245

[chomatdam]

Open issues related to migration

• [OOMKilled] High memory consumption k8snetworkplumbingwg/multus-cni#1346
• Race issue after node reboot k8snetworkplumbingwg/multus-cni#1221