cilium: remove appArmorProfile for k8s<v1.30.0 #19888

ComradeProgrammer · 2024-10-30T22:21:05Z

FIX #19683
cilium: remove spec.template.spec.securityContext.appArmorProfile field when k8s version is smaller than 1.30.0

Before:

./out/minikube start --cni cilium --cpus max --kubernetes-version v1.28.9
😄  minikube v1.34.0 on Darwin 14.6.1 (arm64)
✨  Automatically selected the docker driver. Other choices: qemu2, ssh
📌  Using Docker Desktop driver with root privileges
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.45-1730110049-19872 ...
🔥  Creating docker container (CPUs=8, Memory=6100MB) ...
🐳  Preparing Kubernetes v1.28.9 on Docker 27.3.1 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring Cilium (Container Networking Interface) ...
💢  initialization failed, will try again: apply cni: cni apply: cmd: sudo /var/lib/minikube/binaries/v1.28.9/kubectl apply --kubeconfig=/var/lib/minikube/kubeconfig -f /var/tmp/minikube/cni.yaml output: -- stdout --
serviceaccount/cilium created
serviceaccount/cilium-envoy created
serviceaccount/cilium-operator created
configmap/cilium-config created
configmap/cilium-envoy-config created
clusterrole.rbac.authorization.k8s.io/cilium created
clusterrole.rbac.authorization.k8s.io/cilium-operator created
clusterrolebinding.rbac.authorization.k8s.io/cilium created
clusterrolebinding.rbac.authorization.k8s.io/cilium-operator created
role.rbac.authorization.k8s.io/cilium-config-agent created
rolebinding.rbac.authorization.k8s.io/cilium-config-agent created
service/cilium-envoy created
service/hubble-peer created
deployment.apps/cilium-operator created

-- /stdout --
** stderr ** 
Error from server (BadRequest): error when creating "/var/tmp/minikube/cni.yaml": DaemonSet in version "v1" cannot be handled as a DaemonSet: strict decoding error: unknown field "spec.template.spec.securityContext.appArmorProfile"
Error from server (BadRequest): error when creating "/var/tmp/minikube/cni.yaml": DaemonSet in version "v1" cannot be handled as a DaemonSet: strict decoding error: unknown field "spec.template.spec.securityContext.appArmorProfile"

** /stderr **: sudo /var/lib/minikube/binaries/v1.28.9/kubectl apply --kubeconfig=/var/lib/minikube/kubeconfig -f /var/tmp/minikube/cni.yaml: Process exited with status 1
stdout:
serviceaccount/cilium created
serviceaccount/cilium-envoy created
serviceaccount/cilium-operator created
configmap/cilium-config created
configmap/cilium-envoy-config created
clusterrole.rbac.authorization.k8s.io/cilium created
clusterrole.rbac.authorization.k8s.io/cilium-operator created
clusterrolebinding.rbac.authorization.k8s.io/cilium created
clusterrolebinding.rbac.authorization.k8s.io/cilium-operator created
role.rbac.authorization.k8s.io/cilium-config-agent created
rolebinding.rbac.authorization.k8s.io/cilium-config-agent created
service/cilium-envoy created
service/hubble-peer created
deployment.apps/cilium-operator created

stderr:
Error from server (BadRequest): error when creating "/var/tmp/minikube/cni.yaml": DaemonSet in version "v1" cannot be handled as a DaemonSet: strict decoding error: unknown field "spec.template.spec.securityContext.appArmorProfile"
Error from server (BadRequest): error when creating "/var/tmp/minikube/cni.yaml": DaemonSet in version "v1" cannot be handled as a DaemonSet: strict decoding error: unknown field "spec.template.spec.securityContext.appArmorProfile"

^C

After:


$ ./out/minikube start --cni cilium --cpus max --kubernetes-version v1.28.9
😄  minikube v1.34.0 on Darwin 14.6.1 (arm64)
✨  Automatically selected the docker driver. Other choices: qemu2, ssh
📌  Using Docker Desktop driver with root privileges
👍  Starting "minikube" primary control-plane node in "minikube" cluster
🚜  Pulling base image v0.0.45-1730110049-19872 ...
🔥  Creating docker container (CPUs=8, Memory=6100MB) ...
🐳  Preparing Kubernetes v1.28.9 on Docker 27.3.1 ...
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🔗  Configuring Cilium (Container Networking Interface) ...
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v5
🌟  Enabled addons: storage-provisioner, default-storageclass
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default

k8s-ci-robot · 2024-10-30T22:21:14Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ComradeProgrammer
Once this PR has been reviewed and has the lgtm label, please assign spowelljr for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

medyagh · 2024-10-31T17:01:40Z

/ok-to-test

medyagh · 2024-10-31T17:01:53Z

@ComradeProgrammer thanks for the PR :) plz check the lint

medyagh

lets make sure the PR mentions cillim, so in the changelog users know it only affects the cillim

cilium: remove appArmorProfile for k8s<v1.30.0

minikube-pr-bot · 2024-11-01T23:21:00Z

kvm2 driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 19888) |
+----------------+----------+---------------------+
| minikube start | 47.5s    | 48.1s               |
| enable ingress | 15.6s    | 16.6s               |
+----------------+----------+---------------------+

Times for minikube start: 45.3s 46.7s 49.5s 48.6s 47.7s
Times for minikube (PR 19888) start: 50.0s 50.7s 46.9s 46.9s 46.2s

Times for minikube ingress: 15.0s 14.9s 14.4s 19.0s 14.9s
Times for minikube (PR 19888) ingress: 14.9s 18.4s 18.9s 16.0s 14.4s

docker driver with docker runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 19888) |
+----------------+----------+---------------------+
| minikube start | 23.2s    | 23.0s               |
| enable ingress | 12.7s    | 12.5s               |
+----------------+----------+---------------------+

Times for minikube (PR 19888) start: 20.5s 21.2s 27.1s 23.3s 23.0s
Times for minikube start: 24.1s 20.7s 23.4s 23.3s 24.6s

Times for minikube ingress: 12.3s 12.3s 12.3s 13.8s 12.8s
Times for minikube (PR 19888) ingress: 12.8s 12.3s 11.2s 12.7s 13.2s

docker driver with containerd runtime

+----------------+----------+---------------------+
|    COMMAND     | MINIKUBE | MINIKUBE (PR 19888) |
+----------------+----------+---------------------+
| minikube start | 21.6s    | 20.2s               |
| enable ingress | 36.0s    | 36.3s               |
+----------------+----------+---------------------+

Times for minikube (PR 19888) start: 20.0s 19.0s 22.1s 20.1s 19.7s
Times for minikube start: 19.0s 23.1s 23.2s 20.1s 22.4s

Times for minikube ingress: 23.7s 38.7s 39.3s 39.2s 38.7s
Times for minikube (PR 19888) ingress: 39.7s 23.7s 39.3s 39.2s 39.3s

minikube-pr-bot · 2024-11-02T03:13:35Z

Here are the number of top 10 failed tests in each environments with lowest flake rate.

Environment	Test Name	Flake Rate
Docker_Linux_docker_arm64 (1 failed)	TestStartStop/group/old-k8s-version/serial/SecondStart(gopogh)	10.77% (chart)
Hyperkit_macOS (10 failed)	TestMultiNode/serial/RestartMultiNode(gopogh)	25.00% (chart)
Docker_Linux_crio (3 failed)	TestMultiControlPlane/serial/RestartCluster(gopogh)	4.69% (chart)

Besides the following environments also have failed tests:

Docker_Linux_crio_arm64: 2 failed (gopogh)
Docker_Cloud_Shell: 3 failed (gopogh)
KVM_Linux_crio: 32 failed (gopogh)

To see the flake rates of all tests by environment, click here.

spowelljr · 2024-11-15T16:16:29Z

pkg/minikube/cni/cilium.go

 	if err != nil {
 		return errors.Wrap(err, "generating cilium cfg")
 	}

 	return applyManifest(c.cc, r, manifestAsset(ciliumCfg))
 }
+
+func removeAppArorProfile(ciliumConfig string) (string, error) {


Suggested change

func removeAppArorProfile(ciliumConfig string) (string, error) {

func removeAppArmorProfile(ciliumConfig string) (string, error) {

spowelljr · 2024-11-15T16:16:35Z

pkg/minikube/cni/cilium.go

+	// see issue #19683, older Kubernetes versions cannot recognize appArmorProfile fields
+	k8sVersion, err := util.ParseKubernetesVersion(c.cc.KubernetesConfig.KubernetesVersion)
+	if err == nil && k8sVersion.LT(semver.MustParse("1.30.0")) {
+		if ciliumYaml, err = removeAppArorProfile(ciliumYaml); err != nil {


Suggested change

if ciliumYaml, err = removeAppArorProfile(ciliumYaml); err != nil {

if ciliumYaml, err = removeAppArmorProfile(ciliumYaml); err != nil {

spowelljr · 2024-11-15T16:18:41Z

pkg/minikube/cni/cilium.go

+	for {
+		obj := map[string]interface{}{}
+		err := decoder.Decode(&obj)
+		if err != nil && err.Error() == "EOF" {


Suggested change

if err != nil && err.Error() == "EOF" {

if err == io.EOF {

medyagh · 2024-11-22T18:03:37Z

@ComradeProgrammer plz take a look at the review

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Oct 30, 2024

k8s-ci-robot requested review from medyagh and spowelljr October 30, 2024 22:21

ComradeProgrammer requested review from medyagh and removed request for medyagh October 30, 2024 22:21

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Oct 30, 2024

ComradeProgrammer force-pushed the armor branch from 184ebc9 to 7f731c5 Compare October 30, 2024 22:24

k8s-ci-robot added the ok-to-test Indicates a non-member PR verified by an org member that is safe to test. label Oct 31, 2024

medyagh reviewed Oct 31, 2024

View reviewed changes

This comment has been minimized.

Sign in to view

ComradeProgrammer changed the title ~~remove appArmorProfile for k8s<v1.30.0~~ cilium: remove appArmorProfile for k8s<v1.30.0 Oct 31, 2024

ComradeProgrammer force-pushed the armor branch from 7f731c5 to fca5ebd Compare October 31, 2024 20:43

This comment has been minimized.

Sign in to view

cilium: remove appArmorProfile for k8s<v1.30.0

6e61e68

ComradeProgrammer force-pushed the armor branch from fca5ebd to 6e61e68 Compare November 1, 2024 22:25

spowelljr requested changes Nov 15, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cilium: remove appArmorProfile for k8s<v1.30.0 #19888

cilium: remove appArmorProfile for k8s<v1.30.0 #19888

ComradeProgrammer commented Oct 30, 2024 •

edited

Loading

k8s-ci-robot commented Oct 30, 2024

medyagh commented Oct 31, 2024

medyagh commented Oct 31, 2024

medyagh left a comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

minikube-pr-bot commented Nov 1, 2024

minikube-pr-bot commented Nov 2, 2024

spowelljr Nov 15, 2024

spowelljr Nov 15, 2024

spowelljr Nov 15, 2024

medyagh commented Nov 22, 2024

	func removeAppArorProfile(ciliumConfig string) (string, error) {
	func removeAppArmorProfile(ciliumConfig string) (string, error) {

	if ciliumYaml, err = removeAppArorProfile(ciliumYaml); err != nil {
	if ciliumYaml, err = removeAppArmorProfile(ciliumYaml); err != nil {

cilium: remove appArmorProfile for k8s<v1.30.0 #19888

Are you sure you want to change the base?

cilium: remove appArmorProfile for k8s<v1.30.0 #19888

Conversation

ComradeProgrammer commented Oct 30, 2024 • edited Loading

k8s-ci-robot commented Oct 30, 2024

medyagh commented Oct 31, 2024

medyagh commented Oct 31, 2024

medyagh left a comment

Choose a reason for hiding this comment

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

minikube-pr-bot commented Nov 1, 2024

minikube-pr-bot commented Nov 2, 2024

spowelljr Nov 15, 2024

Choose a reason for hiding this comment

spowelljr Nov 15, 2024

Choose a reason for hiding this comment

spowelljr Nov 15, 2024

Choose a reason for hiding this comment

medyagh commented Nov 22, 2024

ComradeProgrammer commented Oct 30, 2024 •

edited

Loading