open5gs installation with AWS EKS - Control plane pods are not coming , they are in init state - help needed

[hannanbeediwale]

Hi

Facing issue in the last step at Helm Deployment:

Install Helm chart with the following command:
helm -n open5gs install -f values.yaml epc-core ./

[[ec2-user@ip-10-0-0-14 x86-Architecture]$ kubectl get -n open5gs pods
NAME READY STATUS RESTARTS AGE
open5gs-hss-deployment-5d856c4f7b-jgmfx 0/1 Init:0/1 0 19m
open5gs-mme-deployment-848cd7766-5t82q 0/1 Init:0/1 0 19m
open5gs-nrf-deployment-55cc9fc696-zczwz 1/1 Running 0 19m
open5gs-pcrf-deployment-f8fb984fb-7w7d8 0/1 Init:0/1 0 19m
open5gs-sgwc-deployment-5bb6d97475-9p6st 0/1 Init:0/1 0 19m
open5gs-sgwu-deployment-bfcc6c8cf-c98rr 1/1 Running 0 19m
open5gs-smf-deployment-7f679b945-x79l4 0/1 Init:0/1 0 19m
open5gs-upf-deployment-64cf6b6499-wpzrz 1/1 Running 0 19m
open5gs-webui-5dd5944c56-jc59c 1/1 Running 0 19m
[ec2-user@ip-10-0-0-14 x86-Architecture]$
[ec2-user@ip-10-0-0-14 x86-Architecture]$

Completed all the install steps Following this link to install:https://aws.amazon.com/blogs/opensource/open-source-mobile-core-network-implementation-on-amazon-elastic-kubernetes-service/

1. Run the CloudFormation for infra creation (open5gs-infra.yaml).
2. Bastion host configuration and K8s ConfigMap update.
3. DocumentDB initialization.
4. CoreDNS ConfigMap update to use Route 53 for 3GPP service interfaces.
5. Run the CloudFormation for Multus worker node group creation (open5gs-worker.yaml).
6. DNS controller and Multus-IP update controller deployment for the automation.
7. Run shell script for cluster initialization (setting up namespace, etc.).
8. Helm installation for all network functions.

[ec2-user@ip-10-0-0-218 x86-Architecture]$ kubectl -n open5gs get secret
NAME TYPE DATA AGE
diameter-ca Opaque 1 8h
hss-tls kubernetes.io/tls 2 8h
mme-tls kubernetes.io/tls 2 8h
mongodb-ca Opaque 1 8h
pcrf-tls kubernetes.io/tls 2 8h
smf-tls kubernetes.io/tls 2 8h

[ec2-user@ip-10-0-0-218 x86-Architecture]$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
aws-node-r5bsr 1/1 Running 0 9m6s
coredns-594b88dc6c-njnw9 1/1 Running 0 19m
coredns-594b88dc6c-pj24l 1/1 Running 0 19m
kube-multus-ds-5nqt9 1/1 Running 0 9m6s
kube-proxy-cdbj4 1/1 Running 0 9m6s
multus-route53-service-operator-645c94c4bb-5c4wf 1/1 Running 0 19m
multus-secondary-ip-ec2-mapping-operator-74586574d4-zrzwz 1/1 Running 0 19m

[ec2-user@ip-10-0-0-14 x86-Architecture]$ kubectl describe pod open5gs-smf-deployment-7f679b945-x79l4 --namespace open5gs
Name: open5gs-smf-deployment-7f679b945-x79l4
Namespace: open5gs
Priority: 0
Node: ip-10-0-2-39.ca-central-1.compute.internal/10.0.2.39
Start Time: Mon, 22 May 2023 20:01:00 +0000
Labels: epc-mode=smf
pod-template-hash=7f679b945
Annotations: k8s.v1.cni.cncf.io/network-status:
[{
"name": "aws-cni",
"interface": "dummybb591efea73",
"ips": [
"10.0.2.228"
],
"mac": "0",
"default": true,
"dns": {}
},{
"name": "open5gs/ipvlan-multus-sub-1-cp",
"interface": "net1",
"ips": [
"10.0.4.208"
],
"mac": "02:13:f3:9b:f0:3c",
"dns": {}
}]
k8s.v1.cni.cncf.io/networks: [ { "name": "ipvlan-multus-sub-1-cp", "interface": "net1" } ]
k8s.v1.cni.cncf.io/networks-status:
[{
"name": "aws-cni",
"interface": "dummybb591efea73",
"ips": [
"10.0.2.228"
],
"mac": "0",
"default": true,
"dns": {}
},{
"name": "open5gs/ipvlan-multus-sub-1-cp",
"interface": "net1",
"ips": [
"10.0.4.208"
],
"mac": "02:13:f3:9b:f0:3c",
"dns": {}
}]
route53-service-name: [ { "name": "s5.smf.open5gs.service", "multus-int": "ipvlan-multus-sub-1-cp" } ]
Status: Pending
IP: 10.0.2.228
IPs:
IP: 10.0.2.228
Controlled By: ReplicaSet/open5gs-smf-deployment-7f679b945
Init Containers:
init-smf:
Container ID: docker://349bab56b9e0e28c70594489ae364e8d96d504c30d2c17611f070f39e986dc84
Image: busybox:1.28
Image ID: docker-pullable://busybox@sha256:141c253bc4c3fd0a201d32dc1f493bcf3fff003b6df416dea4f41046e0f37d47
Port:
Host Port:
Command:
sh
-c
Args:
until nslookup gx.pcrf.open5gs.service >> /dev/null; do echo waiting for pcrf DNS record to be ready;done; until nslookup sx.upf.open5gs.service >> /dev/null; do echo waiting for upf DNS record to be ready; done
State: Running
Started: Mon, 22 May 2023 20:01:02 +0000
Ready: False
Restart Count: 0
Environment:
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s4zq9 (ro)
Containers:
smf:
Container ID:
Image: 523531615794.dkr.ecr.ca-central-1.amazonaws.com/hannanrepo:open5gs-x86-aio
Image ID:
Port:
Host Port:
Command:
/bin/sh
-c
Args:
sleep 10; open5gs-smfd -c /open5gs/config-map/smf.yaml;
State: Waiting
Reason: PodInitializing
Ready: False
Restart Count: 0
Environment:
Mounts:
/open5gs/config-map/diameter-smf.conf from open5gs-smf-diameter (rw,path="diameter-smf.conf")
/open5gs/config-map/smf.yaml from open5gs-smf-config (rw,path="smf.yaml")
/open5gs/diameter-ca/ from diameter-ca (rw)
/open5gs/tls/ from smf-tls (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-s4zq9 (ro)
Conditions:
Type Status
Initialized False
Ready False
ContainersReady False
PodScheduled True
Volumes:
open5gs-smf-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: open5gs-smf-config
Optional: false
open5gs-smf-diameter:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: open5gs-smf-diameter
Optional: false
diameter-ca:
Type: Secret (a volume populated by a Secret)
SecretName: diameter-ca
Optional: false
smf-tls:
Type: Secret (a volume populated by a Secret)
SecretName: smf-tls
Optional: false
kube-api-access-s4zq9:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional:
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: nodegroup=control-plane
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Normal Scheduled 6m4s default-scheduler Successfully assigned open5gs/open5gs-smf-deployment-7f679b945-x79l4 to ip-10-0-2-39.ca-central-1.compute.internal
Normal AddedInterface 6m3s multus Add eth0 [10.0.2.228/32] from aws-cni
Normal AddedInterface 6m3s multus Add net1 [10.0.4.208/24] from open5gs/ipvlan-multus-sub-1-cp
Normal Pulled 6m3s kubelet Container image "busybox:1.28" already present on machine
Normal Created 6m3s kubelet Created container init-smf
Normal Started 6m2s kubelet Started container init-smf
[ec2-user@ip-10-0-0-14 x86-Architecture]$

[mgonzalezo]

I'm facing same error for some pods in open5s ns:

[ec2-user@ip-10-1-0-111 x86-Architecture]$ kubectl get -n open5gs pods
NAME READY STATUS RESTARTS AGE
open5gs-hss-deployment-68b4dc778f-s9p95 0/1 Init:0/1 0 13m
open5gs-mme-deployment-77c4b4f8c-4t7kk 0/1 Init:0/1 0 13m
open5gs-nrf-deployment-54f9cd8b65-5dkhm 1/1 Running 0 13m
open5gs-pcrf-deployment-64ff487f65-85dqx 0/1 Init:0/1 0 13m
open5gs-sgwc-deployment-56944dcd88-ztz7r 0/1 Init:0/1 0 13m
open5gs-sgwu-deployment-76d4f6779b-m5qzq 1/1 Running 0 13m
open5gs-smf-deployment-7775846df8-wmf5k 0/1 Init:0/1 0 13m
open5gs-upf-deployment-78bccbffdd-ztpzc 1/1 Running 0 13m
open5gs-webui-78975d9746-q7d4v 1/1 Running 0 13m

kubectl logs:
[ec2-user@ip-10-1-0-111 x86-Architecture]$ kubectl logs -n open5gs open5gs-mme-deployment-77c4b4f8c-4t7kk
Defaulted container "mme" out of: mme, init-mme (init)
Error from server: Get "https://10.1.2.231:10250/containerLogs/open5gs/open5gs-mme-deployment-77c4b4f8c-4t7kk/mme": remote error: tls: internal error

Same error message for kube-system NS multus pods:

[ec2-user@ip-10-1-0-111 x86-Architecture]$ kubectl get -n kube-system pods
NAME READY STATUS RESTARTS AGE
aws-node-kr87b 2/2 Running 0 109m
aws-node-s75l7 2/2 Running 0 109m
coredns-58488c5db-hg956 1/1 Running 0 118m
coredns-58488c5db-mv9vj 1/1 Running 0 118m
kube-proxy-dll75 1/1 Running 0 109m
kube-proxy-rn2wk 1/1 Running 0 109m
multus-route53-service-operator-7bd78b8d7c-cp54s 0/1 CrashLoopBackOff 6 (98s ago) 7m17s
multus-secondary-ip-ec2-mapping-operator-75c68fb48c-tczb9 0/1 CrashLoopBackOff 6 (85s ago) 7m17s
[ec2-user@ip-10-1-0-111 x86-Architecture]$ kubectl logs -n kube-system multus-route53-service-operator-7bd78b8d7c-cp54s
Error from server: Get "https://10.1.2.158:10250/containerLogs/kube-system/multus-route53-service-operator-7bd78b8d7c-cp54s/shell-operator": remote error: tls: internal error
[ec2-user@ip-10-1-0-111 x86-Architecture]$

Error is observed after executing Staging environment- step 2 - "Run the ./cluster_initializer.sh"