kubernetes

Installation of kubernetes cluster in private cloud using kubeadm.

Configuration

The following table lists the configurable parameters of the kubernetes cluster install and their default values.

Parameter	Description	Default
CLUSTER_NAME	Kubernetes cluster name	cloud.com
MOUNT_PATH	The mount path where kubernetes scripts and certificates are available.	/root
DNS_DOMAIN	The domains where vms are hosted.	cloud.uat
CERTIFICATE_PATH	The path where certificates are present.	/etc/kubernetes/pki
SERVER_DNS	The comma separated dns names where kubernetes master would be running, this also includes name of the kubernetes api service dns names, the dns name where ha proxy is running	master.cloud.com..
SERVER_IP	The comma separated list of all the ip addresses where master and ha proxy would be running (actual & virtual). This should also include the ip addess of kubernetes cluster api service ip.	11.0.0.1,..
HA_PROXY_PORT	Port of HA Proxy.	6443
HA_PROXY_HOSTNAME	IP/Hostname where HA Proxy running.	11.0.0.1
LOAD_BALANCER_URL	Endpoint of HA Proxy.	11.0.0.1:6443
APP_HOST	Hostname that would be put in ingress.	master.cloud.com
API_SERVERS	List of api servers which are used in creating certificates for ha proxy.	11.0.0.1:master.clud.com,..
OIDC_ISSUE_URL	OpenID Connect issuer url.	https://skmaji.auth0.com/
OIDC_CLIENT_ID	OpenID Connect Application ID.	Client ID
OIDC_USERNAME_CLAIM	Field name in the ID Token for username claim.	sub
OIDC_GROUPS_CLAIM	Field name in the ID Token for group claim.	http://localhost:8080/claims/groups

Installation steps:

• Install Cluster in Digital Ocean

cd /root/kubernetes/install_cluster
./install_master_node.sh -e CLOUD

• Install Cluster in Local Ubuntu Cluster

cd /root/kubernetes/install_cluster
./install_master_node.sh -e LOCAL

• Install kubernetes:

  • Master:

  cd /root/kubernetes/install_k8s
  ./gok install kubernetes

  • Worker:

  cd /root/kubernetes/install_k8s
  ./gok install kubernetes-worker

  • To join a worker node:

  join

  • To remove a kubernetes setup:

  ./gok reset kubernetes

  • To create a new join token

  kubeadm token create print-join-command

When installation of master is completed you should see below components in kube-system namespace

When you reboot the vms, the kubelet service may not run, you need to restart the kubelet.

sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
sudo swapoff -a

systemctl stop kubelet
syatemctl start kubelet

Or Execute below command

./gok start proxy

Other debugging steps

systemctl status kubelet
journalctl -u kubelet

Useful commands:

• To change namespace

kcd name_space

• Inorder to login via OpenId Connect username/password

alias kctl='kubectl --kubeconfig=/root/oauth.conf --token=$(python3 /root/kubernetes/install_k8s/kube-login/cli-auth.py)'

In order to use the above approach, you must install and run

1. Ingress Instress ReadME
2. Kubeauthentication service KubeAuth ReadME

• To enable verbose(logging) of kubectl command

kubectl get pods --v 6

Useful commands

kubectl top nodes --v 6

kubectl get pod, svc, hpa -owide
watch -n1 !!

kubectl get componentstatus

Automation of build and deployment

Note: Ingress should be installed before this can begin.

• Execute below command to install ci/cd pipeline

cd /root/kubernetes/install_k8s/
./setuUp-devops.sh 

This would install local docker registry, Git webhook, Docker webhook and Helm webhook

Architecture of the pipeline

In order to test the pipeline use hlw repository, make a sample change and commit.

Application would be accessible in https://master.cloud.com:32028/hlw

Installing components

1. Ingress Controller
2. Cert Manager
3. Keycloak
4. Kube Login Webhook
5. Devops: Githook, Dockerhook,Reghook
6. Kubernetes Dashboard
7. Kube Authentication
8. Ldap: Link
9. Kerberos: Link
10. Ldap Client: Link
11. Ldap Client2: Link
12. Spinnaer
13. Opensearch
14. Prometheus-Grafana
15. Oauth2 Proxy
16. Fluentd

• Add ca.crt and server.crt file in chrome browser, please refer link on how to add certificate. Add server.crt in Other People tab and ca.crt in Trusted Root Certificate Authority tab.
• Add ip_address master.cloud.com to windows host file located in C:\Windows\System32\drivers\etc. e.g. 192.168.1.5 master.cloud.com >> host file.