Skip to content
This repository has been archived by the owner on Oct 30, 2022. It is now read-only.

Latest commit

 

History

History
96 lines (82 loc) · 4.88 KB

README.md

File metadata and controls

96 lines (82 loc) · 4.88 KB
Raw

kube-pet-node

Development has ceased

I no longer run this software. I ended up falling back to 'real' Kubernetes for cluster nodes. The experiment was fun, but I didn't really want to reimplement everything down to CrashLoopBackOff, and there were some struggles tracking Podman as that project continued shaping itself. If I did this again, I would use Typescript instead, and probably reuse more existing work like the Kubernetes CNI ecosystem.

Proof of Concept

dan@penguin ~> kubectl get nodes
NAME                               STATUS   ROLES    AGE     VERSION
pet-laxbox                         Ready    pet      11h     kube-pet/0.2.1-h76.6eb474f
pet-atlbox                         Ready    pet      45m     kube-pet/0.2.1-h76.6eb474f
gke-dust-persist3-190b8935-jvph    Ready    app      89d     v1.16.8-gke.15
gke-dust-volatile1-b331c9e9-fk37   Ready    app      2d16h   v1.16.8-gke.15

The concept is taking a managed cloud-based control plane and adding external bare-metal servers to it to be maintained as pets. I'm developing against a free GKE control plane using a Public control plane and Alias IP pod networking.

goalposts

I'm just tryina run some high-resource stuff out of the cloud so it's cheaper.

v1 goals

  • learn a shitton about what kubelet actually does and how a Kubernetes node works
  • run reasonably sane Kubernetes pods
  • request our own api certificates automatically
  • provide interactive container apis:
    • exec
    • logs
    • attach
    • metrics
  • expose static pod representing the host system (host exec & dmesg logs)
    • (static pods have annotations incl. kubernetes.io/config.source)
  • emit Event resources just like kubelet
    • image pulling
    • container lifecycle
  • image pull backoff
  • restart failed/finished containers
  • support imagepullsecrets
  • report our Internet address in node status (for dynamic dns purposes)
  • require as few permissions as possible - non-root, plus CAP_NET_ADMIN and access to a root podman
    • though, currently using a scoped sudoers file for installing volume contents and other misc tasks
  • SELinux compatibility (the Kubernetes fields, and also volume relabelling via annotation)
  • reasonable volume support
    • Check for secret/configmap updates
    • HostPath volumes
    • Secret volumes
    • ConfigMap volumes
    • Projected volumes (ServiceAccountToken, for EKS pod identity)
      • Automatic token rotation
    • Projected volumes (Secret & ConfigMap)
    • Projected volumes (Downward API)
    • EmptyDir volumes
    • NFS volumes
  • support downward IP in envars
  • include build version in our Node resources
  • support self-upgrading the node itself, on demand (e.g. update target version in a configmap)

stretch goals

  • report pod and node metrics
  • update mounted configmaps/secrets in pods
  • support init containers on pods (and eventually ephemeral containers)
  • support container readiness and liveness probes
  • kubectl port-forward
  • create CRDs to observe and maybe manipulate hardware devices (disk drives, TV tuners, etc)
    • loops probably distributed as a daemonset/deployment, even if it made in-project
  • support drone.io job pods (changing image of running containers)
  • support NetworkPolicy on pod networking
  • support registering pods into systemctl for bringing up at boot, even if kube-pet-node is broken
  • support running entirely rootless (incl. rootless podman), document the limitations (no pod IPs, probably no clusterips, etc)

non goals

  • [not] matching container lifecycle (ok to restart a container instead of replacing each time)
  • [not] feature compatible with kube-proxy (e.g. completely ignoring NodePort/LoadBalancer)
  • [not] supporting alternative CNI or CSI or CRI configurations (sticking with ptp/wg, hostpath, & podman for now)

deps

Current plans are to use all of these programs on the ideal pet node, though none are mission critical to have.

  • wireguard - tunnelling between the 'real' cluster and the pet node - you need some sort of networking for pods to talk properly, can be another VPN though
  • nftables (via library) - configure traffic for ClusterIP services, maybe also network policy - cluster IPs won't exist without nftables
  • podman (via REST API, new in 2.x) - run 'pods' on the host (similar to docker, but with its own pod concept) - required to run pods

opt-in extras for later:

  • zfs - report health and status of storage pools as a CRD, also probably volume provisioning
  • smartctl - report health of storage devices as a CRD
  • systemd - used for scheduling pods and tunnels for server boots
    • goal of the system being able to come up and serve Internet apps without kubernetes/kube-pet operational
  • journald - used internally by podman for storing/watching state changes