RKE2 airgap installation still needs access to docker registry

[Exobitt]

Environmental Info:
RKE2 Version:
v1.27.2+rke2r1

Node(s) CPU architecture, OS, and Version:
N/A

Cluster Configuration:
1 server, 3 agents.

Describe the bug:
Following the airgap instruction on your documentation, as we're deploying RKE2 in an air gapped environment, which doesn't have access to docker registry etc.
But, still when i look in containerd.log, i can see RKE2 is trying to pull https://registry-1.docker.io/v2/rancher/pause/manifests/3.6.
This, ofcourse, cannot happen, as its an airgapped environment.
Is this as expected?

Steps To Reproduce:

• Instal RKE2 in an airgapped environment without access to docker registry.
• Look in containerd log, when the service has started

Expected behavior:

The image should already have been added in your core package with images

Actual behavior:

The service halts, as it cannot pull the image

Additional context / logs:

N/A

[brandond]

when i look in containerd.log, i can see RKE2 is trying to pull https://registry-1.docker.io/v2/rancher/pause/manifests/3.6.

Loading of airgap images does not prevent containerd from attempting to reach out to the internet if images are not available locally. Can you confirm that:

• the images were successfully imported on startup
• the node has not experienced disk pressure that would cause it to prune images from the local image store

It is pretty unlikely that the pause image would get pruned since it is in use by pretty much every pod, so the more likely explanation is that the images were not placed in the correct location to be preloaded.

[Exobitt]

I imported all the images from the repository. Well, not all, but the Cilium images for the CNI, as well as the RKE2 core images. I even tried importing from the complete image packages.

The main issue was that I was unable to initiate the RKE2-server.service and the rke2-agent.service due to a certain "error". Upon further investigation, it was apparent that containerd was attempting to import the rancher/pause image from the Docker registry. However, because I am in an air-gapped environment, this was not successful.

I resolved this issue by extracting the image from the Docker registry, compressing it into a tarball, and placing it where all the other images were stored. This seemed to resolve all the issues and everything started to work properly.

[brandond]

The pause image (docker.io/rancher/pause:3.6) is included in the core airgap image tarball. You can verify this by looking at the image list text file, or by checking the log to see it being imported. Can you confirm specifically which of the tarballs you placed on the nodes, and provide the rke2-server journald logs showing the results of the image import process that runs during startup?

[Mjwagner2277]

This happened to me as well. Just commenting for follow up. Pretty new to RKE and doing it in an airgap can be a pain. This happens randomly for (seemingly) replicas on the master node.

[Mjwagner2277]

I think I will attempt to load this image our private registry.

[c3-jeremyhu]

Hi, running into this issue too. I saw on the airgap guide that the registry requires unauthenticated access (no password). By chance were you able to get it working in a registry that requires authentication?