Debian 11 kernel doesn't boot with vz

[jay7x]

Description

I see (a bit different) connectivity issues when starting VMs of VZ type on MacOS 15.1.

Debian 11 VM cannot pass the Waiting for the essential requirement 1 of 2: "ssh" stage:

INFO[0000] [hostagent] Waiting for the essential requirement 1 of 2: "ssh" 
INFO[0010] [hostagent] Waiting for the essential requirement 1 of 2: "ssh" 
INFO[0014] [hostagent] 2024/11/06 20:51:49 tcpproxy: for incoming conn 127.0.0.1:49488, error dialing "192.168.5.15:22": connect tcp 192.168.5.15:22: no route to host 
[...]

QEMU-type VM runs fine.

Debian 11 debug log

➜  ~ limactl create --name=d11test --vm-type=vz --tty=false template://debian-11 
INFO[0000] Terminal is not available, proceeding without opening an editor 
WARN[0000] vmType vz: ignoring [User]                   
INFO[0000] Attempting to download the image              arch=aarch64 digest="sha512:9851ed1b385a62b1f925616595e129c7db6efbcc08993b5136177f6fe68c6b16ff4c53ec747101cc3f66c64ffbf5ffffc7d8dab9b4065d77a8c3f07be80d9cb6" location="https://cloud.debian.org/images/cloud/bullseye/20241007-1893/debian-11-genericcloud-arm64-20241007-1893.qcow2"
INFO[0000] Using cache "/Users/jay/Library/Caches/lima/download/by-url-sha256/b2c0c127afc170294fd0d5dd6c2f866f946cb471cd1334a1e07c7ac64348aab1/data" 
INFO[0000] Converting "/Users/jay/.lima/d11test/basedisk" (qcow2) to a raw disk "/Users/jay/.lima/d11test/diffdisk" 
2.00 GiB / 2.00 GiB [---------------------------------------] 100.00% 2.33 GiB/s
INFO[0001] Expanding to 100GiB                          
INFO[0001] Attempting to download the nerdctl archive    arch=aarch64 digest="sha256:fe085381a09aa240ae5d1e0bbef1beccfb7c1d6dbb98bdc55bd416581d46ebc8" location="https://github.com/containerd/nerdctl/releases/download/v2.0.0/nerdctl-full-2.0.0-linux-arm64.tar.gz"
INFO[0001] Using cache "/Users/jay/Library/Caches/lima/download/by-url-sha256/1699e54a52757df863155fca76f8a77b50f05d993edca23421798af6635156f0/data" 
INFO[0001] Run `limactl start d11test` to start the instance. 
➜  ~ limactl start --debug d11test                                              
DEBU[0000] interpreting argument "d11test" as an instance name 
DEBU[0000] ResolveVMType: resolved VMType "vz" (explicitly specified in []*LimaYAML{o,y,d}[1]) 
INFO[0000] Using the existing instance "d11test"        
DEBU[0000] ResolveVMType: resolved VMType "vz" (explicitly specified in []*LimaYAML{o,y,d}[1]) 
DEBU[0000] ResolveVMType: resolved VMType "qemu" (existing instance, without "/Users/jay/.lima/docker/vz-identifier") 
WARN[0000] provisioning scripts should not reference the LIMA_CIDATA variables 
DEBU[0000] ResolveVMType: resolved VMType "qemu" (existing instance, without "/Users/jay/.lima/kexecboot/vz-identifier") 
DEBU[0000] Make sure "host" network is stopped          
DEBU[0000] Make sure "user-v2" network is stopped       
DEBU[0000] Make sure usernet network is stopped         
DEBU[0000] Make sure "shared" network is stopped        
DEBU[0000] Make sure "bridged" network is stopped       
INFO[0000] Starting the instance "d11test" with VM driver "vz" 
WARN[0000] vmType vz: ignoring [User]                   
DEBU[0000] [hostagent] ResolveVMType: resolved VMType "vz" (explicitly specified in []*LimaYAML{o,y,d}[1]) 
INFO[0000] [hostagent] debug mode detected, adding more guest agent candidates: /Users/jay/_output/share/lima/lima-guestagent.Linux-aarch64 
DEBU[0000] [hostagent] Creating iso file /Users/jay/.lima/d11test/cidata.iso 
DEBU[0000] [hostagent] Using /var/folders/nm/zwknzrjs7831bt_gvzhs2r5w0000gn/T/diskfs_iso3455402432 as workspace 
DEBU[0000] [hostagent] Failed to detect CPU features. Assuming that AES acceleration is available on this Apple silicon. 
DEBU[0000] [hostagent] OpenSSH version 9.8.1 detected   
DEBU[0000] [hostagent] AES accelerator seems available, prioritizing [email protected] and [email protected] 
INFO[0000] [hostagent] hostagent socket created at /Users/jay/.lima/d11test/ha.sock 
INFO[0000] [hostagent] Starting VZ (hint: to watch the boot progress, see "/Users/jay/.lima/d11test/serial*.log") 
DEBU[0000] [hostagent] Start udp DNS listening on: 127.0.0.1:64724 
DEBU[0000] [hostagent] Start tcp DNS listening on: 127.0.0.1:49470 
DEBU[0000] [hostagent] Kernel file "/Users/jay/.lima/d11test/kernel" not found 
DEBU[0000] [hostagent] Using EFI Boot Loader            
DEBU[0000] [hostagent] new connection from  to          
DEBU[0000] [hostagent] [VZ] - vm state change: "VirtualMachineStateStarting" 
DEBU[0000] received an event                             event="{2024-11-06 20:48:56.620883 +0800 +08 {false false false [] 49469}}"
INFO[0000] SSH Local Port: 49469                        
INFO[0000] [hostagent] [VZ] - vm state change: running  
INFO[0000] [hostagent] Waiting for the essential requirement 1 of 2: "ssh" 
DEBU[0000] [hostagent] executing script "ssh"           
DEBU[0000] [hostagent] executing ssh for script "ssh": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile="/Users/jay/.lima/_config/user" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers="^[email protected],[email protected]" -o User=jay -o ControlMaster=auto -o ControlPath="/Users/jay/.lima/d11test/ssh.sock" -o ControlPersist=yes -p 49469 127.0.0.1 -- /bin/bash -c $'while read -r line; do [ -n "$line" ] && export "$line"; done<<EOF\n$(sudo cat /mnt/lima-cidata/param.env)\nEOF\n/bin/bash'] 
DEBU[0000] [hostagent] stdout="", stderr="ssh: connect to host 127.0.0.1 port 49469: Connection refused\r\n", err=failed to execute script "ssh": stdout="", stderr="ssh: connect to host 127.0.0.1 port 49469: Connection refused\r\n": exit status 255 
INFO[0010] [hostagent] Waiting for the essential requirement 1 of 2: "ssh" 
DEBU[0010] [hostagent] executing script "ssh"           
DEBU[0010] [hostagent] executing ssh for script "ssh": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile="/Users/jay/.lima/_config/user" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers="^[email protected],[email protected]" -o User=jay -o ControlMaster=auto -o ControlPath="/Users/jay/.lima/d11test/ssh.sock" -o ControlPersist=yes -p 49469 127.0.0.1 -- /bin/bash -c $'while read -r line; do [ -n "$line" ] && export "$line"; done<<EOF\n$(sudo cat /mnt/lima-cidata/param.env)\nEOF\n/bin/bash'] 
INFO[0014] [hostagent] 2024/11/06 20:49:09 tcpproxy: for incoming conn 127.0.0.1:49472, error dialing "192.168.5.15:22": connect tcp 192.168.5.15:22: no route to host 
DEBU[0013] [hostagent] stdout="", stderr="kex_exchange_identification: read: Connection reset by peer\r\nConnection reset by 127.0.0.1 port 49469\r\n", err=failed to execute script "ssh": stdout="", stderr="kex_exchange_identification: read: Connection reset by peer\r\nConnection reset by 127.0.0.1 port 49469\r\n": exit status 255 
INFO[0023] [hostagent] Waiting for the essential requirement 1 of 2: "ssh" 
DEBU[0023] [hostagent] executing script "ssh"           
DEBU[0023] [hostagent] executing ssh for script "ssh": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile="/Users/jay/.lima/_config/user" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers="^[email protected],[email protected]" -o User=jay -o ControlMaster=auto -o ControlPath="/Users/jay/.lima/d11test/ssh.sock" -o ControlPersist=yes -p 49469 127.0.0.1 -- /bin/bash -c $'while read -r line; do [ -n "$line" ] && export "$line"; done<<EOF\n$(sudo cat /mnt/lima-cidata/param.env)\nEOF\n/bin/bash'] 
INFO[0027] [hostagent] 2024/11/06 20:49:22 tcpproxy: for incoming conn 127.0.0.1:49473, error dialing "192.168.5.15:22": connect tcp 192.168.5.15:22: no route to host 
DEBU[0026] [hostagent] stdout="", stderr="kex_exchange_identification: read: Connection reset by peer\r\nConnection reset by 127.0.0.1 port 49469\r\n", err=failed to execute script "ssh": stdout="", stderr="kex_exchange_identification: read: Connection reset by peer\r\nConnection reset by 127.0.0.1 port 49469\r\n": exit status 255 
INFO[0036] [hostagent] Waiting for the essential requirement 1 of 2: "ssh" 
DEBU[0036] [hostagent] executing script "ssh"           
DEBU[0036] [hostagent] executing ssh for script "ssh": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile="/Users/jay/.lima/_config/user" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers="^[email protected],[email protected]" -o User=jay -o ControlMaster=auto -o ControlPath="/Users/jay/.lima/d11test/ssh.sock" -o ControlPersist=yes -p 49469 127.0.0.1 -- /bin/bash -c $'while read -r line; do [ -n "$line" ] && export "$line"; done<<EOF\n$(sudo cat /mnt/lima-cidata/param.env)\nEOF\n/bin/bash'] 
INFO[0040] [hostagent] 2024/11/06 20:49:35 tcpproxy: for incoming conn 127.0.0.1:49474, error dialing "192.168.5.15:22": connect tcp 192.168.5.15:22: no route to host 
DEBU[0039] [hostagent] stdout="", stderr="kex_exchange_identification: read: Connection reset by peer\r\nConnection reset by 127.0.0.1 port 49469\r\n", err=failed to execute script "ssh": stdout="", stderr="kex_exchange_identification: read: Connection reset by peer\r\nConnection reset by 127.0.0.1 port 49469\r\n": exit status 255 
[...]

[AkihiroSuda]

Does debian-12 work?

Do you see other errors in /Users/jay/.lima/d11test/*.log ?

[afbjorklund]

If it doesn't boot at all, you might want to try with --video

[jay7x]

Does debian-12 work?

Yes, I tried Debian 12 and Ubuntu 20.04 and it was fine. I'll recheck on Almalinux 8 today also.

Do you see other errors in /Users/jay/.lima/d11test/*.log ?

I'll recheck and update later today. IIRC there was nothing else (serialv.log was empty, so maybe only ha.stdout.log has something else)

[AkihiroSuda]

debian-11 hangs on Intel Mac too (macOS 15.1)

Probably the kernel of Debian 11 has a compatibility issue with Virtualization.framework

[afbjorklund]

Probably needs https://packages.debian.org/bullseye-backports/kernel/ and a custom image.

[jay7x]

I guess it's easier to say that Debian 11 works in QEMU mode only maybe..