kvm on arm64

[Ross Vandegrift]

(please CC me, as I'm not subscribed)

Hi folks,

New to running arm64 stuff on physical arm64 hardware, and I'm unable to start
a kvm guest. I'm sure I'm missing something, hoping someone can point me in
the right direction.

I'm running arm64 bookworm (plus vendor junk for orangepi 5 plus). Without
kvm, I can boot a trixie arm64 image [1] just fine. But if I enable kvm, qemu
spins without booting. It produces no output, not even the UEFI firmware
output.

This works:

dd if=/dev/zero of=flash1.img bs=1M count=64
dd if=/dev/zero of=flash0.img bs=1M count=64
dd if=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd of=flash0.img conv=notrunc
qemu-system-aarch64 \
-nographic \
-machine virt,gic-version=max \
-m 512M \
-cpu max \
-netdev user,id=vnet \
-device virtio-net-pci,netdev=vnet \
-drive file=debian-13-nocloud-arm64-daily-20230809-1467.qcow2,if=none,id=drive0 \
-device virtio-blk,drive=drive0,bootindex=0 \
-drive file=flash0.img,format=raw,if=pflash \
-drive file=flash1.img,format=raw,if=pflash \
-snapshot


But it breaks if I add -enable-kvm. Any hints?

Thanks,
Ross

[1] - https://cloud.debian.org/images/cloud/trixie/daily/20230809-1467/debian-13-nocloud-arm64-daily-20230809-1467.qcow2

[Emanuele Rocca]

Hi Ross,

Post by Ross Vandegrift
New to running arm64 stuff on physical arm64 hardware, and I'm unable to start
a kvm guest. I'm sure I'm missing something, hoping someone can point me in
the right direction.

[...]

Post by Ross Vandegrift
qemu-system-aarch64 \
-nographic \
-machine virt,gic-version=max \
-m 512M \
-cpu max \
-netdev user,id=vnet \
-device virtio-net-pci,netdev=vnet \
-drive file=debian-13-nocloud-arm64-daily-20230809-1467.qcow2,if=none,id=drive0 \
-device virtio-blk,drive=drive0,bootindex=0 \
-drive file=flash0.img,format=raw,if=pflash \
-drive file=flash1.img,format=raw,if=pflash \
-snapshot
But it breaks if I add -enable-kvm. Any hints?

Nothing wrong with your command, I've tried the exact same thing (with
-enable-kvm) and it works fine on my M1.

- What happens if you try with: -m 1G -smp 2, any differences?
- What does ls -l /dev/kvm say? Is your user in the kvm group?
- Anything interesting in dmesg?
- What if you add -serial file:serial.log ? Anything in serial.log?

[Ross Vandegrift]

Post by Emanuele Rocca

Post by Ross Vandegrift
But it breaks if I add -enable-kvm. Any hints?

Nothing wrong with your command, I've tried the exact same thing (with
-enable-kvm) and it works fine on my M1.

Thanks a bunch for confirming on other hardware, that makes me feel
better!

Post by Emanuele Rocca
- What happens if you try with: -m 1G -smp 2, any differences?

No change.

Post by Emanuele Rocca
- What does ls -l /dev/kvm say? Is your user in the kvm group?

All looks good:
$ ls -la /dev/kvm
crw-rw---- 1 root kvm 10, 232 Aug 10 22:30 /dev/kvm
$ groups
ross adm tty disk dialout sudo audio video plugdev games users input kvm netdev libvirt systemd-journal libvirt-qemu

Post by Emanuele Rocca
- Anything interesting in dmesg?

Nothing at all. :(

Post by Emanuele Rocca
- What if you add -serial file:serial.log ? Anything in serial.log?

No, even if I modify the image to enable early printk. Since I'm not
seeing the UEFI firmware banner, I think it's failing before the kernel
starts. Here's where my arm ignorance steps in:

- Is it possible my hardware is lacking virt support?
- Could my hardware not support UEFI even with qemu providing firmware?

Ross

[Ross Vandegrift]

Post by Ross Vandegrift
No, even if I modify the image to enable early printk. Since I'm not
seeing the UEFI firmware banner, I think it's failing before the kernel
- Is it possible my hardware is lacking virt support?

Not the issue - I extracted the kernel/initrd, and I can boot with kvm by
loading linux directly.

Post by Ross Vandegrift
- Could my hardware not support UEFI even with qemu providing firmware?

So something is going wrong with the UEFI firmware or qemu. I tried upgrading
to qemu-efi-aarch64 from sid, but it doesn't help. I looked into backporting a
newer qemu, but it'd take more time than I have tonight.

Ross

[Ross Vandegrift]

Post by Ross Vandegrift
So something is going wrong with the UEFI firmware or qemu. I tried upgrading
to qemu-efi-aarch64 from sid, but it doesn't help. I looked into backporting a
newer qemu, but it'd take more time than I have tonight.

Closing the loop on this: it's a bug in the crappy vendor kernel.
Armbian has trixie-based images with mainline 6.5-rc5 kernel and kvm
boots the UEFI firmware right up.

Ross