TL;DR:

This is a guide on running 64-bit ARM operating systems on Proxmox (amd64) via emulation. Although cloudinit is the charm in here, an ordinary ISO mount for manually installing an OS works as well.

This guide assumes previous experience in setting up a VM on Proxmox VE Web UI and acquaintance with CLI.

Following are the steps taken in an ordered list. Terminal use is minimal, however, checkout Techno Tim’s Notes and Proxmox VE documentation for terminal use for creating and modifying a VM. If have not set cloudinit before, watching Techno Tim’s video is a good start.

Why?

Beside showing off, the main utility in this that I can think of is pipelining software tests for an arm64 platform. Yet, the problem with that is the absolute necessity of having a generic enough work load and not being able to validate that on another platform (say, amd64/x86_64).

That would not fly with an embedded platform due to hardware configurations being too specific, so the reason could be either tests with lacking finances for an early “migration/expension to ARM” or “because I can”. Let’s dive in…

Observations

  1. Machine (motherboard) type must be i440fx. For q35, following error was encountered. I honestly do not know the reason, seems like enumeration error for whatever reason.

    1

    qemu-system-aarch64: -device usb-ehci,id=ehci,bus=pci.0,addr=0x1: Duplicate ID 'ehci' for device

  2. BIOS type should be OVMF (UEFI), VM would not boot otherwise.

    • This should be due to ARM family not having the booting conventions of x86 family. Necessity of EDK2 must be enforcing UEFI just like on Android firmwares. Don’t quote me on this.

  3. EDK2 firmware for ARM must be installed on any hypervisor node to run/store arm64 VMs: pve-edk2-firmware-aarch64.

    • EFI disk must be added after the fact that CPU type is changed on config file.
    • Ensure APT sources are properly configured for either of subscription and non-subscription repositories.

  4. Disk types:

    • VirtIO Block Device works flawlessly.
    • SCSI
      • Only VirtIO SCSI works.
      • VMWare compatible and VirtIO SCSI Single errs with: 
        qemu-system-aarch64: -device virtio-scsi-pci,id=virtioscsi0,bus=pcie.3,addr=0x1: Bus 'pcie.3' not found
      • MegaRAID cards (e.g. LSI 53C895A) do let VM to start, but it cannot boot due to lack of driver/kernel module.
    • IDE errs with:
    qemu-system-aarch64: -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,rotation_rate=1: Bus 'ide.0' not found

  5. Necessary changes in config file, these are not present on WebUI thus should be performed over CLI:

    • arch: aarch64 must be added.
    • vmgenid: must be removed or errs with: 
      qemu-system-aarch64: -device vmgenid,guid=c2826a61-f1e6-44b3-876a-b5f82a0d17cb: 'vmgenid' is not a valid device model name
    • cpu: easiest way is to remove, or set to max. Available options are listed at QEMU documentation. However for instance with cortex-a55 VM warns with vm 100 - unable to parse value of 'cpu' - Built-in cputype 'cortex-a55' is not defined (missing 'custom-' prefix?) and continues to boot. Cortex A55 Welcome GRUB

  6. It is best to start and open console (Serial 0) after a few seconds. Otherwise, screen may not be properly updated until user input, which is not ideal.

Step-by-Step

Pop-out a shell window for the target hypervisor node and change directory into a new workspace:

1
2

mkdir -p ~/prj/arm-deb-cloud
cd ~/prj/arm-deb-cloud

  1. Download Debian Bookworm: latest generic cloud image as qcow2 onto the target node. Proof read to ensure it is arm64 not amd64 to not loose an hour trying to boot the VM, don’t ask me why!

    1

    wget https://cloud.debian.org/images/cloud/bookworm/latest/debian-12-genericcloud-arm64.qcow2

  2. Create a diskless VM, do not add EFI disk as well:

    • create_vm_01
    • create_vm_02
    • create_vm_03
    • create_vm_04
    • create_vm_05
    • create_vm_06
    • create_vm_07
    • create_vm_08

  3. Modify VM config file by-hand to set CPU architecture and change (or remove) CPU model. File: nano /etc/pve/qemu-server/20001.conf

     1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

    <<<<<<<
cores: 6
cpu: host
ide2: none,media=cdrom
=======
cores: 6
arch: aarch64
cpu: max
ide2: none,media=cdrom
>>>>>>>

<<<<<<<
vmgenid: c2826a61-f1e6-44b3-876a-b5f82a0d17cb
=======
# removed vmgenid
>>>>>>>

  4. Set UEFI firmware

    1. Install the firmware package or following error would be encountered while cloning. This is the generic UEFI firmware (pre-bootloader) that is also present in Android firmwares by TianoCore. Trivia: the Proxmox logo on VM start would actually spell TianoCore by default on KVM/QEMU. 
      1
2

      # ensure you have subscription or no-subscription repository working
apt install -y pve-edk2-firmware-aarch64
      • If it is missing, then errs with:
      1

      TASK ERROR: clone failed: EFI base image '/usr/share/pve-edk2-firmware//AAVMF_CODE.fd' not found
    2. Add EFI disk. This simply instantiates the correct firmware after architecture change. add-efi-disk-after-arch-change
      • If you had mistakenly added an EFI disk in step 2, then following error would be encountered while cloning.
      1

      TASK ERROR: clone failed: command 'qemu-img dd -n -O raw 'bs=1048576' 'osize=67108864' 'if=rbd:pool1/base-20001-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/pool1.keyring' 'of=rbd:pool1/vm-20002-disk-0:conf=/etc/pve/ceph.conf:id=admin:keyring=/etc/pve/priv/ceph/pool1.keyring'' failed: exit code 1
    3. Ensure you have installed firmware package on the clone target node, if in a cluster, or following error would be encountered while starting the VM. 
      1

      TASK ERROR: EFI base image '/usr/share/pve-edk2-firmware//AAVMF_CODE.fd' not found

  5. Attach a copy of downloaded qcow2 file as a VM disk. File downloaded can be deleted afterwards.

    1
2

    qm importdisk 20001 ./debian-12-genericcloud-arm64.qcow2 pool1
# rm ./debian-12-genericcloud-amd64.qcow2

    Better select discard. add-virtio-disk

  6. Add Cloudinit. This generation needs to be overwritten on the clones.

    • Must be SCSI. add-cloudinit-scsi
    • Regenerate CloudInit. regenerate-cloudinit
    • You can check on boot order to see if it worked. check-boot-order-for-cloudinit

  7. Must remove IDE CD/DVD Drive. Even if you will use an ISO image, IDE would hinder the boot so re-add as SCSI.

    • Could simply remove ide2: none,media=cdrom from config file as well. remove-cd-drive

  8. Set boot order to system disk. Cloudinit need not be enabled. set-boot-order

  9. Turn VM into a template. (Could be done in any step) convert-to-template

  10. Clone the template. Any of the errors mentioned so far would be encountered at this step or at starting the cloned VM. clone-vm-to-pve1

  11. Start VM

    • Increase disk size. resize-virtio-disk

    • Set User set-user

    • Password set-password

    • SSH keys. Created a key by: ssh-keygen -t ed25519 -f ~/.ssh/keypair/hlab2-cbugk -C "" -N "" set-ssh-key

    • Set network, IPv4 DHCP in this case: set-network-dhcp

      • If network is not set, the VM stalls indefinitely: forgot-cloudinit-network

    • Start will result in regeneration of cloudinit ISO.

      generating cloud-init ISO

      booted-up

  1. If you want to install with an ISO, e.g. Fedora 40 IoT arm64.

    • Add CD/DVD drive add-dvd-drive
    • Set as first boot device set-dvd-boot-order

  2. Install OS: install-fedora-serial

  3. Remember to remove CD/DVD drive.

End result

Resulting config files are as follows.

  • /etc/pve/qemu-server/20001.conf on pve3:
agent: 1
arch: aarch64
balloon: 4096
bios: ovmf
boot: order=virtio0
cores: 6
cpu: max
efidisk0: pool1:base-20001-disk-1,efitype=4m,pre-enrolled-keys=1,size=64M
memory: 8192
meta: creation-qemu=8.1.2,ctime=1713062225
name: arm-deb-cloud
net0: virtio=BC:24:11:2F:83:17,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: pool1:vm-20001-cloudinit,media=cdrom
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=79271524-d044-4ceb-9684-dd1282afff38
sockets: 1
tags: arm64;cloudinit
template: 1
vga: serial0
virtio0: pool1:base-20001-disk-0,discard=on,iothread=1,size=2G
  • /etc/pve/qemu-server/20002.conf on pve1:
agent: 1
arch: aarch64
balloon: 4096
bios: ovmf
boot: order=scsi1;virtio0
cipassword: # REDACTED
ciuser: cbugk
cores: 6
cpu: max
efidisk0: pool1:vm-20002-disk-0,efitype=4m,pre-enrolled-keys=1,size=64M
hotplug: disk,network,usb
ipconfig0: ip=dhcp
memory: 8192
meta: creation-qemu=8.1.2,ctime=1713062225
name: arm-vm-1
net0: virtio=BC:24:11:4C:68:FB,bridge=vmbr0,firewall=1
numa: 0
ostype: l26
scsi0: pool1:vm-20002-cloudinit,media=cdrom,size=4M
scsi1: local-btrfs:iso/Fedora-IoT-ostree-40-20240319.2.aarch64.iso,media=cdrom,size=2457002K
scsihw: virtio-scsi-pci
serial0: socket
smbios1: uuid=0d2efdf5-32bd-4eb5-a1c0-ade1f909cd1e
sockets: 1
sshkeys: ssh-ed25519%20AAAAC3NzaC1lZDI1NTE5AAAAIAVWLc79p258262z5EF%2FRnYYw9KvbYEiiSNXYHqFTCSs
tags: arm64;cloudinit
vga: serial0
virtio0: pool1:vm-20002-disk-1,discard=on,iothread=1,size=29G

Bibliography

  1. Techno Tim’s Proxmox Cloudimage Cloudinit Video
  2. Techno Tim’s Notes on 1
  3. i12bretro’s tutorial
  4. OCTigg’s reddit dump
  5. Roletok’s ARM64 Debian 10 Guide (found while writing, a concise summary)