maybe it is a bit too "fat" for your liking, but what i liked for testing a lot of different systems is proxmox[1]
you can also run it (instead of a headless server install) just as a normal operating system as well. In the end it boils down to KVM/QEMU. It is pretty nice to work with and you can passthrough your GPU to a specific VM if you like. For some time i used a raspi4 as a thin client and connected via SPICE to my vm's with shared storage it was pretty nice to work with.


[1] https://www.proxmox.com