I've been trying to architect a cloud-based testing tool that would allow you to setup a testing "workspace" with one or more emulated SBCs (Beaglebone Black, cubieboard2, etc.) and also setup an ethernet network between devices (using open or custom proprietary protocols).
I originally planned to build this tool using QEMU running on an x86 host and found initial testing to be painfully slow.
I'm wondering what my best options are for accelerating performance, could I emulate these SBCs on AWS bare-metal ARM instances? I know AWS doesn't currently support nested virtualization so I'm not sure of the scope of my options here. What would be the best approach to virtualize these testing setups and execute code at bare-metal hardware speeds?
The Arm Virtualization Extensions don't support "emulate an old v7 CPU on a v8 host" (you can emulate a 32-bit guest, but it will be a 32-bit guest that sees a v8 CPU, specifically the one your host has). So QEMU won't let you virtualize this kind of board model using KVM. You also can't virtualize TrustZone (EL3) under KVM, so if your guest expects to run at EL3 then you're out of luck. Similarly you can't run a guest that expects EL2 under KVM either, unless your host CPU is new enough to have the nested virtualization extensions.
You could hack something together that used KVM and just gave the guest a v8 CPU with the devices/etc your SBC has, and hope that the guest kernel can deal with that odd hybrid setup. Whether that's useful depends on what you're trying to test.
In my experience a lot of people are happy with "the thing I'm trying to test just needs a standard Linux userspace on Arm, and it doesn't care about the specific details of the hardware and devices" (in which case you can use QEMU/KVM and its 'virt' board, and just give the guest the same kind of CPU as the host). Another group of people really care about the details of the hardware because they're trying to test device drivers or other talks-to-the-hardware code; often here the fidelity of the device emulation in QEMU is not sufficient and these people find they need to work with a board farm of the real hardware (either exclusively, or in addition to a QEMU-based setup for smoketests and developer convenience). I'm not sure about the size of the group in the middle who need something that looks like a specific board but are happy with it being a relatively low-fidelity model of that board...