Why ARM cores behaving differently with an ELF and binary file

I am doing baremetal development on ARM and emulating Raspi 3 on QEMU. Below is my minimal assembly code :

.section ".text.boot"

.global _start
_start:
1:  wfe
    b 1b

Below is my linker script :

SECTIONS
{
    . = 0x80000;
    .text : {*(.text.boot)}

    /DISCARD/ : { *(.comment) *(.gnu*) *(.note*) *(.eh_frame*) }
}

Below is my Makefile :

CC = /opt/gcc-arm-10.3-2021.07-x86_64-aarch64-none-elf/bin/aarch64-none-elf
CFLAGS = -Wall -O2 -ffreestanding -nostdinc -nostartfiles -nostdlib -g

all: clean kernel8.img

start.o: start.S
    ${CC}-gcc $(CFLAGS) -c start.S -o start.o

kernel8.img: start.o
    ${CC}-ld -nostdlib start.o -T link.ld -o kernel8.elf
    ${CC}-objcopy -O binary kernel8.elf kernel8.img

clean:
    rm kernel8.elf kernel8.img *.o >/dev/null 2>/dev/null || true

Now from one terminal, I am loading my kernel8.elf like below :

$ /opt/qemu-6.2.0/build/qemu-system-aarch64 -M raspi3b -kernel kernel8.elf -display none -S -s

From another terminal, I connect my gdb :

$ /opt/gcc-arm-10.3-2021.07-x86_64-aarch64-none-elf/bin/aarch64-none-elf-gdb ./kernel8.elf -ex 'target remote localhost:1234' -ex 'break *0x80000' -ex 'continue'
(gdb) info threads
  Id   Target Id                    Frame
  1    Thread 1.1 (CPU#0 [running]) _start () at start.S:6
  2    Thread 1.2 (CPU#1 [running]) _start () at start.S:5
* 3    Thread 1.3 (CPU#2 [running]) _start () at start.S:5
  4    Thread 1.4 (CPU#3 [running]) _start () at start.S:5

My cores are OK in this case, as all the 4 cores are running my assembly code. Upon continue the cores randomly hit breakpoints, which is perfect.

However, if I use the kernel8.img (objcopy binary output) instead of kernel8.elf, I see that only Core 1 is running my assembly, but other 3 cores seem to be stuck. Upon continue only Core 1 repeatedly hits breakpoint everytime.

(gdb) info threads
  Id   Target Id                    Frame
* 1    Thread 1.1 (CPU#0 [running]) _start () at start.S:5
  2    Thread 1.2 (CPU#1 [running]) 0x0000000000000300 in ?? ()
  3    Thread 1.3 (CPU#2 [running]) 0x0000000000000300 in ?? ()
  4    Thread 1.4 (CPU#3 [running]) 0x0000000000000300 in ?? ()

I tried set scheduler-locking on and continue on other 3 cores, but they seem to be stuck.

Why the kernel8.img is not working as kernel8.elf? I expected all ARM cores to be running the same code on reset, (as is happening with kernel8.elf) but its not happening with kernel8.img.

Solution

The QEMU -kernel option treats the file it loads differently depending on whether it is an ELF file or not.

If it is an ELF file, it is loaded according to what the ELF file says it should be loaded as, and started by executing from the ELF entry point. If it is not an ELF file, it is assumed to be a Linux kernel, and started in the way that the Linux kernel's booting protocol requires.

In particular, for a multi-core board, if -kernel gets an ELF file it starts all the cores at once at the entry point. If it gets a non-ELF file then it will do whatever that hardware is supposed to do for loading a Linux kernel. For raspi3b this means emulating the firmware behaviour of "secondary cores sit in a loop waiting for the primary core to release them by writing to a 'mailbox' address. This is the behaviour you're seeing in gdb -- the 0x300 address that cores 1-3 are at is in the "spin in a loop waiting" code.

In general, unless your guest code is a Linux kernel or is expecting to be booted in the same way as a Linux kernel, don't use the -kernel option to load it. -kernel is specifically "try to do what Linux kernels want", and it also tends to have a lot of legacy "this seemed like a useful thing to somebody" behaviour that differs from board to board or between different guest CPU architectures. The "generic loader" is a good way to load ELF files if you want complete manual control for "bare metal" work.

For more info on the various QEMU options for loading guest code, see this answer.