QEMU AARCH64 "virt" Machine SMP CPUs Starting in "running" vs. "halted" State

Issue

I'm working on bare-metal. No Linux, libraries, etc. I'm writing processor boot code in ASM and jumping to my compiled C code.

My command line is:

% qemu-system-aarch64 \
   -s -S \
   -machine virt,secure=on,virtualization=on \
   -cpu cortex-a53 \
   -d int \
   -m 512M \
   -smp 4 \
   -display none \
   -nographic \
   -semihosting \
   -serial mon:stdio \
   -kernel my_file.elf \
   -device loader,addr=0x40004000,cpu-num=0 \
   -device loader,addr=0x40004000,cpu-num=1 \
   -device loader,addr=0x40004000,cpu-num=2 \
   -device loader,addr=0x40004000,cpu-num=3 \
   ;

When I connect gcc at the beginning, I can see:

(gdb) info threads
  Id   Target Id     Frame
 * 1   Thread 1.1 (CPU#0 [running]) _start () at .../start.S:20
   2   Thread 1.2 (CPU#1 [halted ]) _start () at .../start.S:20
   3   Thread 1.3 (CPU#2 [halted ]) _start () at .../start.S:20
   4   Thread 1.4 (CPU#3 [halted ]) _start () at .../start.S:20

I want those other three processors to start in the "running" state, not "halted". How?

Note that my DTS contains this section:

psci {
    migrate = < 0xc4000005 >;
    cpu_on = < 0xc4000003 >;
    cpu_off = < 0x84000002 >;
    cpu_suspend = < 0xc4000001 >;
    method = "smc";
    compatible = "arm,psci-0.2\0arm,psci";
};

However, I'm not sure what to do with that. Adding many different lines of this form, don't seem to help:

-device loader,addr=0xc4000003,data=0x80000000,data-len=4

I'm not sure if I'm on the right track with this ARM PSCI thing? ARM's specification seems to define the "interface", not the system "implementation". However, I don't see the PSCI as "real" registers mentioned in the "virt" documentation/source. There is no "SMC" device mentioned in the DTS.

How does QEMU decide whether an SMP processor is "running" or "halted" on start and how can I influence that?

---

Based on @Peter-Maydell's answer below, I need to do one of two things...

1. Switch "-kernel" to "-bios". I do this, but my code doesn't load as I expect. My *.elf file has several sections; some in FLASH and some in DDR (above 0x40000000). Maybe that's the problem?

2. Change my boot code to setup and issue the SMC instruction to make the ARM PSCI "CPU_ON" call that QEMU will recognize and powerup the other processors. Code like this runs but doesn't seem to "do" anything...

   ldr   w0, =0xc4000003   // CPU_ON code from the DTS file
   mov   x1, 1             // CPU #1 in cluster zero (format of MPIDR register?)
   ldr   x2, _boot         // Jump address 0x40006000 (FYI)
   mov   x3, 1             // context ID (meaningful only to caller)
   smc   #0                // GO!
   // result is in x0 -> PSCI_RET_INVALID_PARAMS
   

Solution

This depends on the board model -- generally we follow what the hardware does, and some boards start all CPUs from power-on, and some don't. For the 'virt' board (which is specific to QEMU) what we generally do is use PSCI, which is the Arm standard firmware interface for powering SMP CPUs up and down (among other things; you can also use it for 'power down entire machine', for instance). On startup only the primary CPU is running, and it's the job of guest code to use the PSCI API to start the secondaries. That's what that psci node in the DTS is telling the guest -- it tells the guest what specific form of the PSCI ABI QEMU implements, and in particular whether the guest should use the 'hvc' or 'smc' instruction to call PSCI functions. What QEMU is doing here is emulating a "hardware + firmware" combination -- the guest executes an 'smc' instruction and QEMU performs the actions that on real hardware would be performed by a bit of firmware code running at EL3.

The virt board does also have another mode of operation which is intended for when you want to run a guest which is itself EL3 firmware (for instance if you want to run OVMF/UEFI at EL3). If you start QEMU with -machine secure=true to enable EL3 emulation and you also provide a guest firmware blob via either -bios or -drive if=pflash,..., then QEMU will assume your firmware wants to run at EL3 and provide PSCI services itself, so it will start with all CPUs powered on and let the firmware deal with sorting them out.

A simple example of making a PSCI call to turn on another CPU (in this case cpu #4 of 8):

    .equ PSCI_0_2_FN64_CPU_ON, 0xc4000003
    ldr x0, =PSCI_0_2_FN64_CPU_ON
    ldr x1, =4         /* target CPU's MPIDR affinity */
    ldr x2, =0x10000   /* entry point */
    ldr x3, =0         /* context ID: put into target CPU's x0 */
    smc 0

Answered By - Peter Maydell