qemu-system-aarch64 always entering exception level 2 and hangs when switching to EL1. Why?

Issue

I am using qemu-system-aarch64 to emulate raspi3 (ARM Cortex A-53) on a x86 host. Here's the command that I use to launch the baremetal image :

$ qemu-system-aarch64 --version
QEMU emulator version 4.2.1 (Debian 1:4.2-3ubuntu6.18)
Copyright (c) 2003-2019 Fabrice Bellard and the QEMU Project developers

$ qemu-system-aarch64 -M raspi3 -kernel kernel8.img -serial null -serial stdio -display none

The code for kernel8.img is taken from here : https://github.com/s-matyukevich/raspberry-pi-os/tree/master/src/lesson02/src . The following function is used to get the current exception level :

.globl get_el
get_el:
    mrs x0, CurrentEL
    lsr x0, x0, #2
    ret

The following piece of code tries to switch to EL1 on boot, which I had to comment out, otherwise this piece of code hangs indefinitely :

master:
    ldr x0, =SCTLR_VALUE_MMU_DISABLED
    msr sctlr_el1, x0       

    ldr x0, =HCR_VALUE
    msr hcr_el2, x0

    ldr x0, =SCR_VALUE
    msr scr_el3, x0

    ldr x0, =SPSR_VALUE
    msr spsr_el3, x0

    adr x0, el1_entry       
    msr elr_el3, x0

    eret        

After the above piece of code is commented, my program at least runs, but prints Exception level as 2.

I have following two questions :

1. As per the corresponding repo docs, when we are using a hypervisor, EL2 is used by the host OS so that the guest OS uses EL1 or EL0. But why my baremetal guest OS is entering EL2 by default? It should have been EL1 as per my understanding. Also, this older post on SO also had got EL1 on boot. Not sure if something changed or if I am missing anything.
2. Why the code to switch to EL1 hangs at line msr scr_el3, x0 ?

Solution

You are running at EL2. The instruction "msr scr_el3, x0" tries to write to an EL3-only register, and it will cause an UNDEF exception if you try it from a lower EL. In other words, the code fragment you're trying to use is written to run at EL3, and needs changes if you want to run it at EL2.

The reason your code is running at EL2 is because you have used QEMU's "-kernel" argument and passed it a binary (non-ELF) file, which is telling QEMU "You should boot me in the way that the Linux kernel booting protocol documents that a Linux kernel should be booted". That protocol says "start me in EL2 or EL1, EL2 is recommended", and so that's what QEMU does.

For more information on the various options QEMU provides for loading guest code, see this answer. Note that QEMU does not provide a "boot this image or ELF file in the way that the Raspberry PI bootrom code would boot it": you may have to adapt code intended to run on real bare-metal boards a little bit.

Answered By - Peter Maydell