I can set up an arm64 docker image, but can't run anything

Issue

I have a CentOS 7 x86_64 virtual machine. I need to do some things in an Oracle Linux 8 arm64 docker container.

I installed docker and qemu:

$ sudo yum install -y yum-utils
$ sudo yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo 
$ sudo yum install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin \
 qemu-system-arm qemu qemu-user qemu-kvm qemu-kvm-tools libvirt virt-install \
 libvirt-python libguestfs-tools-c

I then started docker and set up qemu-user-static:

$ sudo systemctl start docker
$ docker run --rm --privileged multiarch/qemu-user-static --reset

My Dockerfile is:

FROM oraclelinux:8.5
ENV container docker
RUN dnf -y install sssd nscd unzip zip xz which file
RUN useradd --uid 1001 -ms /bin/bash cm
RUN uname -m

Then created x86_64 and arm64 images using the following:

docker buildx create --name ol8_builder
docker buildx --builder ol8_builder build -t frickson/oracle8 . --platform=linux/amd64 --load --progress=plain
docker buildx --builder ol8_builder build -t frickson/oracle8arm . --platform=linux/arm64 --load --progress=plain

During the build on both images, the RUN useradd works and the user is created. The RUN uname -m works on both systems, producing the correct output (x86_64 and aarch64 respectively).

The x86_64 machine works correctly:

$ docker run --rm --privileged frickson/oracle8 uname -m
x86_64

But the arm64 machine does not:

$ docker run --platform linux/arm64 --rm --privileged frickson/oracle8arm uname -a
exec /usr/bin/uname: no such file or directory

I'm guessing it's not finding one or more libraries, since it worked during the image build. Trying to run any command (bash, ls, true) all result in the same error message.

Any ideas what I'm missing? Thanks!

EDIT: In response to @JosephSible-ReinstateMonica:

It didn't work. And I don't have access to an ARM host to check my image.

$ sudo docker run --platform linux/arm64 -it --rm --privileged oraclelinux:8 /bin/bash
Unable to find image 'oraclelinux:8' locally
8: Pulling from library/oraclelinux
e12ce518ed4b: Pull complete 
Digest: sha256:cad3a3dc2dd9f03dec02642dce1f194562e7f21820bc36ceb9f7b070d337fb45
Status: Downloaded newer image for oraclelinux:8
exec /bin/bash: no such file or directory

Solution

When you run docker run --rm --privileged multiarch/qemu-user-static --reset, it sets the handler for aarch64 binaries to:

[root@localhost ~]# cat /proc/sys/fs/binfmt_misc/qemu-aarch64
enabled
interpreter /usr/bin/qemu-aarch64-static
flags:
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff

Unfortunately, CentOS 7 doesn't have the -static versions of these binaries:

[root@localhost ~]# ls /usr/bin/qemu-aarch64*
/usr/bin/qemu-aarch64

Fortunately, the multiarch/qemu-user-static container includes the necessary binaries in the image; you can copy onto your host like this:

$ docker run --rm  --entrypoint tar multiarch/qemu-user-static \
    -C /usr/bin -cf- . | tar -C /usr/bin -xf-

Now we have:

[root@localhost bin]# ls /usr/bin/qemu-aarch64*
/usr/bin/qemu-aarch64  /usr/bin/qemu-aarch64_be-static  /usr/bin/qemu-aarch64-static

This gets us so close, but we need to mount the static binary into the image as described in the docs for multiarch/qemu-user-static:

[root@localhost bin]# docker run --platform linux/arm64 --rm -v /usr/bin/qemu-aarch64-static:/usr/bin/qemu-aarch64-static  oraclelinux:8.5 uname -a
Linux 2d7e2e0ce6d7 3.10.0-1160.66.1.el7.x86_64 #1 SMP Wed May 18 16:02:34 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

(Or you need to build the interpreter into the image.)

We need to mount qemu-aarch64-static inside the container on CentOS 7 because the kernel doesn't appear to support the F flag for binfmt_misc interpreters. The F flag means:

F - fix binary

The usual behaviour of binfmt_misc is to spawn the binary lazily when the misc format file is invoked. However, this doesn’t work very well in the face of mount namespaces and changeroots, so the F mode opens the binary as soon as the emulation is installed and uses the opened image to spawn the emulator, meaning it is always available once installed, regardless of how the environment changes.

(from the kernel documentation)

multiarch/qemu-user-static will set the F flag if you set --persistent yes on the command line, but on CentOS 7 this fails with:

$ docker run --rm --privileged multiarch/qemu-user-static --reset
--persistent yes
...
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
sh: write error: Invalid argument

---

With recent Fedora distributions (and probably others as well), you can simply install the qemu-user-static package and everything will Just Work (no need to use the multiarch/qemu-user-static image):

$ cat /etc/fedora-release
Fedora release 34 (Thirty Four)
$ rpm -q qemu-user-static
qemu-user-static-5.2.0-9.fc34.x86_64
$ uname -m
x86_64
$ docker run --platform linux/arm64 --rm oraclelinux:8.5 uname -a
Linux 6f04d17234f5 5.17.6-100.fc34.x86_64 #1 SMP PREEMPT Mon May 9 14:41:31 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

The qemu-user-static package installs all the binfmt_misc registrations for you.

Answered By - larsks