Podman in Podman, similar to Docker in Docker?

Issue

Is there a way to run Podman inside Podman, similar to the way you can run Docker inside Docker?

Here is a snippet of my Dockerfile which is strongly based on another question:

FROM debian:10.6

RUN apt update && apt upgrade -qqy && \
    apt install -qqy iptables bridge-utils \
                     qemu-kvm libvirt-daemon libvirt-clients virtinst libvirt-daemon-system \
                     cpu-checker kmod && \
    apt -qqy install curl sudo gnupg2 && \
    echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Debian_10/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list && \
    curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/Debian_10/Release.key | sudo apt-key add - && \
    apt update && \
    apt -qqy install podman

Now trying some tests:

$ podman run -it my/test bash -c "podman --storage-driver=vfs info"
... (long output; this works fine)

$ podman run -it my/test bash -c "podman --storage-driver=vfs images"
ERRO[0000] unable to write system event: "write unixgram @000ec->/run/systemd/journal/socket: sendmsg: no such file or directory" 
REPOSITORY  TAG     IMAGE ID  CREATED  SIZE

$ podman run -it my/test bash -c "podman --storage-driver=vfs run docker.io/library/hello-world"
ERRO[0000] unable to write system event: "write unixgram @000ef->/run/systemd/journal/socket: sendmsg: no such file or directory" 
Trying to pull docker.io/library/hello-world...
Getting image source signatures
Copying blob 0e03bdcc26d7 done  
Copying config bf756fb1ae done  
Writing manifest to image destination
Storing signatures
ERRO[0003] unable to write pod event: "write unixgram @000ef->/run/systemd/journal/socket: sendmsg: no such file or directory" 
ERRO[0003] Error preparing container 66692b7ff496775499d405d538769a078f2794549955cf2409fcbcbf87f42e94: error creating network namespace for container 66692b7ff496775499d405d538769a078f2794549955cf2409fcbcbf87f42e94: mount --make-rshared /var/run/netns failed: "operation not permitted" 
Error: failed to mount shm tmpfs "/var/lib/containers/storage/vfs-containers/66692b7ff496775499d405d538769a078f2794549955cf2409fcbcbf87f42e94/userdata/shm": operation not permitted

I've also tried a suggestion from the other question, passing --cgroup-manager=cgroupfs, but without success:

$ podman run -it my/test bash -c "podman --storage-driver=vfs --cgroup-manager=cgroupfs run docker.io/library/hello-world"
Trying to pull docker.io/library/hello-world...
Getting image source signatures
Copying blob 0e03bdcc26d7 done  
Copying config bf756fb1ae done  
Writing manifest to image destination
Storing signatures
ERRO[0003] unable to write pod event: "write unixgram @000f3->/run/systemd/journal/socket: sendmsg: no such file or directory" 
ERRO[0003] Error preparing container c3fff4d8161903aaebd6f89f3b3c06b55038e11e07b6b561dc6576ca675747a3: error creating network namespace for container c3fff4d8161903aaebd6f89f3b3c06b55038e11e07b6b561dc6576ca675747a3: mount --make-rshared /var/run/netns failed: "operation not permitted" 
Error: failed to mount shm tmpfs "/var/lib/containers/storage/vfs-containers/c3fff4d8161903aaebd6f89f3b3c06b55038e11e07b6b561dc6576ca675747a3/userdata/shm": operation not permitted

Seems like some network configuration is needed. I found the project below which suggests that some tweaking on network configurations might be necessary, but I don't know what would be the context of that and whether it would apply here or not. https://github.com/joshkunz/qemu-docker

EDIT: I've just discovered /var/run/podman.sock, but also without success:

$ sudo podman run -it -v /run/podman/podman.sock:/run/podman/podman.sock my/test bash -c "podman --storage-driver=vfs --cgroup-manager=cgroupfs run docker.io/library/hello-world"
Trying to pull my/test...
  denied: requested access to the resource is denied
Trying to pull my:test...
  unauthorized: access to the requested resource is not authorized
Error: unable to pull my/text: 2 errors occurred:
        * Error initializing source docker://my/test: Error reading manifest latest in docker.io/my/test: errors:
denied: requested access to the resource is denied
unauthorized: authentication required

        * Error initializing source docker://quay.io/my/test:latest: Error reading manifest latest in quay.io/my/test: unauthorized: access to the requested resource is not authorized

Seems like root cannot see the images I've created under my user.

Any ideas? Thanks.

Solution

Assume we would like to run ls / in a docker.io/library/alpine container.

Standard Podman

podman run --rm docker.io/library/alpine ls /

Podman in Podman

Let's run ls / in a docker.io/library/alpine container, but this time we run podman in a quay.io/podman/stable container.

Update June 2021

A GitHub issue comment shows an example of how to run Podman in Podman as a non-root user both on the host and in the outer container. Slightly modified it would look like this:

podman \
  run \
    --rm \
    --security-opt label=disable \
    --user podman \
    quay.io/podman/stable \
      podman \
        run \
          --rm \
          docker.io/library/alpine \
            ls / 

Here is a full example:

$ podman --version
podman version 3.2.1
$ cat /etc/fedora-release 
Fedora release 34 (Thirty Four)
$ uname -r
5.12.11-300.fc34.x86_64
$ podman \
  run \
    --rm \
    --security-opt label=disable \
    --user podman \
    quay.io/podman/stable \
      podman \
        run \
          --rm \
          docker.io/library/alpine \
            ls / 
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob sha256:5843afab387455b37944e709ee8c78d7520df80f8d01cf7f861aae63beeddb6b
Copying config sha256:d4ff818577bc193b309b355b02ebc9220427090057b54a59e73b79bdfe139b83
Writing manifest to image destination
Storing signatures
bin
dev
etc
home
lib
media
mnt
opt
proc
root
run
sbin
srv
sys
tmp
usr
var
$ 

To avoid repeatedly downloading the inner container image, create a volume

podman volume create mystorage

and add the command-line option -v mystorage:/home/podman/.local/share/containers:rw to the outer Podman command. In other words

podman \
  run \
    -v mystorage:/home/podman/.local/share/containers:rw \
    --rm \
    --security-opt label=disable \
    --user podman \
    quay.io/podman/stable \
      podman \
        run \
          --rm \
          docker.io/library/alpine \
            ls / 

Podman in Podman (outdated answer)

(The old outdated answer from Dec 2020. I'll probably remove this when it's clear that the method described here is outdated)

Let's run ls / in a docker.io/library/alpine container, but this time we run podman in a quay.io/podman/stable container.

The command will look like this:

podman \
  run \
    --privileged \
    --rm \
    --ulimit host \
    -v /dev/fuse:/dev/fuse:rw \
    -v ./mycontainers:/var/lib/containers:rw \
    quay.io/podman/stable \
      podman \
        run \
          --rm \
          --user 0 \
          docker.io/library/alpine ls 

(The directory ./mycontainers is here used for container storage)

Here is a full example

$ podman --version
podman version 2.1.1
$ mkdir mycontainers
$ podman run --privileged --rm --ulimit host -v /dev/fuse:/dev/fuse:rw -v ./mycontainers:/var/lib/containers:rw   quay.io/podman/stable podman run --rm --user 0 docker.io/library/alpine ls | head -5
Trying to pull docker.io/library/alpine...
Getting image source signatures
Copying blob sha256:188c0c94c7c576fff0792aca7ec73d67a2f7f4cb3a6e53a84559337260b36964
Copying config sha256:d6e46aa2470df1d32034c6707c8041158b652f38d2a9ae3d7ad7e7532d22ebe0
Writing manifest to image destination
Storing signatures
bin
dev
etc
home
lib
$ podman run --privileged --rm --ulimit host -v /dev/fuse:/dev/fuse:rw -v ./mycontainers:/var/lib/containers:rw  quay.io/podman/stable podman images
REPOSITORY                TAG     IMAGE ID      CREATED     SIZE
docker.io/library/alpine  latest  d6e46aa2470d  4 days ago  5.85 MB

If you would leave out -v ./mycontainers:/var/lib/containers:rw you might see the slightly confusing error message

Error: executable file `ls` not found in $PATH: No such file or directory: OCI runtime command not found error

References:

• How to use Podman inside of a container Red Hat blog post from July 2021.

• discussion.fedoraproject.org (discussion about not found in $PATH)

• github comment (that gives advice about the correct way to run Podman in Podman)

Answered By - Erik Sjölund