Migrating from Docker Desktop to colima

Time has come to migrate away from Docker Desktop on my Mac work laptop. Rancher Desktop is one of the usual suggestions, but since it uses a tool called colima underneath, and I really never used the Docker Desktop GUI, might as well go full-on command line. Let’s get started!

What is colima?

colima is a nice little project which facilitates installing a container runtime on macOS (and Linux). On macOS Docker is never fully native, the CLI tool connects to a Docker daemon running on a tiny Linux VM.

colima uses lima under the hood to accomplish this. It automates the creation of a small Alpine Linux based VM to run Docker on.

Getting everything installed

Assuming you already uninstalled Docker Desktop, let’s go ahead and install colima and the Docker CLI tools:

brew install colima docker

Since I will need them later, let’s also install Docker Compose and Buildx:

brew install docker-buildx docker-compose
mkdir -p ~/.docker/cli-plugins
ln -sfn /opt/homebrew/opt/docker-buildx/bin/docker-buildx ~/.docker/cli-plugins/docker-buildx
ln -sfn /opt/homebrew/opt/docker-compose/bin/docker-compose ~/.docker/cli-plugins/docker-compose

Once we have everything installed we can get colima started and test if Docker is running:

colima start
docker run --rm hello-world

You should get something like this:

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
70f5ac315c5a: Pull complete
Digest: sha256:dcba6daec718f547568c562956fa47e1b03673dd010fe6ee58ca806767031d1c
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (arm64v8)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

Assigning more resources to the colima VM

The default VM colima creates has 2 vCPUs, 2GBs of RAM and 60GBs of disk storage, let’s customize that:

colima stop
colima start --cpu 4 --memory 8 --disk 60

That’s it, now we have colima running with 4 vCPUs and 8GBs of RAM, nice!

Problem 1: permission errors when running chown

Here I feel like I need to do a quick aside. Docker is not part of my everyday development workflow when working on Jitsi, but I do maintain our Docker setup. Our Docker setup is reasonably involved, and we do a couple of non-mainstream things here and there, such as fiddling with permissions in files / folders inside mounted volumes.

The first problem I found was issue #83, I was getting permission errors when starting my containers. Thankfully, the answer was right there in the comments, we need to change the mount type to 9p (the default is sshfs) and add an override file.

First make sure you stop and delete the colima VM:

colima stop
colima delete

Add this content into a file in `/Users/<username>/.lima/_config/override.yaml`:

mountType: 9p
mounts:
  - location: "/Users/<username>"
    writable: true
    9p:
      securityModel: mapped-xattr
      cache: mmap
  - location: "~"
    writable: true
    9p:
      securityModel: mapped-xattr
      cache: mmap
  - location: /tmp/colima
    writable: true
    9p:
      securityModel: mapped-xattr
      cache: mmap

Start colima again, with the 9p mount type (remember to still set the previous CPU and RAM options!):

colima start --cpu 4 --memory 8 --disk 60 --mount-type 9p

No more permission errors, nice!

Problem 2: UDP forwarding to the host does not work

UDP support in the Docker ecosystem has always felt a bit like an afterthought and this time it took me a bit to figure it out.

With colima all the (TCP) ports used by your containers are forwarded to the host machine, so you can access them comfortably in localhost. That doesn’t work for UDP though. The reason is that lima uses SSH to forward the ports, which doesn’t support UDP. Here is the lima issue.

In order to get Jitsi Meet working on macOS we need to configure the JVB_ADVERTISED_IPS environment variable, which must point to the public (or LAN since it’s for testing) IP of the host machine running the JVB container. A way around this is to tell colima to assign an IP address to the VM, and then use that:

colima stop
colima start --cpu 4 --memory 8 --disk 60 --mount-type 9p --network-address

We can now check with colima status:

INFO[0000] colima is running using QEMU
INFO[0000] arch: aarch64
INFO[0000] runtime: docker
INFO[0000] mountType: 9p
INFO[0000] address: 192.168.106.2
INFO[0000] socket: unix:///Users/saghul/.colima/default/docker.sock

The IP address we can use is 192.168.106.2 and won’t change, so that’s handy. Another potential solution would be to configure bridging and have the VM get a “real IP”, but honestly this Just Works (TM) so I didn’t bother.

Problem 3: docker login

When attempting to use docker login I stumbled upon this error:

Error saving credentials: error storing credentials

This looked very similar to issue #167, likely caused by the fact I had Docker Desktop installed before.

Edit ~/.docker/config.json and remove the "credsStore" line, that will do!

Problem 4: Buildx woes

Buildx is the new way to build Docker images using BuildKit. It allows one to build multi-architecture images easily. Since we release amd64 and arm64 images for Jitsi Meet, Buildx is a tool I’d like to use, but ran into colima issue #792, all amd64 images were producing an error at build time.

The problem seems to be some cgroups v2 shenanigans with the default Alpime image colima uses, but thankfully there is a workaround, creating a custom Buildx builder with the rootless image:

docker buildx create --name mybuilder --driver-opt 'image=moby/buildkit:rootless' --bootstrap --use

That did it! Now we can build multi-arch images with no problem.

Conclusion

While it wasn’t a walk in the park, replacing Docker Desktop with colima forced me to look under the hood and understand how a number of the moving parts work. Overall I’m happy I made the switch!