Time has come to migrate away from Docker Desktop on my Mac work laptop. Rancher Desktop is one of the usual suggestions, but since it uses a tool called colima underneath, and I really never used the Docker Desktop GUI, might as well go full-on command line. Let’s get started!
What is colima?
colima is a nice little project which facilitates installing a container runtime on macOS (and Linux). On macOS Docker is never fully native, the CLI tool connects to a Docker daemon running on a tiny Linux VM.
colima uses lima under the hood to accomplish this. It automates the creation of a small Alpine Linux based VM to run Docker on.
Getting everything installed
Assuming you already uninstalled Docker Desktop, let’s go ahead and install colima and the Docker CLI tools:
brew install colima docker
Since I will need them later, let’s also install Docker Compose and Buildx:
brew install docker-buildx docker-compose mkdir -p ~/.docker/cli-plugins ln -sfn /opt/homebrew/opt/docker-buildx/bin/docker-buildx ~/.docker/cli-plugins/docker-buildx ln -sfn /opt/homebrew/opt/docker-compose/bin/docker-compose ~/.docker/cli-plugins/docker-compose
Once we have everything installed we can get colima started and test if Docker is running:
colima start docker run --rm hello-world
You should get something like this:
Unable to find image 'hello-world:latest' locally latest: Pulling from library/hello-world 70f5ac315c5a: Pull complete Digest: sha256:dcba6daec718f547568c562956fa47e1b03673dd010fe6ee58ca806767031d1c Status: Downloaded newer image for hello-world:latest Hello from Docker! This message shows that your installation appears to be working correctly. To generate this message, Docker took the following steps: 1. The Docker client contacted the Docker daemon. 2. The Docker daemon pulled the "hello-world" image from the Docker Hub. (arm64v8) 3. The Docker daemon created a new container from that image which runs the executable that produces the output you are currently reading. 4. The Docker daemon streamed that output to the Docker client, which sent it to your terminal. To try something more ambitious, you can run an Ubuntu container with: $ docker run -it ubuntu bash Share images, automate workflows, and more with a free Docker ID: https://hub.docker.com/ For more examples and ideas, visit: https://docs.docker.com/get-started/
Assigning more resources to the colima VM
The default VM colima creates has 2 vCPUs, 2GBs of RAM and 60GBs of disk storage, let’s customize that:
colima stop colima start --cpu 4 --memory 8 --disk 60
That’s it, now we have colima running with 4 vCPUs and 8GBs of RAM, nice!
Problem 1: permission errors when running chown
Here I feel like I need to do a quick aside. Docker is not part of my everyday development workflow when working on Jitsi, but I do maintain our Docker setup. Our Docker setup is reasonably involved, and we do a couple of non-mainstream things here and there, such as fiddling with permissions in files / folders inside mounted volumes.
The first problem I found was issue #83, I was getting permission errors when starting my containers. Thankfully, the answer was right there in the comments, we need to change the mount type to
9p (the default is
sshfs) and add an override file.
First make sure you stop and delete the colima VM:
colima stop colima delete
Add this content into a file in `
mountType: 9p mounts: - location: "/Users/<username>" writable: true 9p: securityModel: mapped-xattr cache: mmap - location: "~" writable: true 9p: securityModel: mapped-xattr cache: mmap - location: /tmp/colima writable: true 9p: securityModel: mapped-xattr cache: mmap
Start colima again, with the
9p mount type (remember to still set the previous CPU and RAM options!):
colima start --cpu 4 --memory 8 --disk 60 --mount-type 9p
No more permission errors, nice!
Problem 2: UDP forwarding to the host does not work
UDP support in the Docker ecosystem has always felt a bit like an afterthought and this time it took me a bit to figure it out.
With colima all the (TCP) ports used by your containers are forwarded to the host machine, so you can access them comfortably in
localhost. That doesn’t work for UDP though. The reason is that lima uses SSH to forward the ports, which doesn’t support UDP. Here is the lima issue.
In order to get Jitsi Meet working on macOS we need to configure the
JVB_ADVERTISED_IPS environment variable, which must point to the public (or LAN since it’s for testing) IP of the host machine running the JVB container. A way around this is to tell colima to assign an IP address to the VM, and then use that:
colima stop colima start --cpu 4 --memory 8 --disk 60 --mount-type 9p --network-address
We can now check with
INFO colima is running using QEMU INFO arch: aarch64 INFO runtime: docker INFO mountType: 9p INFO address: 192.168.106.2 INFO socket: unix:///Users/saghul/.colima/default/docker.sock
The IP address we can use is
192.168.106.2 and won’t change, so that’s handy. Another potential solution would be to configure bridging and have the VM get a “real IP”, but honestly this Just Works (TM) so I didn’t bother.
Problem 3: docker login
When attempting to use
docker login I stumbled upon this error:
Error saving credentials: error storing credentials
This looked very similar to issue #167, likely caused by the fact I had Docker Desktop installed before.
~/.docker/config.json and remove the
"credsStore" line, that will do!
Problem 4: Buildx woes
Buildx is the new way to build Docker images using BuildKit. It allows one to build multi-architecture images easily. Since we release amd64 and arm64 images for Jitsi Meet, Buildx is a tool I’d like to use, but ran into colima issue #792, all amd64 images were producing an error at build time.
The problem seems to be some cgroups v2 shenanigans with the default Alpime image colima uses, but thankfully there is a workaround, creating a custom Buildx builder with the
docker buildx create --name mybuilder --driver-opt 'image=moby/buildkit:rootless' --bootstrap --use
That did it! Now we can build multi-arch images with no problem.
While it wasn’t a walk in the park, replacing Docker Desktop with colima forced me to look under the hood and understand how a number of the moving parts work. Overall I’m happy I made the switch!