Ollama on Fedora Silverblue

I found myself dealing with various rough edges and questions around running Ollama on Fedora Silverblue for the past few months. These arise from the fact that there are a few different ways of installing Ollama, /usr is a read-only mount point on Silverblue, people have different kinds of GPUs or none at all, the program that’s using Ollama might be a graphical application in a Flatpak or part of the operating system image, and so on. So, I thought I’ll document a few different use-cases in one place for future reference or maybe someone will find it useful.

Different ways of installing Ollama

There are at least three different ways of installing Ollama on Fedora Silverblue. Each of those have their own nuances and trade-offs that we will explore later.

First, there’s the popular single command POSIX shell script installer:

$ curl -fsSL https://ollama.com/install.sh | sh

There is a manual step by step variant for those who are uncomfortable with running a script straight off the Internet. They both install Ollama in the operating system’s /usr/local or /usr or / prefix, depending on which one comes first in the PATH environment variable, and attempts to enable and activate a systemd service unit that runs ollama serve.

Second, there’s a docker.io/ollama/ollama OCI image that can be used to put Ollama in a container. The container runs ollama serve by default.

Finally, there’s Fedora’s ollama RPM.

Surprise

Astute readers might be wondering why I mentioned the shell script installer in the context of Fedora Silverblue, because /usr is a read-only mount point. Won’t it break the script? Not really, or the script breaks but not in the way one might expect.

Even though, /usr is read-only on Silverblue, /usr/local is not, because it’s a symbolic link to /var/usrlocal, and Fedora defaults to putting /usr/local/bin earlier in the PATH environment variable than the other prefixes that the installer attempts to use, as long as pkexec(1) isn’t being used. This happy coincidence allows the installer to place the Ollama binaries in their right places.

The script does fail eventually when attempting to create the systemd service unit to run ollama serve, because it tries to create an ollama user with /usr/share/ollama as its home directory. However, this half-baked installation works surprisingly well as long as nobody is trying to use an AMD GPU.

NVIDIA GPUs work, if the proprietary driver and nvidia-smi(1) are present in the operating system, which are provided by the kmod-nvidia and xorg-x11-drv-nvidia-cuda packages from RPM Fusion; and so does CPU fallback.

Unfortunately, the results would be the same if the shell script installer is used inside a Toolbx container. It will fail to create the systemd service unit because it can’t connect to the system-wide instance of systemd.

Using AMD GPUs with Ollama is an important use-case. So, let’s see if we can do better than trying to manually work around the hurdles faced by the script.

OCI image

The docker.io/ollama/ollama OCI image requires the user to know what processing hardware they have or want to use. To use it only with the CPU without any GPU acceleration:

$ podman run \
    --name ollama \
    --publish 11434:11434 \
    --rm \
    --security-opt label=disable \
    --volume ~/.ollama:/root/.ollama \
    docker.io/ollama/ollama:latest

This will be used as the baseline to enable different kinds of GPUs. Port 11434 is the default port on which the Ollama server listens, and ~/.ollama is the default directory where it stores its SSH keys and artificial intelligence models.

To enable NVIDIA GPUs, the proprietary driver and nvidia-smi(1) must be present on the host operating system, as provided by the kmod-nvidia and xorg-x11-drv-nvidia-cuda packages from RPM Fusion. The user space driver has to be injected into the container from the host using NVIDIA Container Toolkit, provided by the nvidia-container-toolkit package from Fedora, for Ollama to be able to use the GPUs.

The first step is to generate a Container Device Interface (or CDI) specification for the user space driver:

$ sudo nvidia-ctk cdi generate --output /etc/cdi/nvidia.yaml
…
…

Then the container needs to be run with access to the GPUs, by adding the --gpus option to the baseline command above:

$ podman run \
    --gpus all \
    --name ollama \
    --publish 11434:11434 \
    --rm \
    --security-opt label=disable \
    --volume ~/.ollama:/root/.ollama \
    docker.io/ollama/ollama:latest

AMD GPUs don’t need the driver to be injected into the container from the host, because it can be bundled with the OCI image. Therefore, instead of generating a CDI specification for them, an image that bundles the driver must be used. This is done by using the rocm tag for the docker.io/ollama/ollama image.

Then container needs to be run with access to the GPUs. However, the --gpus option only works for NVIDIA GPUs. So, the specific devices need to be spelled out by adding the --devices option to the baseline command above:

$ podman run \
    --device /dev/dri \
    --device /dev/kfd \
    --name ollama \
    --publish 11434:11434 \
    --rm \
    --security-opt label=disable \
    --volume ~/.ollama:/root/.ollama \
    docker.io/ollama/ollama:rocm

However, because of how AMD GPUs are programmed with ROCm, it’s possible that some decent GPUs might not be supported by the docker.io/ollama/ollama:rocm image. The ROCm compiler needs to explicitly support the GPU in question, and Ollama needs to be built with such a compiler. Unfortunately, the binaries in the image leave out support for some GPUs that would otherwise work. For example, my AMD Radeon RX 6700 XT isn’t supported.

This can be verified with nvtop(1) in a Toolbx container. If there’s no spike in the GPU and its memory then its not being used.

It will be good to support as many AMD GPUs as possible with Ollama. So, let’s see if we can do better.

Fedora’s ollama RPM

Fedora offers a very capable ollama RPM, as far as AMD GPUs are concerned, because Fedora’s ROCm stack supports a lot more GPUs than other builds out there. It’s possible to check if a GPU is supported either by using the RPM and keeping an eye on nvtop(1), or by comparing the name of the GPU shown by rocminfo with those listed in the rocm-rpm-macros RPM.

For example, according to rocminfo, the name for my AMD Radeon RX 6700 XT is gfx1031, which is listed in rocm-rpm-macros:

$ rocminfo
ROCk module is loaded
=====================    
HSA System Attributes    
=====================    
Runtime Version:         1.1
Runtime Ext Version:     1.6
System Timestamp Freq.:  1000.000000MHz
Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
Machine Model:           LARGE                              
System Endianness:       LITTLE                             
Mwaitx:                  DISABLED
DMAbuf Support:          YES

==========               
HSA Agents               
==========               
*******                  
Agent 1                  
*******                  
  Name:                    AMD Ryzen 7 5800X 8-Core Processor 
  Uuid:                    CPU-XX                             
  Marketing Name:          AMD Ryzen 7 5800X 8-Core Processor 
  Vendor Name:             CPU                                
  Feature:                 None specified                     
  Profile:                 FULL_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        0(0x0)                             
  Queue Min Size:          0(0x0)                             
  Queue Max Size:          0(0x0)                             
  Queue Type:              MULTI                              
  Node:                    0                                  
  Device Type:             CPU                                
…
…
*******                  
Agent 2                  
*******                  
  Name:                    gfx1031                            
  Uuid:                    GPU-XX                             
  Marketing Name:          AMD Radeon RX 6700 XT              
  Vendor Name:             AMD                                
  Feature:                 KERNEL_DISPATCH                    
  Profile:                 BASE_PROFILE                       
  Float Round Mode:        NEAR                               
  Max Queue Number:        128(0x80)                          
  Queue Min Size:          64(0x40)                           
  Queue Max Size:          131072(0x20000)                    
  Queue Type:              MULTI                              
  Node:                    1                                  
  Device Type:             GPU
…
…

The ollama RPM can be installed inside a Toolbx container, or it can be layered on top of the base registry.fedoraproject.org/fedora image to replace the docker.io/ollama/ollama:rocm image:

FROM registry.fedoraproject.org/fedora:42
RUN dnf --assumeyes upgrade
RUN dnf --assumeyes install ollama
RUN dnf clean all
ENV OLLAMA_HOST=0.0.0.0:11434
EXPOSE 11434
ENTRYPOINT ["/usr/bin/ollama"]
CMD ["serve"]

Unfortunately, for obvious reasons, Fedora’s ollama RPM doesn’t support NVIDIA GPUs.

Conclusion

From the puristic perspective of not touching the operating system’s OSTree image, and being able to easily remove or upgrade Ollama, using an OCI container is the best option for using Ollama on Fedora Silverblue. Tools like Podman offer a suite of features to manage OCI containers and images that are far beyond what the POSIX shell script installer can hope to offer.

It seems that the realities of GPUs from AMD and NVIDIA prevent the use of the same OCI image, if we want to maximize our hardware support, and force the use of slightly different Podman commands and associated set-up. We have to create our own image using Fedora’s ollama RPM for AMD, and the docker.io/ollama/ollama:latest image with NVIDIA Container Toolkit for NVIDIA.

Toolbx — about version numbers

Those of you who follow the Toolbx project might have noticed something odd about our latest release that came out a month ago. The version number looked shorter than usual even though it only had relatively conservative and urgent bug-fixes, and no new enhancements.

If you were wondering about this, then, yes, you are right. Toolbx will continue to use these shorter version numbers from now on.

The following is a brief history of how the Toolbx version numbers evolved over time since the beginning of the project till this present moment.

Toolbx started out with a MAJOR.MINOR.MICRO versioning scheme. eg., 0.0.1, 0.0.2, etc.. Back then, the project was known as fedora-toolbox, was implemented in POSIX shell, and this versioning scheme was meant to indicate the nascent nature of the project and the ideas behind it.

To put it mildly, I had absolutely no idea what I was doing. I was so unsure that for several weeks or few months before the first Git commit in August 2018, it was literally a single file that implemented the fedora-toolbox(1) executable and a Dockerfile for the fedora-toolbox image on my laptop that I would email around to those who were interested.

A nano version was reserved for releases to address brown paper bag bugs or other critical issues, and for release candidates. eg., several releases between 0.0.98 and 0.1.0 used it to act as an extended set of release candidates for the dot-zero 0.1.0 release. More on that later.

After two years, in version 0.0.90, Toolbx switched from the POSIX shell implementation to a Go implementation authored by Ondřej Míchal. The idea was to do a few more 0.0.9x releases to shake out as many bugs in the new code as possible, implement some of the bigger items on our list that had gotten ignored due to the Go rewrite, and follow it up with a dot-zero 0.1.0 release. That was in May 2020.

Things went according to plan until the beginning of 2021, when a combination of factors put a spanner in the works, and it became difficult to freeze development and roll out the dot-zero release. It was partly because we kept getting an endless stream of bugs and feature requests that had to be addressed; partly because real life and shifting priorities got in the way for the primary maintainers of the project; and partly because I was too tied to the sanctity of the first dot-zero release. This is how we ended up doing the extended set of release candidates with a nano version that I mentioned above.

Eventually, version 0.1.0 arrived in October 2024, and since then we have had three more releases — 0.1.1, 0.1.2 and 0.2. Today, the Toolbx project is seven years old, and some things have changed enough that it requires an update to the versioning scheme.

First, both Toolbx and the ideas that it implements are a lot more mature and widely adopted than they were at the beginning. So much so, that there are a few independent reimplementations of it. It’s time for the project to stop hiding behind a micro version.

Second, the practice of bundling and statically linking the Go dependencies sometimes makes it necessary to update the dependencies to address security bugs or other critical issues. It’s more convenient to do this as part of an upstream release than through downstream patches by distributors. So far, we have managed to avoid the need to do minimal releases targeting only specific issues for conservative downstream distributors, but the recent NVIDIAScape or CVE-2025-23266 and CVE-2025-23267 in the NVIDIA Container Toolkit gave me pause. We managed to escape this time too, but it’s clear that we need a plan to deal with these scenarios.

Hence, from now on, Toolbx releases will default to not having a micro version and use a MAJOR.MINOR versioning scheme. A micro version will be reserved for the same purposes that a nano version was reserved for until now — to address critical issues and for release candidates.

It’s easier to read and remember a shorter MAJOR.MINOR version than a longer one, and appropriately conveys the maturity of the project. When a micro version is needed, it will also be easier to read and remember than a longer one with a nano version. Being easy to read and remember is important for version numbers, because it separates them from Git commit hashes.

So, this is why the latest release is 0.2, not 0.1.3.

Toolbx containers hit by CA certificates breakage

If you are using Toolbx 0.1.2 or newer, then some existing containers have been hit by a bug that breaks certificates from certificate authorities (or CAs) that are used for secure network communication. The bug prevents OpenSSL and other cryptographic components from finding any certificates, which means that programs like pacman and DNF that use OpenSSL are unable to download any metadata or packages. For what it’s worth GnuTLS and, possibly, Network Security Services (or NSS) are unaffected.

This is a serious breakage because OpenSSL is widely used, and when it breaks the standard mechanisms for shipping updates, it can be difficult for users to find a way forward. So, without going into too many details, here’s how to diagnose if you are in this situation and how to repair your containers.

Diagnosis

Among the different operating system distributions that we regularly test Toolbx on, so far it seems that Arch Linux and Fedora containers are affected. However, if you suddenly start experiencing network errors related to CA certificates inside your Toolbx containers, then you might be affected too.

Within Arch Linux containers, you will see that files like /etc/ca-certificates/extracted/ca-bundle.trust.crt, /etc/ca-certificates/extracted/edk2-cacerts.bin, /etc/ca-certificates/extracted/tls-ca-bundle.pem, etc. have been emptied out and /etc/ca-certificates/extracted/java-cacerts.jks looks suspiciously truncated at 32 bytes.

Within Fedora containers, you will see the same, but the file paths are slightly different. They are /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt, /etc/pki/ca-trust/extracted/edk2/cacerts.bin, /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem, /etc/pki/ca-trust/extracted/java/cacerts, etc..

Workaround

First, you need to disable something called remote p11-kit inside the containers.

This requires getting rid of /etc/pkcs11/modules/p11-kit-trust.module from the containers:

⬢ $ sudo rm /etc/pkcs11/modules/p11-kit-trust.module

… and the p11-kit-client.so PKCS #11 module.

On Arch Linux:

⬢ $ sudo rm /usr/lib/pkcs11/p11-kit-client.so

On Fedora 43 onwards, it’s provided by the p11-kit-client RPM, and on older releases it’s the p11-kit-server RPM:

⬢ $ sudo dnf remove p11-kit-client
Package                 Arch   Version                 Repository           Size
Removing:
 p11-kit-client         x86_64 0.25.5-9.fc43           ab42c14511ba47b   1.2 MiB

Transaction Summary:
 Removing:           1 package

After this operation, 1 MiB will be freed (install 0 B, remove 1 MiB).
Is this ok [y/N]: y
Running transaction
[1/2] Prepare transaction                                                                                                                                                   100% |  66.0   B/s |   1.0   B |  00m00s
[2/2] Removing p11-kit-client-0:0.25.5-9.fc43.x86_64                                                                                                                        100% |  16.0   B/s |   5.0   B |  00m00s
>>> Running %triggerpostun scriptlet: systemd-0:257.7-1.fc43.x86_64                                                                                                                                                 
>>> Finished %triggerpostun scriptlet: systemd-0:257.7-1.fc43.x86_64                                                                                                                                                
>>> Scriptlet output:                                                                                                                                                                                               
>>> Failed to connect to system scope bus via local transport: No such file or directory                                                                                                                            
>>>                                                                                                                                                                                                                 
>>> Running %triggerpostun scriptlet: systemd-0:257.7-1.fc43.x86_64                                                                                                                                                 
>>> Finished %triggerpostun scriptlet: systemd-0:257.7-1.fc43.x86_64                                                                                                                                                
>>> Scriptlet output:                                                                                                                                                                                               
>>> Failed to connect to system scope bus via local transport: No such file or directory                                                                                                                            
>>>                                                                                                                                                                                                                 
Complete!

Then, you need to restart the container, run update-ca-trust(8), and that’s all:

⬢ $
logout
$ podman stop <CONTAINER>
<CONTAINER>
$ toolbox enter <CONTAINER>
⬢ $ sudo update-ca-trust

Toolbx now enables the proprietary NVIDIA driver

… and why did it take so long for that to happen?

If you build Toolbx from Git and install it to your usual prefix on a host operating system with the proprietary NVIDIA driver, then you will be able to use the driver on all freshly started Toolbx containers. Just like that. There’s no need to recreate your containers or to use some special option. It will just work.

How does it work?

Toolbx uses the NVIDIA Container Toolkit to generate a Container Device Interface specification on the host during the toolbox enter and toolbox run commands. This is a JSON or YAML description of the environment variables, files and other miscellaneous things required by the user space part of the proprietary NVIDIA driver. Containers share the kernel space driver with the host, so we don’t have to worry about that. This specification is then shared with the Toolbx container’s entry point, which is the toolbox init-container command running inside the container. The entry point handles the hooks and bind mounts, while the environment variables are handled by the podman exec process running on the host.

It’s worth pointing out that right now this neither uses nvidia-ctk cdi generate to generate the Container Device Interface specification nor podman create --device to consume it. We may decide to change this in the future, but right now this is the way it is.

The main problem with podman create is that the specification must be saved in /etc/cdi or /var/run/cdi, both of which require root access, for it to be visible to podman create --device. Toolbx containers are often used rootless, so requiring root privileges for hardware support, something that’s not necessary on the host, will be a problem.

Secondly, updating the toolbox(1) binary won’t enable the proprietary NVIDIA driver in existing containers, because podman create only affects new containers.

Therefore, Toolbx uses the tags.cncf.io/container-device-interface Go APIs, which are also used by podman create, to parse and apply the specification itself. The hooks in the specification are a bit awkward to deal with. So, at the moment only ldconfig(8) is supported.

The issue with nvidia-ctk is relatively minor and is because it’s another different binary. It makes error handling more difficult, and downstream distributors and users of Toolbx need to be aware of the dependency. Instead, it’s better to directly use the github.com/NVIDIA/go-nvlib and github.com/NVIDIA/nvidia-container-toolkit Go APIs that nvidia-ctk also uses. This offers all the usual error handling facilities in Go and ensures that the dependency won’t go missing.

Why did it take so long?

Well, hardware support needs hardware, and sometimes it takes time to get access to it. I didn’t want to optimistically throw together a soup of find(1), grep(1), sed(1), etc. calls without any testing in the hope that it will all work out. That approach may be fine for some projects, but not for Toolbx.

Red Hat recently got me a ThinkPad P72 laptop with a NVIDIA Quadro P600 GPU that let me proceed with this work.

Toolbx is a release blocker for Fedora 39 onwards

This is the second instalment of my 2023 retrospective series on Toolbx. ¹

One very important thing that we did behind the scenes was to make Toolbx a release blocker for Fedora 39 and onwards. This means that the registry.fedoraproject.org/fedora-toolbox OCI image is considered a release-blocking deliverable, and there are release-blocking test criteria to ensure that the toolbox RPM is usable.

Why do that?

Earlier, there was no formal requirement for Toolbx to be usable when a new Fedora was released. That was a problem for a tool that’s so popular and provides something as fundamental as an interactive command line environment for software development and troubleshooting the host operating system. Everybody expects their CLI environment to just work even under very adverse conditions, and Toolbx should be no different. Except that Toolbx is slightly more complicated than running Bash or Z shell directly on the host OS, and, therefore, requires a bit more diligence.

Toolbx has two parts — an OCI image, which defaults to registry.fedoraproject.org/fedora-toolbox on Fedora hosts, and the toolbox RPM. The OCI image is pulled by the RPM to set up a containerized interactive CLI environment.

Let’s look at each separately.

The image

First, we wanted to ensure that there is an up to date fedora-toolbox OCI image published on registry.fedoraproject.org as a release-blocking deliverable at critical points in the development schedule, just like the installation ISOs for the Editions from download.fedoraproject.org. For example, when an upcoming Fedora release is branched from Rawhide, and for the Beta and Final releases.

One of the recurring complaints that we used to get were from users of Fedora Rawhide Toolbx containers, when Rawhide gets branched in preparation for the Beta for the next Fedora release. At this point, the previous Rawhide version becomes the Branched version, and the current Rawhide version increases by one. If the fedora-toolbox images aren’t part of the mass branching performed by Fedora Release Engineering, then someone has to quickly step in after they have finished to refresh the images to ensure consistency. This sort of ad hoc manual co-ordination rarely works, and it left users in the lurch.

With this change, the fedora-toolbox image is part of the nightly Fedora composes, and the branching is handled by Fedora Release Engineering just like any other release-blocking deliverable. This makes the image as readily available and updated as the fedora and fedora-minimal OCI images or any other deliverable, and we hope that it will improve the user experience for Rawhide Toolbx containers.

If someone installs the Fedora Beta or the Final on their host, and creates a Toolbx container using the default image, then, barring exceptions, the host and the container now have the same RPM versions for all packages. Just like Fedora Silverblue and Workstation are released with the same versions. This ensures greater consistency in terms of bug-fixes, features and pending updates.

In the past, this wasn’t the case and it led to occasional surprises. For example, the change to make RPM use a Sequoia based OpenPGP parser made it impossible to install third party RPMs in the fedora-toolbox image, even long after the actual bug was fixed.

The RPM

Second, we wanted to have release-blocking test criteria to ensure that the toolbox RPM is usable at critical points in the development schedule. This is to ensure that changes in the Toolbx stack, and future changes in other parts of the operating system do not break Toolbx — at least not for the Beta and Final releases. It’s good to have the fedora-toolbox image be more readily available and updated, but it’s better if Toolbx works more reliably as a whole.

Examples of changes in the Toolbx stack causing breakage can be FUSE preventing RPMs with file capabilities from being installed inside Toolbx containers, Toolbx bind mounts preventing RPMs with %attr() from being installed or causing systemd-tmpfiles(8) to throw errors, etc.. Examples of changes in other parts of the OS can be changes to Fedora’s Kerberos stack causing Kerberos to stop working inside Toolbx containers, changes to the sysctl(8) configuration breaking ping(8), changes in Mutter breaking graphical applications, etc..

The test criteria for the toolbox RPM also implicitly tests the fedora-toolbox image, and co-ordinates several disparate groups of developers to ensure that the containerized interactive command line Toolbx environments on Fedora are just as reliable as those running directly on the host OS.

Tooling changes

This does come with a significant tooling change that isn’t obvious at first. The fedora-toolbox OCI image is no longer defined as a layered image through a Container/Dockerfile. Instead, it’s built as a base image through Kickstarts and Pungi, just like the fedora and fedora-minimal images.

This was necessary because the nightly Fedora composes work with Kickstarts and Pungi, not Container/Dockerfiles. Moreover, building Fedora OCI images from a Dockerfile with fedpkg container-build uses an ancient unmaintained version of OpenShift Build Service that requires equally unmaintained ancient versions of Fedora to run, and the fedora-toolbox image was the only thing using Container/Dockerfiles in Fedora.

We either had to update the Fedora infrastructure to use OpenShift Build Service 2.x; or use Kickstarts and Pungi, which uses Image Factory, to build the fedora-toolbox image. We chose the latter, because updating the infrastructure would be a significant effort, and by using Kickstarts and Pungi we get to stay close to the fedora and fedora-minimal images and simplify the infrastructure.

The Fedora Flatpaks were also being built using the same ancient and unmaintained version of OpenShift Build Service, and they too are in the process being migrated. However, that’s outside the scope of this post.

One big benefit of fedora-toolbox not being a layered image based on top of the fedora image is that it removes the constant fight against the efforts to minimize the size of the latter. The fedora-toolbox image is designed for interactive command line use in long-lived containers, and not for deploying server-side applications and services in ephemeral ones. This means that dictionaries, documentation, locales, iconv converter modules, translations, etc. are more important than reducing the size of the images. Now that the image is built from scratch, it has full control over what goes into it.

Unfortunately, Image Factory is weakly maintained and setting it up on one’s local machine is a lot more complicated than using podman build. One can do scratch builds on the Fedora infrastructure with koji image-build --scratch, but only if they have been explicitly granted permissions, and then they have to download the tarball and use skopeo copy to place them in containers-storage so that Podman can see it. All that is again more complicated than doing a podman build.

Due to this difficulty of untangling the image build from the Fedora infrastructure, we haven’t published the sources of the fedora-toolbox image for recent Fedora versions upstream. We do have a fedora-toolbox:39 image defined through a Container/Dockerfile, but that was done purely as a contingency during the Fedora 39 development cycle.

This does degrade the developer experience of working on the fedora-toolbox image, but, given all the other advantages, we think that it’s worth it.

As of this writing, there’s a Fedora 40 Change to switch to using KIWI to build the OCI images, including fedora-toolbox, instead of Image Factory. KIWI seems more strongly maintained and a lot easier to set up locally, which is fantastic. So, it should be all rainbows and unicorns, once we soldier through another port of the fedora-toolbox image to a different tooling and source language.

Acknowledgements

Last but not the least, getting all this done on time required a good deal of co-ordination and help from several different individuals. I must thank Sumantro for leading the effort; Kevin, Tomáš and Samyak for all the infrastructure and release engineering work; and Adam and Kamil for all the testing and validation.

1. Toolbx now offers built-in support for Arch Linux and Ubuntu ↩︎

Toolbx now offers built-in support for Arch Linux and Ubuntu

… and why did it take so long for that to happen?

The year 2023 was a busy one in Toolbx land. We had two big releases, and one of the important things we did was to start offering built-in support for Arch Linux and Ubuntu.

What does that mean?

It means that if you have the latest Toolbx release, then a simple toolbox create on Arch Linux and Ubuntu hosts will create matching Toolbx containers, instead of a fallback Fedora container. If you are using some other host operating system, then you can get the same with:

$ toolbox create --distro arch
$ toolbox create --distro ubuntu --release 22.04

It also means that we will do our best to treat Arch Linux and Ubuntu on equal terms with Fedora and avoid regressions.

Now, that last sentence has a lot more to it than may seem at first. So, I am going to try to unwrap it to show why I think this one of the important things we did in 2023. You may find it interesting if you want the set of supported Linux distributions to expand further. You can also skip the details and go straight to the end to get the jist of it.

Why go beyond Fedora?

Even though Toolbx was created for the immediate needs of Fedora Silverblue, it has long since exceeded the bounds of Fedora and OSTree based operating systems. At the same time, Toolbx only had built-in support for Fedora and Red Hat Enterprise Linux. This disconnect led to a lot of people eagerly asking for better support for Linux distributions beyond the Fedora family.

From the outside, it looks like just a matter of putting together a Containerfile that layers on some extra content on top of a base image, and then publishing it on some OCI registry somewhere. However, it’s not that simple.

Why did it take so long?

Toolbx is a glue. Given an existing host operating system and a desired command line environment, it will create and set up an OCI container with the desired CLI environment that has seamless access to the user’s home directory, the Wayland and X11 sockets, networking (including Avahi), removable devices (like USB sticks), systemd journal, SSH agent, D-Bus, ulimits, /dev and the udev database, etc..

The closest analogy I can think of is libosinfo generating scripts to perform automated installations of various operating systems as virtual machines. Except, Toolbx containers aren’t expected to be as cleanly or formally separated from the host as VMs are, and the world of OCI containers is designed for deploying server-side applications and services, not for the persistent interactive use that Toolbx offers. This means that Toolbx has to carefully make each host OS version match the container, and make the container fit for interactive use. For example, libosinfo doesn’t have to worry about making the VM’s command line shell prompt and /etc/resolv.conf work with that of the host’s, nor does it have to convert the cloud image of an OS into a variant suited for a desktop or laptop.

This means that Toolbx has to very carefully set up the container to work with the particular host operating system version in use. For example, there can be subtle differences between running a Fedora 39 container on a Fedora 38 host and vice versa. If there are three different Fedora versions that we care about (such as Fedoras 38, 39 and 40), then that’s nine different combinations of container and host OSes to support. Sometimes, the number of relevant versions goes from three to four, and the number of combinations jumps to sixteen. Now, add Red Hat Enterprise Linux to the mix. Assuming that we only care about the latest point-releases (such as RHELs 8.9 and 9.3), we are now looking at twenty-five to thirty-six combinations. In reality, it’s a lot more, because we care about more than just the latest RHEL point-releases.

You can see where this is going.

With this addition of Arch Linux and Ubuntu, we have added at least seven new versions that we care about. That’s a total of approximately one hundred and fifty combinations!

I can assure you that this isn’t a theoretical concern. Here’s a bug about domain name resolution being completely broken in Toolbx containers for Fedoras 39 and 40 on Red Hat Enterprise Linux 9 hosts.

Tests

So, to promise with a straight face that we will do our best to treat Arch Linux and Ubuntu on equal terms with Fedora and avoid regressions, we need to automate this. There’s no way a test matrix of this size can be tackled manually. Hence, tests.

However, it’s easier said than done, because it’s not just about having the tests. We also need to run them on as many different host operating system versions as possible. Unfortunately, most continuous integration systems out there either only offer containers, which are useless because Toolbx needs to be tested on the host OS, or they offer Ubuntu hosts. One exception is Software Factory, which runs an instance of Zuul CI, and offers Fedora and CentOS Stream hosts.

Currently, we have a test suite with more than three hundred tests that covers a good chunk of all supported OCI containers and images. All these tests are run separately on hosts representing all active Fedora versions, with subsets being run on CentOS Stream 9 and Ubuntu 22.04 hosts. We are working on ensuring that the entire test suite gets run on CentOS Stream 9 and Ubuntu 22.04, and are hoping to introduce CentOS Stream 8 and Ubuntu 24.04 hosts to the mix.

Plus, these aren’t just simple smoke tests. They don’t just create and start the containers, and check for a successful exit code. They comprehensively poke at various attributes of the containers’ runtime environment and error conditions to ensure that things really do work as advertised.

I think this is a pretty impressive set-up. It took time to build it, and it’s still not done, but I think it was worth it.

We have also been busy in Fedora, pushing the quality of the Toolbx stack up a notch, but that’s a topic for another post.

So, if you want to see built-in support for your favourite Linux distribution, then please help us by providing some test runners. Right now we could really use one for Arch Linux to maintain our existing support for it, and one for Debian because we want to include it in the near future.

Maintainers

Finally, since Toolbx is a glue, sometimes we need to drive changes into the Linux distributions that we claim to support. For example, changing the sysctl(8) configuration to make ping(8) work, fixes to the start-up scripts for Bash and Z shell, etc.. This means that we need maintainers who will own the work that’s specific to a particular Linux distribution. As a Fedora contributor, I can take care of Fedora, but I cannot sign up to take care of every single distribution that’s out there.

In that sense, I am delighted that we have a bunch of dedicated folks taking care of the Arch Linux and Ubuntu support. Namely, Morten Linderud for Arch Linux, and Andrej Shadura and Ievgen Popovych for Ubuntu.

Conclusion

Two things need to happen for us to add built-in support for a new Linux distribution.

First, we need test runners that let us run our upstream test suite on the host operating system, and not inside a container. Right now we could really use one for Arch Linux to maintain our existing support for it, and one for Debian because we want to include it in the near future.

Second, we need someone to step up to drive changes into the Linux distribution in question, and own the work that’s specific to it.

I am also going to add this to the Toolbx documentation, so that it’s easy to find in future.

Toolbx — running the same host binary on Arch Linux, Fedora, Ubuntu, etc. containers

This is a deep dive into some of the technical details of Toolbx and is a continuation from the earlier post about bypassing the immutability of OCI containers.

The problem

As we saw earlier, Toolbx uses a special entry point for its containers. It’s the toolbox executable itself.

$ podman inspect --format "{{.Config.Cmd}}" --type container fedora-toolbox-36
toolbox --log-level debug init-container ...

This is achieved by bind mounting the toolbox executable invoked by the user on the hosts to /usr/bin/toolbox inside the containers. While this has some advantages, it opens the door to one big problem. It means that executables from newer or different host operating systems might be running against older or different run-time environments inside the containers. For example, an executable from a Fedora 36 host might be running inside a Fedora 35 Toolbx, or one from an Arch Linux host inside an Ubuntu container.

This is very unusual. We only expect executables from an older version of an OS to keep working on newer versions of the same OS, but never the other way round, and definitely not across different OSes.

When binaries are compiled and linked against newer run-time environments, they may start relying on symbols (ie., non-static global variables, functions, class and struct members, etc.) that are missing in older environments. For example, glibc-2.32 (used in Fedora 33 onwards) added a new version of the pthread_sigmask symbol. If toolbox binaries built and linked against glibc-2.32 are run against older glibc versions, then they will refuse to start.

$ objdump -T /usr/bin/toolbox | grep GLIBC_2.32
0000000000000000      DO *UND*        0000000000000000  GLIBC_2.32  pthread_sigmask

This means that one couldn’t use Fedora 32 Toolbx containers on Fedora 33 hosts, or similarly any containers with glibc older than 2.32 on hosts with newer glibc versions. That’s quite the bummer.

If the executables are not ELF binaries, but carefully written POSIX shell scripts, then this problem goes away. Incidentally, Toolbx used to be implemented in POSIX shell, until it was re-written in Go two years ago, which is how it managed to avoid this problem for a while.

Fortunately, Go binaries are largely statically linked, with the notable exception of the standard C library. The scope of the problem would be much bigger if it involved several other dynamic libraries, like in the case of C or C++ programs.

Potential options

In theory, the easiest solution is to build the toolbox binary against the oldest supported run-time environment so that it doesn’t rely on newer symbols. However, it’s easier said than done.

Usually downstream distributors use build environments that are composed of components that are part of that specific version of the distribution. For example, it will be unusual for an RPM for a certain Fedora version to be deliberately built against a run-time from an older Fedora. Carlos O’Donell had an interesting idea on how to implement this in Fedora by only ever building for the oldest supported branch, adding a noautobuild file to disable the mass rebuild automation, and having newer branches always inherit the builds from the oldest one. However, this won’t work either. Building against the oldest supported Fedora won’t be enough for Fedora’s Toolbx because, by definition, Toolbx is meant to run different kinds of containers on hosts. The oldest supported Fedora hosts might still be too new compared to containers of supported Debian, Red Hat Enterprise Linux, Ubuntu etc. versions.

So, yes, in theory, this is the easiest solution, but, in practice, it requires a non-trivial amount of cross-distribution collaboration, and downstream build system and release engineering effort.

The second option is to have Toolbx containers provide their own toolbox binary that’s compatible with the run-time environment of the container. This would substantially complicate the communication between the toolbox binaries on the hosts and the ones inside the containers, because the binaries on the hosts and containers will no longer be exactly the same. The communication channel between commands like toolbox create and toolbox enter running on the hosts, and toolbox init-container inside the containers can no longer use a private and unstable interface that can be easily modified as necessary. Instead, it would have complicated backwards and forwards compatibility requirements. Other than that, it would complicate bug reports, and every single container on a host may need to be updated separately to fix bugs, with updates needing to be co-ordinated across downstream distributors.

The next option is to either statically link against the standard C library, or disable its use in Go. However, that would prevent us from using glibc’s Name Service Switch to look up usernames and groups, or to resolve host names. The replacement code, written in pure Go, can’t handle enterprise set-ups involving Network Information Service and Lightweight Directory Access Protocol, nor can it talk to host OS services like SSSD, systemd-userdbd or systemd-resolved.

It’s true that Toolbx currently doesn’t support enterprise set-ups with NIS and LDAP, but not using NSS will only make it more difficult to add that support in future. Similarly, we don’t resolve any host names at the moment, but given that we are in the business of pulling content over the network, it can easily become necessary in the future. Disabling the use of NSS will leave the toolbox binary as this odd thing that behaves differently from the rest of the OS for some fundamental operations.

An extension of the previous option is to split the toolbox executable into two. One dynamically linked against the standard C library for the hosts, and another that has no dynamic linkage to run inside the containers as their entry point. This can impact backwards compatibility and affect the developer experience of hacking on Toolbx.

Existing Toolbx containers want to bind mount the toolbox executable from the host to /usr/bin/toolbox inside the containers and run toolbox init-container as their entry point. This can’t be changed because of the immutability of OCI containers, and Toolbx simply can’t afford to break existing containers in a way where they can no longer be entered. This means that the toolbox executable needs to become a shim, without any dynamic linkage, that forwards the invocation to the right executable depending on whether it’s running on the hosts or inside the containers.

That brings us to the developer experience of hacking on Toolbx. The first thing note is that we don’t to go back to using POSIX shell to implement the executable that’s meant to run inside the container. Ondřej spent a lot of effort replacing the POSIX shell implementation of Toolbx, and we don’t want to undo any part of that. Ideally, we would use the same programming language (ie., Go) to implement both executables so that one doesn’t need to learn multiple disparate languages to work on Toolbx. However, even if we do use Go, we would have to be careful not to share code across the two executables, or be aware that they may have subtle differences in behaviour depending on how they might be linked.

Then there’s the developer experience of hacking on Toolbx on Fedora Silverblue and similar OSTree-based OSes, which is what you would do to eat your own dog food. Experiences are always subjective and this one is unique to hacking Toolbx inside a Toolbx. So let’s take a moment to understand the situation.

On OSTree-based OSes, Toolbx containers are used for development, and, generally speaking, it’s better to use container-specific locations invisible to the host as the development prefixes because the generated executables are specific to each container. Executables built on one container may not work on another, and not on the hosts either, because of the run-time problems mentioned above. Plus, it’s good hygiene not to pollute the hosts.

Similar to Flatpak and Podman, Toolbx is a tool that sets up containers. This means that unlike most other executables, toolbox must be on the hosts because, barring the init-container command, it can’t work inside the containers. The easiest way to do this, is to have a separate terminal emulator with a host shell, and invoke toolbox directly from Meson’s build directory in $HOME that’s shared between the hosts and the Toolbx containers, instead of installing toolbox to the container-specific development prefixes. Note that this only works because toolbox has always been implemented in programming languages with none to minimal dynamic linking, and only if you ensure that the Toolbx containers for hacking on Toolbx matches the hosts. Otherwise, you might run into the run-time problems mentioned above.

The moment there is one executable invoking another, the executables need to be carefully placed on the file system so that one can find the other one. This means that either the executables need to be installed into development prefixes or that the shim should have special logic to work out the location of the other binary when invoked directly from Meson’s build directory.

The former is a problem because the development prefixes will likely default to container-specific locations invisible from the hosts, preventing the built executables from being trivially invoked from the host. One could have a separate development prefix only for Toolbx that’s shared between the containers and the hosts. However, I suspect that a lot of existing and potential Toolbx contributors would find that irksome. They either don’t know or want to set up a prefix manually, but instead use something like jhbuild to do it for them.

The latter requires two different sets of logic depending on whether the shim was invoked directly from Meson’s build directory or from a development prefix. At the very least this would involve locating the second executable from the shim, but could grow into other areas as well. These separate code paths would be crucial enough that they would need to be thoroughly tested. Otherwise, Toolbx hackers and users won’t share the same reality. We could start by running our test suite in both modes, and then meticulously increase coverage, but that would come at the cost of a lengthier test suite.

Failed attempts

Since glibc uses symbol versioning, it’s sometimes possible to use some .symver hackery to avoid linking against newer symbols even when building against a newer glibc. This is what Toolbox used to do to ensure that binaries built against newer glibc versions still ran against older ones. However, this doesn’t defend against changes to the start-up code in glibc, like the one in glibc-2.34 that performed some security hardening.

Current solution

Alexander Larsson and Ray Strode pointed out that all non-ancient Toolbx containers have access to the hosts’ /usr at /run/host/usr. In other words, Toolbx containers have access to the host run-time environments. So, we decided to ensure that toolbox binaries always run against the host run-time environments.

The toolbox binary has a rpath pointing to the hosts’ libc.so somewhere under /run/host/usr and it’s dynamic linker (ie., PT_INTERP) is changed to the one inside /run/host/usr. Unfortunately, there can only be one PT_INTERP entry inside the binary, so there must be a /run/host on the hosts too for the binary to work on the hosts. Therefore, a /run/host symbolic link is also created on the host pointing to the hosts’ /.

The toolbox binary now looks like this, both on the hosts and inside the Toolbx containers:

$ ldd /usr/bin/toolbox
    linux-vdso.so.1 (0x00007ffea01f6000)
    libc.so.6 => /run/host/usr/lib64/libc.so.6 (0x00007f6bf1c00000)
    /run/host/usr/lib64/ld-linux-x86-64.so.2 => /lib64/ld-linux-x86-64.so.2 (0x00007f6bf289a000)

It’s been almost a year and thus far this approach has held its own. I am mildly bothered by the presence of the /run/host symbolic link on the hosts, but not enough to lose sleep over it.

Other options

Recently, Robert McQueen brought up the idea of possibly using the Linux kernel’s binfmt_misc mechanism to modify the toolbox binary on the fly. I haven’t explored this in any seriousness, but maybe I will if the current set-up doesn’t work out.

Toolbx @ Community Central

At 15:00 UTC today, I will be talking about Toolbx on a new episode of Community Central. It will be broadcast live on BlueJeans Events (formerly Primetime) and the recording will be available on YouTube. I am looking forward to seeing some friendly faces in the audience.

Toolbx — bypassing the immutability of OCI containers

This is a deep dive into some of the technical details of Toolbx. I find myself regularly explaining them to various people, so I thought that I should write them down. Feel free to read and comment, or you can also happily ignore it.

The problem

OCI containers are famous for being immutable. Once a container has been created with podman create, it’s attributes can’t be changed anymore. For example, the bind mounts, the environment variables, the namespaces being used, and all the other attributes that can be specified via options to the podman create command. This means that once there’s a Toolbx, it wouldn’t be possible to give it access to a new set of files from the host if the need arose. The Toolbx would have to be deleted and re-created with access to the new paths.

This is a problem, because a Toolbx is where the user sets up her development and troubleshooting environment. Re-creating a Toolbx might mean reinstalling a number of different packages, tweaking configuration files, redeploying various artifacts and so on. Having to repeat all that in the middle of a long hacking session, just because the container’s attributes need to be tweaked, can be annoying.

This is unlike Flatpak containers, where it’s possible to override the permissions of a Flatpak either persistently through flatpak override or temporarily during flatpak run.

Secondly, as the Toolbx code evolves, we want to be able to transparently update existing Toolbxes to enable new features and fix bugs. It would be a real drag if users had to consciously re-create their containers.

The solution

Toolbx bypasses this by using a special entry point for the container. Those inquisitive types who have run podman inspect on a Toolbx container might have noticed that the toolbox executable itself is the container’s entry point.

$ podman inspect --format "{{.Config.Cmd}}" --type container fedora-toolbox-36
toolbox --log-level debug init-container ...

This means that when Toolbx starts a container using podman start, the toolbox init-container command gets run as the first process inside the container. Only after this has run, does the user’s interactive shell get spawned.

Instead of setting up the container entirely through podman create, Toolbx tries to use this reflexive entry point as much as possible. For example, Toolbx doesn’t use podman create --volume /tmp:/tmp to give access to the host’s /tmp inside the container. It bind mounts the entire root filesystem from the host at /run/host in the container with podman create --volume /:/run/host. Then, later when the container is started, toolbox init-container recursively bind mounts the container’s /run/host/tmp to /tmp. Since the container has its own mount namespace, the /run/host and /tmp bind mounts are neatly hidden away from the host.

Therefore, if in future additional host locations need to be exposed within the Toolbx, then those can be added to toolbox init-container, and once the user restarts the container after updating the toolbox executable, the new locations will show up inside the existing container. Similarly, if the mount parameters of an existing location need to be tweaked, or if a host location needs to be removed from the container.

This is not restricted to just bind mounts from the host. The same approach with toolbox init-container is used to configure as many different aspects of the container as possible. For example, setting up users, keeping the timezone and DNS configuration synchronized with the host, and so on.

Further details

One might wonder how a Toolbx container manages to have a toolbox executable inside it, especially since the toolbox package is not installed within the container. It is achieved by bind mounting the toolbox executable invoked by the user on the host to /usr/bin/toolbox inside the container.

This has some advantages.

There is always only one version of the toolbox executable that’s involved — the one that’s on the host. This means that the exact invocation of toolbox init-container, which is baked into the Toolbx and shows up in podman inspect, is the only interface that needs to be kept stable as the Toolbx code evolves. As long as toolbox init-container can be invoked with that specific command line, everything else can be changed because it’s the same executable on both the host and inside the container.

If the container had a separate toolbox package in it, then the user might have to separately update another executable to get the expected results, and we would have to ensure that different mismatched versions of the executable can work with each other across the host and the container. With a growing number of containers, the former would be a nightmare for the user, while the latter would be almost impossible to test.

Finally, having only one version of the toolbox executable makes it a lot easier for users to file bug reports. There’s only one version to report, not several spread across different environments.

This leads to another problem

Once you let this sink in, you might realize that bind mounting the toolbox executable from the host into the Toolbx means that an executable from a newer or different operating system might be running against an older or different run-time environment inside the container. For example, an executable from a Fedora 36 host might be running inside a Fedora 35 Toolbx, or one from an Arch Linux host inside an Ubuntu container.

This is very unusual. We only expect executables from an older version of an OS to keep working on newer versions of the same OS, but never the other way round, and definitely not across different OSes.

I will leave you with that thought and let you puzzle over it, because it will be the topic of a future post.

Toolbx is now on Matrix

Toolbx now has its own room on matrix.org. Point your Matrix clients to #toolbx:matrix.org and join the conversation.

We are working on setting up an IRC bridge with Libera.Chat but that will take a few more months as we go through the process to register our project channel.