Toolbox is now Toolbx

Toolbox is being renamed to Container Toolbx or just Toolbx.

I had always been uncomfortable by the generic nature of the term toolbox and people keep complaining that it’s terribly difficult to search for. Recently, we have been trying to improve the online presence of the project by creating a website and a Twitter handle, and it’s impossible to find any decent Internet real estate with anything toolbox.

It looks like dropping the penultimate character from words to form names is a thing these days, hence Toolbx.

We haven’t yet renamed the Git repository or anything in the code or the binary or the manuals. Renaming the binary, for example, has implications for existing containers, and we don’t want to cause any needless disruption for users. So, those will gradually happen over time with all the necessary compatibility aliases and such.

Meanwhile, Fedora Magazine has published an interview with yours truly about Toolbx that talks about the history, latest improvements, future direction, and various other aspects of the project.

It should be obvious, but the Toolbx website was made by Jakub Steiner.

Fedora Toolbox is now just Toolbox

Fedora Toolbox has been renamed to just Toolbox. Even though the project is obviously driven by the needs of Fedora Silverblue and uses technologies like Buildah and Podman that are driven by members of the wider Fedora project, it was felt that a toolbox container is a generic concept that appeals to a lot many more communities than just Fedora. You can also think of it as a nod to coreos/toolbox which served as the original inspiration for the project, and there are plans to use it in Fedora CoreOS too.

If you’re curious, here’s a subset of the discussion that drove the renaming.

There have already been two releases with the new name, so I assume that almost all users have been migrated.

Note that the name of the base OCI image for creating Fedora toolbox containers is still fedora-toolbox for obvious namespacing reasons, but the names of the client-side command line tool, and the overall project itself have changed. That way you could have a debian-toolbox, a centos-toolbox and so on.

It should be obvious, but the Toolbox logo was designed and created by Jakub Steiner.

Fedora Toolbox — Under the hood

A few months ago, we had a glimpse at Fedora Toolbox setting up a seamlessly integrated RPM based environment, complete with dnf, on Fedora Silverblue. But isn’t dnf considered a persona non grata on Silverblue? How is this any different from using the existing Fedora Workstation then? What’s going on here?

Today we shall look under the covers to answer some of these questions.

The problem

The immutable nature of Silverblue makes it difficult to install arbitrary RPMs on the operating system. It’s designed to install graphical applications as Flatpaks, and that’s it. This has many advantages. For example, robust upgrades.

However, there are legitimate cases when one does want to install some random RPMs. For example, when you need things like *-devel packages, documentation, GCC, gofmt, strace, valgrind or whatever else is necessary for your development workflow. While rpm-ostree does offer a way around this, it’s painful to have to reboot every time you change the set of packages on the system, and it negates the advantages of immutability in the first place.

Containers

By this time some of you are surely thinking that containers ought to solve this somehow; and you’d be right. Fedora Toolbox uses containers to set up an elaborate chroot-like environment that’s separate from the immutable OSTree based operating system.

And once you are down to containers, Docker isn’t far away — surely this can be hacked together with Docker; and you’d be right again. Almost. You can hack it up with Docker but it wouldn’t be ideal.

The problem with Docker is that it requires root privileges. Every time you invoke the docker command, it has to be prefixed with sudo or be run as root. That’s fine if you all you want is a place to install some RPMs. It would’ve required root anyway. However, it’s annoying if you want GNOME Terminal to default to running a shell inside your RPM based development environment. You’d have to enter the root password to even get to an unprivileged shell prompt.

So, instead of using Docker, Fedora Toolbox uses something called Podman. Podman is a fully-featured container engine that aims to be a drop-in replacement for Docker. Thanks to the Open Container Initiative (or OCI) standardizing the interfaces to Docker images and runtimes, every OCI container and image can be used with either Docker or Podman.

The good thing about Podman is that can be used rootless — that is, without root privileges. So, that’s great.

Containers are weird, though

Containers are pretty widely popular these days, but not everybody who is transitioning from the current RPM based Fedora Workstation to Silverblue can be expected to set things up from first principles using nothing but the podman command line. It will surely increase the cognitive load of undergoing the transition, hindering Silverblue adoption.

Even if someone familiar with the technology is able to set things up, pitfalls abound. For example, why is the display server not working, why is the SSH agent not working, why are OpenGL and Vulkan programs not working, or why is sshfs not working, or why LLVM and LibreOffice are failing to build, etc..

Let’s be honest. The number of people who understand both container technology and the workings of a modern graphical operating system well enough to sort these problems in a jiffy is vanishingly small. I know that at least I don’t belong to that group.

Container images are optimized for non-interactive use and size, whereas we are talking about the interactive shell running in your virtual terminal emulator. For example the fedora OCI image comes with the coreutils-single RPM, which doesn’t have the same user experience as the coreutils package that we are all familiar with.

So, it’s clear that we need a pre-configured, and at times, opinionated, solution on top of Podman.

The solution

Fedora Toolbox starts with the similarly named fedora-toolbox OCI image hosted on the Fedora Container Registry. There’s one for every Fedora branch. Currently those are Fedoras 28, 29 and 30. These images are based on the fedora image, with an altered package set to offer an interactive user experience that’s similar to the one on Silverblue.

When you invoke the fedora-toolbox create command, it pulls the image from the registry, and then tailors it to the local user. It creates a user with a UID matching $UID, a home directory matching $HOME and the right group memberships; and it ensures that various bits and pieces from the host, such as the home directory, the display server, the D-Bus instances, various pieces of hardware, etc. are available inside the container. These customizations are saved as another image named fedora-toolbox-user. Finally, an OCI container, also named fedora-toolbox-user, is created out of this image.

If you are curious, run podman images and podman ps –all to verify the above.

Once the toolbox container has been created, subsequent fedora-toolbox enter commands execute the users preferred shell inside it, giving the impression of being in an alternate RPM flavoured reality on a Silverblue system.

If you are still curious, then open /usr/bin/fedora-toolbox and have a peek. It’s just a shell script, after all.

Fedora Toolbox — Hacking on Fedora Silverblue

Fedora Silverblue is a modern and graphical operating system targeted at laptops, tablets and desktop computers. It is the next-generation Fedora Workstation that promises painless upgrades, clear separation between the OS and applications, and secure and cross-platform applications. The basic operating system is an immutable OSTree image, and all the applications are Flatpaks.

It’s great!

However, if you are a hacker and decide to set up a development environment, you immediately run into the immutable OS image and the absence of dnf. You can’t install your favourite tools, editors and SDKs the way you’d normally do on Fedora Workstation. You can either unlock your immutable OS image to install RPMs through rpm-ostree and give up the benefit of painless upgrades; or create a Docker container to get an RPM-based toolbox but be prepared to mess around with root permissions and having to figure out why your SSH agent or display server isn’t working.

Enter Fedora Toolbox.

It makes it trivial to get a mutable development environment on Silverblue:

[rishi@bollard ~]$ fedora-toolbox create
[rishi@bollard ~]$ fedora-toolbox enter
🔹[rishi@toolbox ~]$


It uses OCI containers underneath, but takes away the cognitive overhead of thinking about containers by providing a seamless integration with the host environment. It uses rootless podman and buildah, so there’s no root in the picture either.

If you are going to try it out, make sure that you have the runc-1.0.0-56.dev.git78ef28e package in your Silverblue image. There’s also an ongoing review to get fedora-toolbox added to Fedora. If you don’t feel comfortable mucking around with rpm-ostree on the command-line, then fear not. Very soon all the necessary pieces will be part of the OS image, making it that much easier to start hacking on your Silverblue.

GNOME Photos: an overview of thumbnailing

From time to time, I find myself being asked about various details about how content is thumbnailed in GNOME Photos, and the reasons behind various implementation decisions. I can never remember all the details, and always have to dig through Git history and bug reports across multiple modules to come up with an answer. I am hoping that this brain dump will be more persistent than my memory, and more holistic than random comments here and there.

Feel free to read and comment, or you can also happily ignore it.

Background

Having accurate and quality thumbnails is absolutely crucial for Photos. The main user interface is a grid of thumbnails. By design, it tries hard not to expose the filesystem, which means that the user doesn’t have the path or directory hierarchy to complement the contents of the grid. In comparison, thumbnails can be optional in a file manager. Note how Files has settings to disable thumbnailing, and defaults to not thumbnailing remote content, but users can still go about interacting with their files.

Thumbnailing in GNOME is spread across GIO, GVfs, GnomeDesktopThumbnailFactory, and together they implement the Thumbnail Managing Standard. Usually, one uses GIO to lookup thumbnails from the cache and the state they are in, while GnomeDesktopThumbnailFactory is used to create and store the thumbnail files. These thumbnails are stored in the global thumbnail cache in $XDG_CACHE_HOME/thumbnails, and are often, but not necessarily, created by the thumbnailers listed under /usr/share/thumbnailers. This is how most components (eg., GTK+’s GtkFileChooserWidget), and applications (eg., Files and Videos) show thumbnails.

Then there are those “odd” ones that have their own custom setup.

Prior to version 3.24, Photos entirely relied on the global cache and the aforementioned GNOME APIs for its thumbnails. That changed in 3.24 when it switched to its own custom thumbnailer and application specific cache.

Requirements

Ever since editing was added in 3.20, we felt the need to ensure that the thumbnail represents the current state of each item. Being a non-destructive editor, Photos never modifies the original file but separately serializes the edits to disk. The image is rendered by loading the original file, deserializing the edits into objects in memory and running the pixels through them [1]. Therefore, to have the thumbnails accurately represent the current state of the item, it would have to do something similar. However, the edits are application-specific [2], so it is not reasonable to expect the generic OS-wide thumbnailers to be able to handle them.

I believe this is a requirement that all non-destructive image editors have [3]. Notable examples are Darktable and Shotwell.

Secondly, it is important to be able to create and lookup thumbnails of a specific size, as opposed to enumerated constants with pre-determined presets.

The standard specifies two sizes – normal, which is 128×128, and large, which is 256×256. I think this was alright in a world without HiPPI, and is also fine if the thumbnails are either too small or are not an existential necessity for the application. For a HiPPI display with a scaling factor of N, we want to make the thumbnail grid as visually appealing as possible by pumping in NxN times more pixels. Since Photos wants the thumbnails to be 256×256 logical pixels, they should be 256Nx256N raw device pixels on HiPPI. To make things complicated, the cache might get used across different scaling factors – either display or disk got switched, multi-monitor with different resolutions, etc..

Upscaling the low-resolution counterpart of a thumbnail by N is still passable, but it looks much worse if the thumbnail is significantly smaller. Although, I must note that this was the easiest hurdle to surmount. It originates from GIO’s desire to fallback to 128×128 thumbnails, even if the application asked for 256×256. This is pretty straightforward to fix, if necessary.

Last but not the least, I find it important to version the cache to tide over bugs in the thumbnailer. If the cache isn’t versioned, then it is difficult to discard thumbnails that might have been generated by a broken thumbnailer. Hopefully, such bugs would be rare enough that it won’t be necessary to invalidate the cache very often, but when they do happen, it is very reassuring to be able to bump the version, and be guaranteed that users won’t be looking at a broken user interface.

Solution

Starting from version 3.24, Photos uses its own out-of-process thumbnailer and cache [4]. The cache is at $XDG_CACHE_HOME/gnome-photos/thumbnails/$SIZE-$GENERATION, where SIZE is the thumbnail size in raw device pixels and GENERATION is the cache’s version. The main application talks to the thumbnailer over peer-to-peer D-Bus and a simple, cancellable private D-Bus API.

The thumbnailer isn’t separately sandboxed, though. It might be an interesting thing to look at for those who don’t use Flatpak, or to restrict it even more than the main application when running inside Flatpak’s sandbox.

Known bugs

Photos’ thumbnailing code can be traced back to its origins in GNOME Documents. They don’t persistently track thumbnailing failures, and will attempt to re-thumbnail an item that had previously failed when any metadata change is detected. In short, they don’t use G_FILE_ATTRIBUTE_THUMBNAILING_FAILED. The current behaviour might help to overcome a temporary glitch in the network, or it can be simply wasteful.

They predate the addition of G_FILE_ATTRIBUTE_THUMBNAIL_IS_VALID and don’t update the thumbnail once an item gets updated. This could have still been done using GnomeDesktopThumbnailFactory, but that’s water under the bridge, and should possibly be fixed. Although, images don’t tend to get updated so often, which is probably why nobody notices it.

Related to the above point, currently the modification time of the original doesn’t get stored in the thumbnail. It slipped through the cracks while I was reading the sources of the various modules involved in creating thumbnails in GNOME. However, a versioned cache makes it possible to fix it.

[1] If you are reading between the lines, then you might be thinking that it is serializing and deserializing GeglOperations, and you’d be right.

[2] GEGL might be a generic image processing library with its set of built-in operations, but for various reasons, an application can end up carrying its own custom operations.

[3] The idea of an application storing its edits separately from the original can strike as unusual, but this is how most modern image editors work.

[4] Both Darktable and Shotwell have similar thumbnailing infrastructure. You can read about them here and here respectively.

Stable GNOME Photos Flatpaks moved to Flathub

Starting from version 3.26, the stable GNOME Photos Flatpaks have been moved to Flathub. They are no longer available from GNOME’s Flatpak repository.

To migrate, first delete the old build:

$ flatpak uninstall org.gnome.Photos/x86_64/stable


Then install it from Flathub:

$ flatpak remote-add --from flathub https://flathub.org/repo/flathub.flatpakrepo
$ flatpak install flathub org.gnome.Photos


Note that this is only about the stable build. The nightly continues to be available from its existing location in GNOME’s repository. You can keep updating it with:

$ flatpak update --user org.gnome.Photos/x86_64/master


GNOME Photos Flatpaks

I joined the recent buzz around Flatpak manifests in GNOME, and gave the GNOME Photos builds some routine maintenance. The stable build has been updated to the latest 3.22.x point releases; and the nightly, which I had broken, is again tracking Git master.

To install the stable build:

$ flatpak remote-add --from gnome https://sdk.gnome.org/gnome.flatpakrepo
$ flatpak remote-add --from gnome-apps https://sdk.gnome.org/gnome-apps.flatpakrepo
$ flatpak install gnome-apps org.gnome.Photos


To install the nightly:

$ flatpak remote-add --from gnome-nightly https://sdk.gnome.org/gnome-nightly.flatpakrepo
$ flatpak remote-add --from gnome-apps-nightly https://sdk.gnome.org/gnome-apps-nightly.flatpakrepo
$ flatpak install gnome-apps-nightly org.gnome.Photos


They can be run directly from gnome-shell. However, if you have installed both stable and nightly builds, then you can specifically select one by:

$ flatpak run --branch=stable org.gnome.Photos
$ flatpak run --branch=master org.gnome.Photos