Fedora meets RHEL: upgrading UBI to RHEL

Six years ago

As part of our efforts to make Fedora Workstation more attractive for developers, particularly those building applications that would be deployed on Red Hat Enterprise Linux, we had made it easy to create gratis, self-supported Red Hat Enterprise Linux virtual machines inside GNOME Boxes. You had to join the Red Hat Developer Program by creating an account on developers.redhat.com and with that you not only had gratis, self-supported access to RHEL, but also a series of other products like Red Hat JBoss Middleware, Red Hat OpenShift Container Platform and so on.

Few years later

Fedora Silverblue became a thing, and so did Toolbx. So, we decided to take this one step further.

Toolbx already had support for Red Hat Enterprise Linux for the past two and a half years. It means that on RHEL hosts, Toolbx sets up a RHEL container that has access to all the software and support that the host is entitled to. On hosts that aren’t running RHEL, if you want, it will set up a gratis, self-supported container based on the Red Hat Universal Base Image, which is a limited subset of RHEL:

$ toolbox create --distro rhel --release 9.2

However

This works well only as long as you are running a Red Hat Enterprise Linux host. Otherwise, you quickly run into the limitations of the Red Hat Universal Base Image, which is designed for distributing server-side applications, and not so much for persistent interactive CLI environments. For example, you won’t enjoy hacking on GNOME components like GTK, Settings, Shell or WebKitGTK in it because of the sheer amount of missing dependencies.

Today is glorious

You can now have gratis, self-supported Red Hat Enterprise Linux Toolbx containers on Fedora hosts that have access to the entire set of RHEL software, beyond the limited subset offered by UBI.

As always, you need to join the Red Hat Developer Program. Make sure that your account has simple content access enabled, by logging into access.redhat.com and then clicking the subscriptions link at the very top left.

Install subscription-manager on your Fedora host to register it with your Red Hat Developer Program account, create a RHEL Toolbx container as before, and you are off to the races!

Thanks to Pino Toscano for helping me iron out all the wrinkles in the pipeline to get everything working smoothly.

Toolbox — after a gap of 15 months

We just released version 0.0.99, and I realized that it’s been a while since I blogged about Toolbox. So it’s time to address that.

Rewritten in Go

About a year ago, Ondřej Míchal single-handedly rewrote Toolbox in Go, making it massively easier to work on the code compared to the previous POSIX shell implementation. Go comes with much nicer facilities for command line parsing, error handling, logging, parsing JSON, and in general is a lot more pleasant to program in. Plus all the container tools in the OCI ecosystem are written in Go anyway, so it was a natural fit.

Other than the obvious benefits of Go, the rewrite immediately fixed a few bugs that were inherently very cumbersome to fix in the POSIX shell implementation. Something as simple as offering a –version option, or avoiding duplicate entries when listing containers or images was surprisingly difficult to achieve in the past.

What’s more, we managed to pull this off by retaining full compatibility with the previous code. So users and distributors should have no hesitation to update.

Towards version 0.1.0

We have been very conservative about our versioning scheme so far due to the inherently prototype nature of Toolbox. All our release numbers have followed the 0.0.x format. We thought that the move to Go deserves at least a minor version bump, but we also wanted to give it some time to shake out any bugs that might have crept in; and implement the features and fix the bugs that have been on our short-term wish list before putting a 0.1.0 stamp on it.

Therefore, we started a series of 0.0.9x releases to work our way towards version 0.1.0. The first one was 0.0.90 which shipped the Go code in March 2020, and we are currently at 0.0.99. Suffice to say that we are very close to the objective.

Rootful Toolboxes

Sometimes a rootless OCI container just isn’t enough because it can’t do things that require privilege escalation beyond the user’s current user ID on the host. This means that various debugging tools, such as Nmap, don’t work.

Therefore, we added support for running toolbox as root in version 0.0.98.1. This should hopefully unlock various new use-cases that were so far not possible when running rootless.

When running as root, Toolbox cannot rely on things like the user’s session D-Bus instance or the XDG_RUNTIME_DIR environment variable, because sudo doesn’t create a full-fledged user session that offers them. This means that graphical applications can only work by connecting to a X11 server, but then again running graphical applications as root is never a good idea to begin with.

Red Hat Universal Base Image (or UBI)

We recently took the first step towards supporting operating system distributions other than Fedora as first class citizens. From version 0.0.99 onwards, Toolbox supports Red Hat Enterprise Linux hosts where it will create containers based on the Red Hat Universal Base Image by default.

On hosts that aren’t running RHEL, one can still create UBI containers as:
$ toolbox create --distro rhel --release 8.3


Read more

Those were some of the big things that have happened in Toolbox land since my last update. If you are interested in more details, then you can read Ondřej’s posts where he writes at length about the port to Go and the changes in each of the releases since then.

Toolbox: a fall 2019 update

Things have been moving fast in Toolbox land, and it’s time to talk about what we have been doing lately.

New home

Toolbox is now part of the containers organization on GitHub. We felt that the project had outgrown the prototype stage — going by the activity on the GitHub project it’s safe to say that there are at least a few thousand users who rely on it to get their work done; and we are increasingly working towards expanding the scope of the project to go beyond just setting up a development environment.

Housing the project in my personal GitHub namespace meant that I couldn’t share admin access with other contributors, and this was a problem we had to address as more and more people keep joining the project. Over the past year, we have developed a really good working relationship with the Podman team and other members of the containers organization, without whom Toolbox wouldn’t exist, so moving in under the same umbrella felt like a natural next step towards growing the project.

Migration to cgroups v2

Fedora 31 ships with cgroups v2 by default. The major blocker for cgroups v2 adoption so far was the lack of support in the various container and virtualization tools, including the Podman stack. Since Toolbox containers are just OCI containers managed with Podman, we saw some action too.

After updating the host operating system to Fedora 31, Toolbox will try to migrate your existing containers to work with cgroups v2. Sadly, this is a somewhat complicated move, and in theory it’s possible that the migration might break some containers depending on how they were configured. So far, as per our testing, it seems that containers created by Toolbox do get smoothly migrated, so hopefully you won’t notice.

However, if things go wrong, barring a delicate surgery on the container requiring some pretty arcane knowledge, your only option might be to do a factory reset of your local Podman installation. As factory resets go, you will lose all your existing OCI containers and images on your local system. This is a sad outcome for those unfortunate enough to encounter it. However, if you do find yourself in this quagmire then take a look at the toolbox reset command.

Note that you need to have podman-1.6.2 and toolbox-0.0.16 for the above to work.

Also, this is one of those changes where it bears repeating that online RPM package updates are fragile. They are officially unsupported on Fedora Workstation, and variants like CoreOS and Silverblue make it even harder. A cgroups v2 migration is only expected to work on a freshly booted system.

Improvements

The last six months have seen a whole boatload of new features and improvements. Here are some highlights.

On Fedora Silverblue and Workstation, GNOME Terminal keeps track of the current Toolbox container, and just like it preserves the current working directory when opening a new terminal, it’s also able to preserve the Toolbox environment. This is quite convenient when hacking on a Silverblue system, because it removes the extra step of entering a toolbox after opening a new tab or window.

The integration with the host operating system has been deepened. Toolbox containers can now access virtual machines managed by the host’s system libvirt instance, and the host’s ulimits are preserved. The entirety of /dev is made available inside the toolbox as a step towards supporting the proprietary Nvidia driver to enable CUDA for AI/ML frameworks like TensorFlow.

The container’s /run/host now has big chunks of the host’s file hierarchy. This is handy for one-off use-cases which require access to parts of the host that aren’t covered by Toolbox by default.

Last but not the least, Kerberos now works inside Toolbox containers. This will make it easier to contribute to Fedora itself from inside a toolbox.

GtkBuilder, Vala and WebKit

This article is about a set of bugs that used to exist, and are in various stages of getting fixed. As such this is merely a historical anecdote. I had planned to write about something entirely different, and hopefully more useful, but I ended up having too much fun with this to just ignore it.

To use a WebKitWebView inside a GTK+ template, one needs to workaround the fact that WebKitWebView breaks the heuristics in GtkBuilder to guess the GType from the human readable type name. That’s easy. Anybody who has used GObject is likely to have encountered some dialect of g_type_ensure, or, as the more learned will point out, GtkBuilder has a type-func attribute for cases like these.

The fun begins when you start debating which workaround to use.

It turns out that type-func doesn’t work with Vala.

A few buglets in GtkBuilder means that if you use class and type-func together, the latter will be ignored. It’s likely nobody used them together because even if class is specified as mandatory the parser doesn’t enforce that. On the other hand, the Vala compiler effectively treats class as mandatory because it doesn’t understand type-func. So, you must use both to avoid a build failure, but if you do, you get a run-time failure because your type-func is ignored.

So, typeof (WebKit.WebView) wins, which is Vala’s equivalent of g_type_ensure.

I don’t know how things are with other language bindings. Vala is what I happen to be using right now, so that’s where I chose to focus.

Thanks to Saiful, for pointing out the problem with WebKitWebView and GtkBuilder. It was fascinating.

GdMainBox — the new content-view widget in libgd

Now that I have written at length about the new fluid overview grids in GNOME Photos, it is time to talk a bit about the underlying widgets doing the heavy lifting. Hopefully some of my fellow GNOME developers will find this interesting.

Background

Ever since its incubation inside Documents, libgd has had a widget called GdMainView. It is the one which shows the grid or list of items in the new GNOME applications — Boxes, Photos, Videos, etc.. It is where drag-n-drop, rubber band selection and the selection mode pattern are implemented.

However, as an application developer, I think its greatest value is in making it trivial to switch the main content view from a grid to a list and back. No need to worry about the differences in how the data will be modelled or rendered. No need to worry about all the dozens of little details that arise when the main UI of an application is switched like that. For example, this is all that the JavaScript code in Documents does:

  let view = new Gd.MainView({ shadow_type: Gtk.ShadowType.NONE });
  …
  view.view_type = Gd.MainViewType.LIST; // use a list
  …
  view.view_type = Gd.MainViewType.ICON; // use a grid

Unfortunately, GdMainView is based on GtkIconView and GtkTreeView. By this time we all know that GtkIconView has various performance and visual problems. While GtkTreeView might not be slow, the fact that it uses an entirely separate class of visual elements that are not GtkWidgets limits what one can render using it. That’s where GdMainBox comes in.

GdMainBox

GdMainBox is a replacement for GdMainView that is meant to use GtkFlowBox and GtkListBox instead.

GListModel *model;
GtkWidget *view;

model = /* a GListModel containing GdMainBoxItems */
view = gd_main_box_new (GD_MAIN_BOX_ICON);
gd_main_box_set_model (GD_MAIN_BOX (view), model);
g_signal_connect (view,
                  "item-activated",
                  G_CALLBACK (item_activated_cb),
                  data);
g_signal_connect (view,
                  "selection-mode-request",
                  G_CALLBACK (selection_mode_request_cb),
                  data);
g_signal_connect (view,
                  "selection-changed", /* not view-selection-changed */
                  G_CALLBACK (selection_changed_cb),
                  data);

If you are familiar with with old GdMainView widget, you will notice the striking similarity with it. Except one thing. The data model.

GdMainView expected applications to offer a GtkTreeModel with a certain number of columns arranged in a certain order with certain type of values in them. Nothing surprising since both GtkIconView and GtkTreeView rely on the existence of a GtkTreeModel.

In the world of GtkListBoxes and GtkFlowBoxes, the data model is GListModel, a list-like collection of GObjects [*]. Therefore, instead of columns in a table, they need objects with certain properties, and methods to access them. These are codified in the GdMainBoxItem interface which every rendered object needs to implement. You can look at this commit for an example. A nice side-effect is that an interface is inherently more type-safe than a GtkTreeModel whose expected layout is expressed as enumerated types. The compiler can not assert that a certain column does have the expected data type, so it left us vulnerable to bugs caused by inadvertent changes to either libgd or an application.

But why a new widget?

You can definitely use a GtkFlowBox or GtkListBox directly in an application, if that’s what you prefer. However, the vanilla GTK+ widgets don’t offer all the necessary features. I think there is value in consolidating the implementation of those features in a single place that can be shared across modules. It serves as a staging area for prototyping those features in a reasonably generic way so that they can eventually be moved to GTK+ itself. If nothing else, I didn’t want to duplicate the same code across the two applications that I am responsible for — Documents and Photos.

One particularly hairy thing that I encountered was the difference between how selections are handled by the stock GtkFlowBox and the intended behaviour of the content-view. Other niceties on offer are expanding thumbnails, selection mode, and drag-n-drop.

If you do decide to directly use the GTK+ widgets, then I would suggest that you at least use the same CSS style classes as GdMainBox — “content-view” for the entire view and “tile” for each child.

The future

I mentioned changing lists to grids and vice versa. Currently, GdMainBox only offers a grid of icons because Photos is the only user and it doesn’t offer a list view. That’s going to change when I port Documents to it. When that happens, changing the view is going to be just as easy as it used to be.

gd_main_view_set_view_type (GD_MAIN_BOX (view), GD_MAIN_BOX_LIST);

[*] Yes, it’s possible to use them without a model, but having a GListModel affords important future performance optimizations, so we will ignore that possibility.

WilberWeek 2017

For the past three days, I am in El Bruc, a little village on the side of Montserrat near Barcelona, for WilberWeek — the annual retreat for members of the GIMP and GEGL communities. We have rented out half of the Can Serrat art residency for 10 days of good food, idyllic surroundings, sedated discussions and a bit of moody hacking.

So far, I have spent my time eating paella; understanding the nuances of non-destructive image editing from Øyvind Kolås; walking in the countryside; and poring over Darktable and Shotwell to learn the workings of various “exposure and blacks” tools and get RAW decoding right. I have vague expectations that this will greatly improve the image editing experience in GNOME Photos.

I am grateful to the GIMP project for inviting me and sponsoring my stay, and especially to Jehan Pagès and Aryeom for coming all the way to Barcelona to pick me up.

Photographs featuring Wilber are from Michael Natterer’s Twitter feed.

Core Apps Hackfest 2016: report

I spent last weekend at the Core Apps Hackfest in Berlin. The agenda was to work on GNOME’s core applications: Documents, Files, Music, Photos, Videos, Usage, etc.; to raise their overall standard and to make them push beyond the limits of the framework. There were 19 of us and among us we covered a wide range of modules and areas of expertise.

I spent most of my time on the plumbing necessary for Documents and Photos to use GtkFlowBox and GtkListBox. The innards of Photos had already been overhauled to reduce its dependency on GtkTreeModel. Going into the hackfest we were sorely lacking a widget that had all the bells and whistles we need — the idiomatic GNOME 3 selection mode, and seamlessly switching between a list and grid view. So, this is where I decided to focus my energy. As a result, we now have a work-in-progress GdMainBox widget in libgd to replace the old GtkIconView/GtkTreeView-based GdMainView.

In fact, GtkListBox and GtkFlowBox was a recurring theme at the hackfest. Carlos Soriano and Georges were working on using them in Files, and whenever anybody uses them in a non-trivial manner there is the inevitable discussion about performance. Good thing that Benjamin was around. He spent the better part of a tram ride and more than an hour at the whiteboard, sketching out a strategy to make GtkListBox more efficient than it is today.

Like last year, Øyvind joined us. We talked about GEGL, and I finally saw the new GIMP in action. I rarely use GIMP, and I am not sure I have ever built it from source, but I have been reading its sources on a semi-regular basis for almost a year now. It was good to finally address this aberration. Øyvind had with him a cheap hand-held DLNA media renderer that was getting stuck when trying to render more than one image with dleyna-render and Photos. Zeeshan helped me poke at it, but unfortunately we didn’t get anywhere.

Other than that, Petr Stetka impressed everyone with his progress on the new Usage application. Georges refreshed his patches to implement the new Online Accounts Settings panel, Carlos Garnacho helped me with GtkGesture, and I reviewed various patches and branches that had been on my list for a while.

Many thanks to Red Hat for sponsoring me; to Carlos Soriano and Joaquim for organizing the event; to Kinvolk for being such gracious hosts; and to Collabora for the nice dinner.

Debugging GNOME Online Accounts

I spent some time today documenting how to debug various problems with online account integration in GNOME. It is also linked from the main GNOME Online Accounts wiki page. So, you can find it via the usual click-stream and don’t need to rely on this blog.

It is still basic. I want to expand it further to cover how to debug specific integration points. For example, issues with mounting Google Drive in Nautilus, or some breakage involving Kerberos. Anyway, it’s a start and I am quite happy to have one less thing on my TODO list. Sometimes, things in GNOME seem like a maze of hidden D-Bus daemons, known only to the initiated. Hopefully, this will make it more approachable.

Oh, and don’t shy away from editing it. It’s a Wiki!

MALLOC_PERTURB_

MALLOC_PERTURB_ is a useful thing. If you are developing on glibc-based systems, I encourage you to put this snippet in your ~/.bash_profile:

MALLOC_PERTURB_=$(($RANDOM % 255 + 1))
export MALLOC_PERTURB_


I have been using it for the last six years on all my computers (3 laptops running every Fedora x86_64 build released since then), and while things haven’t exploded, it has helped uncover the odd bug every once in a while. One such occasion presented itself this week.

I was busy following Ondrej’s hint, debugging why Eye of GNOME was taking so long to open a file from ownCloud. Imagine my shock when it would just crash after showing the window. The same optimization was working just fine on the gnome-3-18 branch, while master was crashing even without any changes. How could that happen? Obviously, while it was failing for me, it was working fine for all those who run unstable GNOME versions via jhbuild, gnome-continous, Fedora rawhide, etc.. Otherwise we would have been debugging this crash, and not a performance issue.

I guess, most of them didn’t have MALLOC_PERTURB_.

Here is another such story.

In case you were wondering, there is already an update on its way to Fedora 24 address the crash.

Content Apps Hackfest 2015: report

I spent most of the previous week attending the Content Apps Hackfest in Madrid. The agenda was to work on GNOME’s content applications: Documents, Files, Music, Photos and Videos; to identify missing features and sore points; and raise the standard of the overall content experience in GNOME. Of these, I focused mainly on Documents and Photos.

The first day was spent discussing high-level goals:

• Previews – should the content applications take over the role played traditionally by Evince and Eye of GNOME?
• Status and plans about sharing. We are still waiting for a platform-wide sharing framework, but since it is a crucial feature for some of the applications, it has become a case of perfection being the enemy of good.
• Porting away from GtkIconView to GtkFlowBox. GtkIconView has some very visible problems – it becomes terribly slow when the images are updated; the grid’s content doesn’t re-flow when the size of the widget changes leading to an awkward gutter appearing on the side; cannot pack any GTK+ widget into a GtkCellRenderer.

I set up a live Google Hangouts stream on my laptop to let those who couldn’t attend in person to join us remotely. With Andreas’ help, I fixed the HiDpi breakage in Photos’ cropping tool, Bastien fired some new LibreOffice builds, while the other Bastian revamped Documents’ wiki page.

The second day started with breakfast in Florian’s flat. It was less chatty with people quietly hacking away, presumably, to rest their tired throats. Bastien picked up Pranav’s patches to integrate LOKDocView widget in Documents. I worked a bit on polishing rough edges in Photos’ editing code, and tried to keep up with the endless stream of patches from Alessandro and Umang.

Discussions picked up a bit on the third and final day. Allan spent a lot of time talking to the developers of each application. UX reviews were done, new designs were made, and future plans were sketched out. I spent some time with him working on a roadmap for Photos. We are still cleaning up our rough notes and transferring them to the wiki, but I think it is good enough for others to take a look. Øyvind paid us a surprise visit, and we grabbed the opportunity for some impromptu chat.

Many thanks to Red Hat for letting me attend and sponsoring me, and to Medialab-Prado for being such wonderful hosts.