Debarshi's den

Archive for the ‘OSTree’ Category

Toolbox — A fall 2019 update

with 3 comments

toolbox-logo-landscape

Things have been moving fast in Toolbox land, and it’s time to talk about what we have been doing lately.

New home

Toolbox is now part of the containers organization on GitHub. We felt that the project had outgrown the prototype stage — going by the activity on the GitHub project it’s safe to say that there are at least a few thousand users who rely on it to get their work done; and we are increasingly working towards expanding the scope of the project to go beyond just setting up a development environment.

Housing the project in my personal GitHub namespace meant that I couldn’t share admin access with other contributors, and this was a problem we had to address as more and more people keep joining the project. Over the past year, we have developed a really good working relationship with the Podman team and other members of the containers organization, without whom Toolbox wouldn’t exist, so moving in under the same umbrella felt like a natural next step towards growing the project.

Migration to cgroups v2

Fedora 31 ships with cgroups v2 by default. The major blocker for cgroups v2 adoption so far was the lack of support in the various container and virtualization tools, including the Podman stack. Since Toolbox containers are just OCI containers managed with Podman, we saw some action too.

After updating the host operating system to Fedora 31, Toolbox will try to migrate your existing containers to work with cgroups v2. Sadly, this is a somewhat complicated move, and in theory it’s possible that the migration might break some containers depending on how they were configured. So far, as per our testing, it seems that containers created by Toolbox do get smoothly migrated, so hopefully you won’t notice.

However, if things go wrong, barring a delicate surgery on the container requiring some pretty arcane knowledge, your only option might be to do a factory reset of your local Podman installation. As factory resets go, you will lose all your existing OCI containers and images on your local system. This is a sad outcome for those unfortunate enough to encounter it. However, if you do find yourself in this quagmire then take a look at the toolbox reset command.

Note that you need to have podman-1.6.2 and toolbox-0.0.16 for the above to work.

Also, this is one of those changes where it bears repeating that online RPM package updates are fragile. They are officially unsupported on Fedora Workstation, and variants like CoreOS and Silverblue make it even harder. A cgroups v2 migration is only expected to work on a freshly booted system.

Improvements

The last six months have seen a whole boatload of new features and improvements. Here are some highlights.

On Fedora Silverblue and Workstation, GNOME Terminal keeps track of the current Toolbox container, and just like it preserves the current working directory when opening a new terminal, it’s also able to preserve the Toolbox environment. This is quite convenient when hacking on a Silverblue system, because it removes the extra step of entering a toolbox after opening a new tab or window.

The integration with the host operating system has been deepened. Toolbox containers can now access virtual machines managed by the host’s system libvirt instance, and the host’s ulimits are preserved. The entirety of /dev is made available inside the toolbox as a step towards supporting the proprietary Nvidia driver to enable CUDA for AI/ML frameworks like TensorFlow.

The container’s /run/host now has big chunks of the host’s file hierarchy. This is handy for one-off use-cases which require access to parts of the host that aren’t covered by Toolbox by default.

Last but not the least, Kerberos now works inside Toolbox containers. This will make it easier to contribute to Fedora itself from inside a toolbox.

Written by Debarshi Ray

1 November, 2019 at 21:53

Fedora Toolbox is now just Toolbox

with 3 comments

toolbox-logo-landscape

Fedora Toolbox has been renamed to just Toolbox. Even though the project is obviously driven by the needs of Fedora Silverblue and uses technologies like Buildah and Podman that are driven by members of the wider Fedora project, it was felt that a toolbox container is a generic concept that appeals to a lot many more communities than just Fedora. You can also think of it as a nod to coreos/toolbox which served as the original inspiration for the project, and there are plans to use it in Fedora CoreOS too.

If you’re curious, here’s a subset of the discussion that drove the renaming.

There have already been two releases with the new name, so I assume that almost all users have been migrated.

Note that the name of the base OCI image for creating Fedora toolbox containers is still fedora-toolbox for obvious namespacing reasons, but the names of the client-side command line tool, and the overall project itself have changed. That way you could have a debian-toolbox, a centos-toolbox and so on.

It should be obvious, but the Toolbox logo was designed and created by Jakub Steiner.

Written by Debarshi Ray

3 April, 2019 at 19:39

Fedora Toolbox — Under the hood

with 2 comments

rpm-ostree-flatpak-silverblue

A few months ago, we had a glimpse at Fedora Toolbox setting up a seamlessly integrated RPM based environment, complete with dnf, on Fedora Silverblue. But isn’t dnf considered a persona non grata on Silverblue? How is this any different from using the existing Fedora Workstation then? What’s going on here?

Today we shall look under the covers to answer some of these questions.

The problem

The immutable nature of Silverblue makes it difficult to install arbitrary RPMs on the operating system. It’s designed to install graphical applications as Flatpaks, and that’s it. This has many advantages. For example, robust upgrades.

However, there are legitimate cases when one does want to install some random RPMs. For example, when you need things like *-devel packages, documentation, GCC, gofmt, strace, valgrind or whatever else is necessary for your development workflow. While rpm-ostree does offer a way around this, it’s painful to have to reboot every time you change the set of packages on the system, and it negates the advantages of immutability in the first place.

Containers

By this time some of you are surely thinking that containers ought to solve this somehow; and you’d be right. Fedora Toolbox uses containers to set up an elaborate chroot-like environment that’s separate from the immutable OSTree based operating system.

And once you are down to containers, Docker isn’t far away — surely this can be hacked together with Docker; and you’d be right again. Almost. You can hack it up with Docker but it wouldn’t be ideal.

The problem with Docker is that it requires root privileges. Every time you invoke the docker command, it has to be prefixed with sudo or be run as root. That’s fine if you all you want is a place to install some RPMs. It would’ve required root anyway. However, it’s annoying if you want GNOME Terminal to default to running a shell inside your RPM based development environment. You’d have to enter the root password to even get to an unprivileged shell prompt.

So, instead of using Docker, Fedora Toolbox uses something called Podman. Podman is a fully-featured container engine that aims to be a drop-in replacement for Docker. Thanks to the Open Container Initiative (or OCI) standardizing the interfaces to Docker images and runtimes, every OCI container and image can be used with either Docker or Podman.

The good thing about Podman is that can be used rootless — that is, without root privileges. So, that’s great.

Containers are weird, though

Containers are pretty widely popular these days, but not everybody who is transitioning from the current RPM based Fedora Workstation to Silverblue can be expected to set things up from first principles using nothing but the podman command line. It will surely increase the cognitive load of undergoing the transition, hindering Silverblue adoption.

Even if someone familiar with the technology is able to set things up, pitfalls abound. For example, why is the display server not working, why is the SSH agent not working, why are OpenGL and Vulkan programs not working, or why is sshfs not working, or why LLVM and LibreOffice are failing to build, etc..

Let’s be honest. The number of people who understand both container technology and the workings of a modern graphical operating system well enough to sort these problems in a jiffy is vanishingly small. I know that at least I don’t belong to that group.

Container images are optimized for non-interactive use and size, whereas we are talking about the interactive shell running in your virtual terminal emulator. For example the fedora OCI image comes with the coreutils-single RPM, which doesn’t have the same user experience as the coreutils package that we are all familiar with.

So, it’s clear that we need a pre-configured, and at times, opinionated, solution on top of Podman.

The solution

Fedora Toolbox starts with the similarly named fedora-toolbox OCI image hosted on the Fedora Container Registry. There’s one for every Fedora branch. Currently those are Fedoras 28, 29 and 30. These images are based on the fedora image, with an altered package set to offer an interactive user experience that’s similar to the one on Silverblue.

When you invoke the fedora-toolbox create command, it pulls the image from the registry, and then tailors it to the local user. It creates a user with a UID matching $UID, a home directory matching $HOME and the right group memberships; and it ensures that various bits and pieces from the host, such as the home directory, the display server, the D-Bus instances, various pieces of hardware, etc. are available inside the container. These customizations are saved as another image named fedora-toolbox-user. Finally, an OCI container, also named fedora-toolbox-user, is created out of this image.

If you are curious, run podman images and podman ps –all to verify the above.

Once the toolbox container has been created, subsequent fedora-toolbox enter commands execute the users preferred shell inside it, giving the impression of being in an alternate RPM flavoured reality on a Silverblue system.

If you are still curious, then open /usr/bin/fedora-toolbox and have a peek. It’s just a shell script, after all.

Written by Debarshi Ray

21 January, 2019 at 21:20

Fedora Toolbox — Hacking on Fedora Silverblue

with 7 comments

rpm-ostree-flatpak-silverblue

Fedora Silverblue is a modern and graphical operating system targeted at laptops, tablets and desktop computers. It is the next-generation Fedora Workstation that promises painless upgrades, clear separation between the OS and applications, and secure and cross-platform applications. The basic operating system is an immutable OSTree image, and all the applications are Flatpaks.

It’s great!

However, if you are a hacker and decide to set up a development environment, you immediately run into the immutable OS image and the absence of dnf. You can’t install your favourite tools, editors and SDKs the way you’d normally do on Fedora Workstation. You can either unlock your immutable OS image to install RPMs through rpm-ostree and give up the benefit of painless upgrades; or create a Docker container to get an RPM-based toolbox but be prepared to mess around with root permissions and having to figure out why your SSH agent or display server isn’t working.

Enter Fedora Toolbox.

It makes it trivial to get a mutable development environment on Silverblue:

[rishi@bollard ~]$ fedora-toolbox create
[rishi@bollard ~]$ fedora-toolbox enter
🔹[rishi@toolbox ~]$

It uses OCI containers underneath, but takes away the cognitive overhead of thinking about containers by providing a seamless integration with the host environment. It uses rootless podman and buildah, so there’s no root in the picture either.

fedora-toolbox

If you are going to try it out, make sure that you have the runc-1.0.0-56.dev.git78ef28e package in your Silverblue image. There’s also an ongoing review to get fedora-toolbox added to Fedora. If you don’t feel comfortable mucking around with rpm-ostree on the command-line, then fear not. Very soon all the necessary pieces will be part of the OS image, making it that much easier to start hacking on your Silverblue.

Written by Debarshi Ray

22 October, 2018 at 20:07