Archive for the ‘GTK+’ Category
GNOME Photos: an overview of zooming
I was recently asked about how zooming works in GNOME Photos, and given that I spent an inordinate amount of time getting the details right, I thought I should write it down. Feel free to read and comment, or you can also happily ignore it.
Smooth zooming
One thing that I really wanted from the beginning was smooth zooming. When the user clicks one of the zoom buttons or presses a keyboard shortcut, the displayed image should smoothly flow in and out instead of jumping to the final zoom level — similar to the way the image smoothly shrinks in to make way for the palette when editing, and expands outwords once done. See this animated mock-up from Jimmac to get an idea.
For the zooming to be smooth, we need to generate a number of intermediate zoom levels to fill out the frames in the animation. We have to dish out something in the ballpark of sixty different levels every second to be perceived as smooth because that’s the rate at which most displays refresh their screens. This would have been easier with the 5 to 20 megapixel images generated by smart-phones and consumer-grade digital SLRs; but just because we want things to be sleek, it doesn’t mean we want to limit ourselves to the ordinary! There is high-end equipment out there producing images in excess of a hundred megapixels and we want to robustly handle those too.
Downscaling by large factors is tricky. When we are aiming to generate sixty frames per second, there’s less than 16.67 milliseconds for each intermediate zoom level. All we need is a slightly big zoom factor that stresses the CPU and main memory just enough to exceed our budget and break the animation. It’s a lot more likely to happen than a pathological case that crashes the process or brings the system to a halt.
Mipmaps to the rescue!
Photos uses GEGL to deal with images, and image pixels are held in GeglBuffers. Each GeglBuffer implicitly supports 8 mipmap levels. In other words, a GeglBuffer not only has the image pixels at the original resolution, or level zero, at which they were fed into the buffer, but it also caches progressively lower resolution representations of it. For example, at 50% or level one, at 25% or level two, and so on.
This means that we never downscale by more than a factor of two during an animation. If we want to zoom an image down to 30%, we take the first mipmap level, which is already cached at 50%, and from there on it’s just another 60% to reach the originally intended zoom target of 30%. Knowing that we won’t ever have to downscale by more than a factor of two in a sensitive code path is a relief.
But that’s still not enough.
It doesn’t take long to realize that the user barely catches a fleeting glimpse of the intermediate zoom levels. So, we cut corners by using the fast but low quality nearest neighbour sampler for those; and only use a higher quality box or bilinear sampler, depending on the specific zoom level, for the final image that the user will actually see.
With this set-up in place, on the Intel Kaby Lake i7 machine used in the above video, it consistently takes less than 10 milliseconds for the intermediate frames, and less than 26 milliseconds for the final high quality frame. On an Intel Sandybridge i7 with a LoDPI display it takes less than 5 and 15 milliseconds respectively, because there are less pixels to pump. On average it’s a lot more faster than these worst case figures. You can measure for yourselves using the GNOME_PHOTOS_DEBUG environment variable.
A lot of the above was enabled by Øyvind Kolås’ work on GEGL. Donate to his fund-raiser if you want to see more of this.
There’s some work to do for the HiDPI case, but it’s already fast enough to be perceived as smooth by a human. Look at the PhotosImageView widget if you are further interested.
An elastic zoom gesture
While GTK already comes with a gesture for recognizing pinch-to-zoom, it doesn’t exactly match the way we handle keyboard, mouse and touch pad events for zooming. Specifically, I wanted the image to snap back to its best fit size if the user tried to downscale beyond it using a touch screen. You can’t do that with any other input device, so it makes sense that it shouldn’t be possible with a touch screen either. The rationale being that Photos is optimized for photographic content, which are best viewed at their best fit or natural sizes.
For this elastic behaviour to work, the semantics of how GtkGestureZoom calculates the zoom delta had to be reworked. Every time the direction of the fingers changed, the reference separation between the touch points relative to which the delta is computed must be reset to the current distance between them. Otherwise, if the fingers change direction after having moved past the snapping point, the image will abruptly jump instead of sticking to the fingers.
The image refuses to become smaller than the best fit zoom level and snaps back. Note that simultaneously encoding the screencast consumes enough extra resources to make it stutter a bit. That’s not the case otherwise.
With some help from Carlos Garnacho, we have a custom gesture that hooks into GtkGestureZoom’s begin and update signals to implement the above. The custom gesture is slightly awkward because GtkGestureZoom is a final class and can’t be derived, but it’s not too bad for a prototype. It’s called PhotosGestureZoom, in case you want to look it up.
The screencasts feature a 112.5 megapixel or 12500×9000 photo of hot air balloons at ClovisFest taken by Soulmates Photography / Daniel Street available under the Creative Commons Attribution-Share Alike 3.0 Unported license.
The touch points were recorded in an X session with a tool written by Carlos Garnacho.
GNOME Terminal: separate menu items for opening tabs and windows
Astute users might have noticed that the GNOME Terminal binary distributed by Fedora has separate menu items for opening new tabs and windows, while the vanilla version available from GNOME doesn’t.
With separate menu items
This has been the case since Fedora 25 and was achieved by a downstream patch that reverted two upstream commits.
Without separate menu items
I am happy to say that since version 3.28 GNOME Terminal has regained the ability to have separate menu items as a compile time option. The gnome-terminal-server binary needs to be built with the DISUNIFY_NEW_TERMINAL_SECTION pre-processor macro defined. Here’s one way to do so.
GNOME Terminal 3.28.x lands in Fedora
The following screenshots don’t have the correct colours. Their colour channels got inverted because of this bug.
Brave testers of pre-release Fedora builds might have noticed the absence of updates to GNOME Terminal and VTE during the Fedora 28 development cycle. That’s no longer the case. Kalev submitted gnome-terminal-3.28.1 as part of the larger GNOME 3.28.1 mega-update, and it will make its way into the repositories in time for the Fedora 28 release early next month.
The recent lull in the default Fedora Workstation terminal was not due to the lack of development effort, though. The recent GNOME 3.28 release had a relatively large number of changes in both GNOME Terminal and VTE, and it took some time to update the Fedora-specific patches to work with the new upstream version.
Here are some highlights from the past six months.
Unified preferences dialog
The global and profile preferences were merged into a single preferences dialog. I am very fond of this unified dialog because I have a hard time remembering whether a setting is global or not.
Text settings
The profile-specific settings UI has seen some changes. The bulk of these are in the “Text” tab, which used to be known as “General” in the past.
It’s now possible to adjust the vertical and horizontal spacing between the characters rendered by the terminal for the benefit of those with visual impairments. The blinking of the cursor can be more easily tweaked because the setting is now exposed in the UI. Some people are distracted by a prominently flashing cursor block in the terminal, but still want their thin cursors to flash elsewhere for the sake of discoverability. This should help with that.
Last but not the least, it’s nice to see the profile ID occupy a less prominent position in the UI.
Colours and bold text
There are some subtle improvements to the foreground colour selection for bold text. As a result, the “allow bold text” setting has been deprecated and replaced with “show bold text in bright colors” in the “Colors” tab. Various inconsistencies in the Tango palette were also resolved.
Port to GAction and GMenu
The most significant non-UI change was the port to GAction and GMenuModel. GNOME Terminal no longer uses the deprecated GtkAction and GtkUIManager classes.
Blinking text
VTE now supports blinking text. Try this:
$ tput blink; echo "blinking text"; tput sgr0
If you don’t like it, then there’s a setting to turn it off.
Overline and undercurl
Similar to underline and strikethrough, VTE now supports overline and undercurl. These can be interesting for spell checkers and software development tools.
Emojis in VTE
It’s been one of those weeks when gnome-terminal and vte keep stumbling on some really weird edge cases, so it was a happy moment when I saw this on Fedora 27 Workstation.
GNOME Photos: an overview of thumbnailing
From time to time, I find myself being asked about various details about how content is thumbnailed in GNOME Photos, and the reasons behind various implementation decisions. I can never remember all the details, and always have to dig through Git history and bug reports across multiple modules to come up with an answer. I am hoping that this brain dump will be more persistent than my memory, and more holistic than random comments here and there.
Feel free to read and comment, or you can also happily ignore it.
Background
Having accurate and quality thumbnails is absolutely crucial for Photos. The main user interface is a grid of thumbnails. By design, it tries hard not to expose the filesystem, which means that the user doesn’t have the path or directory hierarchy to complement the contents of the grid. In comparison, thumbnails can be optional in a file manager. Note how Files has settings to disable thumbnailing, and defaults to not thumbnailing remote content, but users can still go about interacting with their files.
Thumbnailing in GNOME is spread across GIO, GVfs, GnomeDesktopThumbnailFactory, and together they implement the Thumbnail Managing Standard. Usually, one uses GIO to lookup thumbnails from the cache and the state they are in, while GnomeDesktopThumbnailFactory is used to create and store the thumbnail files. These thumbnails are stored in the global thumbnail cache in $XDG_CACHE_HOME/thumbnails, and are often, but not necessarily, created by the thumbnailers listed under /usr/share/thumbnailers. This is how most components (eg., GTK+’s GtkFileChooserWidget), and applications (eg., Files and Videos) show thumbnails.
Then there are those “odd” ones that have their own custom setup.
Prior to version 3.24, Photos entirely relied on the global cache and the aforementioned GNOME APIs for its thumbnails. That changed in 3.24 when it switched to its own custom thumbnailer and application specific cache.
Requirements
Ever since editing was added in 3.20, we felt the need to ensure that the thumbnail represents the current state of each item. Being a non-destructive editor, Photos never modifies the original file but separately serializes the edits to disk. The image is rendered by loading the original file, deserializing the edits into objects in memory and running the pixels through them [1]. Therefore, to have the thumbnails accurately represent the current state of the item, it would have to do something similar. However, the edits are application-specific [2], so it is not reasonable to expect the generic OS-wide thumbnailers to be able to handle them.
I believe this is a requirement that all non-destructive image editors have [3]. Notable examples are Darktable and Shotwell.
Secondly, it is important to be able to create and lookup thumbnails of a specific size, as opposed to enumerated constants with pre-determined presets.
The standard specifies two sizes – normal, which is 128×128, and large, which is 256×256. I think this was alright in a world without HiPPI, and is also fine if the thumbnails are either too small or are not an existential necessity for the application. For a HiPPI display with a scaling factor of N, we want to make the thumbnail grid as visually appealing as possible by pumping in NxN times more pixels. Since Photos wants the thumbnails to be 256×256 logical pixels, they should be 256Nx256N raw device pixels on HiPPI. To make things complicated, the cache might get used across different scaling factors – either display or disk got switched, multi-monitor with different resolutions, etc..
Upscaling the low-resolution counterpart of a thumbnail by N is still passable, but it looks much worse if the thumbnail is significantly smaller. Although, I must note that this was the easiest hurdle to surmount. It originates from GIO’s desire to fallback to 128×128 thumbnails, even if the application asked for 256×256. This is pretty straightforward to fix, if necessary.
Last but not the least, I find it important to version the cache to tide over bugs in the thumbnailer. If the cache isn’t versioned, then it is difficult to discard thumbnails that might have been generated by a broken thumbnailer. Hopefully, such bugs would be rare enough that it won’t be necessary to invalidate the cache very often, but when they do happen, it is very reassuring to be able to bump the version, and be guaranteed that users won’t be looking at a broken user interface.
Solution
Starting from version 3.24, Photos uses its own out-of-process thumbnailer and cache [4]. The cache is at $XDG_CACHE_HOME/gnome-photos/thumbnails/$SIZE-$GENERATION, where SIZE is the thumbnail size in raw device pixels and GENERATION is the cache’s version. The main application talks to the thumbnailer over peer-to-peer D-Bus and a simple, cancellable private D-Bus API.
The thumbnailer isn’t separately sandboxed, though. It might be an interesting thing to look at for those who don’t use Flatpak, or to restrict it even more than the main application when running inside Flatpak’s sandbox.
Known bugs
Photos’ thumbnailing code can be traced back to its origins in GNOME Documents. They don’t persistently track thumbnailing failures, and will attempt to re-thumbnail an item that had previously failed when any metadata change is detected. In short, they don’t use G_FILE_ATTRIBUTE_THUMBNAILING_FAILED. The current behaviour might help to overcome a temporary glitch in the network, or it can be simply wasteful.
They predate the addition of G_FILE_ATTRIBUTE_THUMBNAIL_IS_VALID and don’t update the thumbnail once an item gets updated. This could have still been done using GnomeDesktopThumbnailFactory, but that’s water under the bridge, and should possibly be fixed. Although, images don’t tend to get updated so often, which is probably why nobody notices it.
Related to the above point, currently the modification time of the original doesn’t get stored in the thumbnail. It slipped through the cracks while I was reading the sources of the various modules involved in creating thumbnails in GNOME. However, a versioned cache makes it possible to fix it.
[1] If you are reading between the lines, then you might be thinking that it is serializing and deserializing GeglOperations, and you’d be right.
[2] GEGL might be a generic image processing library with its set of built-in operations, but for various reasons, an application can end up carrying its own custom operations.
[3] The idea of an application storing its edits separately from the original can strike as unusual, but this is how most modern image editors work.
[4] Both Darktable and Shotwell have similar thumbnailing infrastructure. You can read about them here and here respectively.
GtkBuilder, Vala and WebKit
This article is about a set of bugs that used to exist, and are in various stages of getting fixed. As such this is merely a historical anecdote. I had planned to write about something entirely different, and hopefully more useful, but I ended up having too much fun with this to just ignore it.
To use a WebKitWebView inside a GTK+ template, one needs to workaround the fact that WebKitWebView breaks the heuristics in GtkBuilder to guess the GType from the human readable type name. That’s easy. Anybody who has used GObject is likely to have encountered some dialect of g_type_ensure, or, as the more learned will point out, GtkBuilder has a type-func attribute for cases like these.
The fun begins when you start debating which workaround to use.
It turns out that type-func doesn’t work with Vala.
A few buglets in GtkBuilder means that if you use class and type-func together, the latter will be ignored. It’s likely nobody used them together because even if class is specified as mandatory the parser doesn’t enforce that. On the other hand, the Vala compiler effectively treats class as mandatory because it doesn’t understand type-func. So, you must use both to avoid a build failure, but if you do, you get a run-time failure because your type-func is ignored.
So, typeof (WebKit.WebView) wins, which is Vala’s equivalent of g_type_ensure.
I don’t know how things are with other language bindings. Vala is what I happen to be using right now, so that’s where I chose to focus.
Thanks to Saiful, for pointing out the problem with WebKitWebView and GtkBuilder. It was fascinating.
GdMainBox — the new content-view widget in libgd
Now that I have written at length about the new fluid overview grids in GNOME Photos, it is time to talk a bit about the underlying widgets doing the heavy lifting. Hopefully some of my fellow GNOME developers will find this interesting.
Background
Ever since its incubation inside Documents, libgd has had a widget called GdMainView. It is the one which shows the grid or list of items in the new GNOME applications — Boxes, Photos, Videos, etc.. It is where drag-n-drop, rubber band selection and the selection mode pattern are implemented.
However, as an application developer, I think its greatest value is in making it trivial to switch the main content view from a grid to a list and back. No need to worry about the differences in how the data will be modelled or rendered. No need to worry about all the dozens of little details that arise when the main UI of an application is switched like that. For example, this is all that the JavaScript code in Documents does:
let view = new Gd.MainView({ shadow_type: Gtk.ShadowType.NONE });
…
view.view_type = Gd.MainViewType.LIST; // use a list
…
view.view_type = Gd.MainViewType.ICON; // use a grid
Unfortunately, GdMainView is based on GtkIconView and GtkTreeView. By this time we all know that GtkIconView has various performance and visual problems. While GtkTreeView might not be slow, the fact that it uses an entirely separate class of visual elements that are not GtkWidgets limits what one can render using it. That’s where GdMainBox comes in.
GdMainBox
GdMainBox is a replacement for GdMainView that is meant to use GtkFlowBox and GtkListBox instead.
GListModel *model;
GtkWidget *view;
model = /* a GListModel containing GdMainBoxItems */
view = gd_main_box_new (GD_MAIN_BOX_ICON);
gd_main_box_set_model (GD_MAIN_BOX (view), model);
g_signal_connect (view,
"item-activated",
G_CALLBACK (item_activated_cb),
data);
g_signal_connect (view,
"selection-mode-request",
G_CALLBACK (selection_mode_request_cb),
data);
g_signal_connect (view,
"selection-changed", /* not view-selection-changed */
G_CALLBACK (selection_changed_cb),
data);
If you are familiar with with old GdMainView widget, you will notice the striking similarity with it. Except one thing. The data model.
GdMainView expected applications to offer a GtkTreeModel with a certain number of columns arranged in a certain order with certain type of values in them. Nothing surprising since both GtkIconView and GtkTreeView rely on the existence of a GtkTreeModel.
In the world of GtkListBoxes and GtkFlowBoxes, the data model is GListModel, a list-like collection of GObjects [*]. Therefore, instead of columns in a table, they need objects with certain properties, and methods to access them. These are codified in the GdMainBoxItem interface which every rendered object needs to implement. You can look at this commit for an example. A nice side-effect is that an interface is inherently more type-safe than a GtkTreeModel whose expected layout is expressed as enumerated types. The compiler can not assert that a certain column does have the expected data type, so it left us vulnerable to bugs caused by inadvertent changes to either libgd or an application.
But why a new widget?
You can definitely use a GtkFlowBox or GtkListBox directly in an application, if that’s what you prefer. However, the vanilla GTK+ widgets don’t offer all the necessary features. I think there is value in consolidating the implementation of those features in a single place that can be shared across modules. It serves as a staging area for prototyping those features in a reasonably generic way so that they can eventually be moved to GTK+ itself. If nothing else, I didn’t want to duplicate the same code across the two applications that I am responsible for — Documents and Photos.
One particularly hairy thing that I encountered was the difference between how selections are handled by the stock GtkFlowBox and the intended behaviour of the content-view. Other niceties on offer are expanding thumbnails, selection mode, and drag-n-drop.
If you do decide to directly use the GTK+ widgets, then I would suggest that you at least use the same CSS style classes as GdMainBox — “content-view” for the entire view and “tile” for each child.
The future
I mentioned changing lists to grids and vice versa. Currently, GdMainBox only offers a grid of icons because Photos is the only user and it doesn’t offer a list view. That’s going to change when I port Documents to it. When that happens, changing the view is going to be just as easy as it used to be.
gd_main_view_set_view_type (GD_MAIN_BOX (view), GD_MAIN_BOX_LIST);
[*] Yes, it’s possible to use them without a model, but having a GListModel affords important future performance optimizations, so we will ignore that possibility.
GNOME Photos 3.24.0
After exploring new territory with sharing and non-destructive editing over the last two releases, it was time for some introspection. We looked at some of the long-standing problems within our existing feature set and tried to iron out a few of them.
Overview Grids
It was high time that we overhauled our old GtkIconView-based overview grids. Their inability to reflow the thumbnails leads to a an ugly vertical gutter of empty space unless the window is just the right size. The other problem was performance. GtkIconView gets extremely slow when the icons are updated, which usually happens when content is detected for the first time and start getting thumbnailed.
Fixing this has been a recurrent theme in Photos since the middle of the previous development cycle. The end goal was to use a GtkFlowBox-based grid, but it involved a lot more work than replacing one user interface component with another.
Too many things relied on the existence of a GtkTreeModel, and had to be ported to our custom GListModel implementation before we could achieve any user-visible improvement. Once all those yaks had been shaved, we finally started working on the widget at the Core Apps Hackfest last year.
Anyway, I am happy that all that effort has to come fruition now.
Thumbnails
Closely related to our overview grids are the thumbnails inside them. Photos has perpetually suffered from GIO’s inability to let an application specifically request a high resolution thumbnail. While that is definitely a fixable problem, the fact that we store our edits non-destructively as serialized GEGL graphs makes it very hard to use the desktop-wide infrastructure for thumbnails. One cannot expect a generic thumbnailer to interpret the edits and apply them to the original image because their representation will vary greatly from one application to another. That led to the other problem where the thumbnails wouldn’t reflect the edited state of an image.
Therefore, starting from version 3.24.0, Photos has its own out-of-process thumbnailer and a separate thumbnail cache. They ensure that the thumbnails are of a suitably high resolution, and the edited state of an image is never ignored.
Exposure and Blacks
Personally, I have been a heavy user of Darktable’s exposure and blacks adjustment tool, and I really missed something like that in GNOME Photos. Ultimately, at this year’s WilberWeek I fixed gegl:exposure to imitate its Darktable counterpart, and exposed it as a tool in Photos. I am happy with the outcome and I have so far enjoyed dogfooding this little addition.
Core Apps Hackfest 2016: report
I spent last weekend at the Core Apps Hackfest in Berlin. The agenda was to work on GNOME’s core applications: Documents, Files, Music, Photos, Videos, Usage, etc.; to raise their overall standard and to make them push beyond the limits of the framework. There were 19 of us and among us we covered a wide range of modules and areas of expertise.
I spent most of my time on the plumbing necessary for Documents and Photos to use GtkFlowBox and GtkListBox. The innards of Photos had already been overhauled to reduce its dependency on GtkTreeModel. Going into the hackfest we were sorely lacking a widget that had all the bells and whistles we need — the idiomatic GNOME 3 selection mode, and seamlessly switching between a list and grid view. So, this is where I decided to focus my energy. As a result, we now have a work-in-progress GdMainBox widget in libgd to replace the old GtkIconView/GtkTreeView-based GdMainView.
In fact, GtkListBox and GtkFlowBox was a recurring theme at the hackfest. Carlos Soriano and Georges were working on using them in Files, and whenever anybody uses them in a non-trivial manner there is the inevitable discussion about performance. Good thing that Benjamin was around. He spent the better part of a tram ride and more than an hour at the whiteboard, sketching out a strategy to make GtkListBox more efficient than it is today.
Like last year, Øyvind joined us. We talked about GEGL, and I finally saw the new GIMP in action. I rarely use GIMP, and I am not sure I have ever built it from source, but I have been reading its sources on a semi-regular basis for almost a year now. It was good to finally address this aberration. Øyvind had with him a cheap hand-held DLNA media renderer that was getting stuck when trying to render more than one image with dleyna-render and Photos. Zeeshan helped me poke at it, but unfortunately we didn’t get anywhere.
Other than that, Petr Stetka impressed everyone with his progress on the new Usage application. Georges refreshed his patches to implement the new Online Accounts Settings panel, Carlos Garnacho helped me with GtkGesture, and I reviewed various patches and branches that had been on my list for a while.
Many thanks to Red Hat for sponsoring me; to Carlos Soriano and Joaquim for organizing the event; to Kinvolk for being such gracious hosts; and to Collabora for the nice dinner.
Debarshi's den 
You must be logged in to post a comment.