Debarshi's den

Archive for the ‘Photos’ Category

Libre Graphics Meeting 2018

with 2 comments

I spent the last seven days attending Libre Graphics Meeting in sunny and beautiful Seville. This was my second LGM, the first being six years ago in Vienna, so it was refreshing to be back. I stayed in one of the GIMP apartments near the Alameda de Hércules garden square. Being right in the middle of the city meant that everything of interest was either within walking distance or a short bus ride away.

IMG_20180425_140117284

Unlike other conferences that I have been to, LGM 2018 started at six o’clock in the evening. That was good because one didn’t have to worry about waking up in time not to miss the opening keynote; and you haven’t attended LGM if you haven’t been to the State of Libre Graphics. Other than that I went to Øyvind’s presentation on colour; saw Nara describe her last ten years with Estúdio Gunga; and listened to Dave Crossland and Nathan Willis talk about fonts. There was a lot of live coding based music and algorave going on this year. My favourite was Neil C. Smith’s performance using Praxis LIVE.

IMG_20180425_155837676.jpg

All said and done, the highlight of this LGM had to be the GIMP 2.10.0 release at the beginning of the conference. GEGL 0.4.0 was also rolled out to celebrate the occasion. Much happiness and rejoicing ensued.

I spent my time at LGM alternating between delicious tapas, strolling down the narrow and colourful alleys of Seville, sight-seeing, and hacking on GEGL. I started sketching out a proper codec API for GeglBuffer modelled on GdkPixbuf, and continued to performance tune babl, but those are topics for later blog posts.

IMG_20180430_140438859~2

Advertisements

Written by Debarshi Ray

30 April, 2018 at 22:33

GNOME Photos: an overview of thumbnailing

with 2 comments

From time to time, I find myself being asked about various details about how content is thumbnailed in GNOME Photos, and the reasons behind various implementation decisions. I can never remember all the details, and always have to dig through Git history and bug reports across multiple modules to come up with an answer. I am hoping that this brain dump will be more persistent than my memory, and more holistic than random comments here and there.

Feel free to read and comment, or you can also happily ignore it.

Background

Having accurate and quality thumbnails is absolutely crucial for Photos. The main user interface is a grid of thumbnails. By design, it tries hard not to expose the filesystem, which means that the user doesn’t have the path or directory hierarchy to complement the contents of the grid. In comparison, thumbnails can be optional in a file manager. Note how Files has settings to disable thumbnailing, and defaults to not thumbnailing remote content, but users can still go about interacting with their files.

Thumbnailing in GNOME is spread across GIO, GVfs, GnomeDesktopThumbnailFactory, and together they implement the Thumbnail Managing Standard. Usually, one uses GIO to lookup thumbnails from the cache and the state they are in, while GnomeDesktopThumbnailFactory is used to create and store the thumbnail files. These thumbnails are stored in the global thumbnail cache in $XDG_CACHE_HOME/thumbnails, and are often, but not necessarily, created by the thumbnailers listed under /usr/share/thumbnailers. This is how most components (eg., GTK+’s GtkFileChooserWidget), and applications (eg., Files and Videos) show thumbnails.

Then there are those “odd” ones that have their own custom setup.

Prior to version 3.24, Photos entirely relied on the global cache and the aforementioned GNOME APIs for its thumbnails. That changed in 3.24 when it switched to its own custom thumbnailer and application specific cache.

Requirements

Ever since editing was added in 3.20, we felt the need to ensure that the thumbnail represents the current state of each item. Being a non-destructive editor, Photos never modifies the original file but separately serializes the edits to disk. The image is rendered by loading the original file, deserializing the edits into objects in memory and running the pixels through them [1]. Therefore, to have the thumbnails accurately represent the current state of the item, it would have to do something similar. However, the edits are application-specific [2], so it is not reasonable to expect the generic OS-wide thumbnailers to be able to handle them.

I believe this is a requirement that all non-destructive image editors have [3]. Notable examples are Darktable and Shotwell.

Secondly, it is important to be able to create and lookup thumbnails of a specific size, as opposed to enumerated constants with pre-determined presets.

The standard specifies two sizes – normal, which is 128×128, and large, which is 256×256. I think this was alright in a world without HiPPI, and is also fine if the thumbnails are either too small or are not an existential necessity for the application. For a HiPPI display with a scaling factor of N, we want to make the thumbnail grid as visually appealing as possible by pumping in NxN times more pixels. Since Photos wants the thumbnails to be 256×256 logical pixels, they should be 256Nx256N raw device pixels on HiPPI. To make things complicated, the cache might get used across different scaling factors – either display or disk got switched, multi-monitor with different resolutions, etc..

Upscaling the low-resolution counterpart of a thumbnail by N is still passable, but it looks much worse if the thumbnail is significantly smaller. Although, I must note that this was the easiest hurdle to surmount. It originates from GIO’s desire to fallback to 128×128 thumbnails, even if the application asked for 256×256. This is pretty straightforward to fix, if necessary.

Last but not the least, I find it important to version the cache to tide over bugs in the thumbnailer. If the cache isn’t versioned, then it is difficult to discard thumbnails that might have been generated by a broken thumbnailer. Hopefully, such bugs would be rare enough that it won’t be necessary to invalidate the cache very often, but when they do happen, it is very reassuring to be able to bump the version, and be guaranteed that users won’t be looking at a broken user interface.

Solution

Starting from version 3.24, Photos uses its own out-of-process thumbnailer and cache [4]. The cache is at $XDG_CACHE_HOME/gnome-photos/thumbnails/$SIZE-$GENERATION, where SIZE is the thumbnail size in raw device pixels and GENERATION is the cache’s version. The main application talks to the thumbnailer over peer-to-peer D-Bus and a simple, cancellable private D-Bus API.

The thumbnailer isn’t separately sandboxed, though. It might be an interesting thing to look at for those who don’t use Flatpak, or to restrict it even more than the main application when running inside Flatpak’s sandbox.

Known bugs

Photos’ thumbnailing code can be traced back to its origins in GNOME Documents. They don’t persistently track thumbnailing failures, and will attempt to re-thumbnail an item that had previously failed when any metadata change is detected. In short, they don’t use G_FILE_ATTRIBUTE_THUMBNAILING_FAILED. The current behaviour might help to overcome a temporary glitch in the network, or it can be simply wasteful.

They predate the addition of G_FILE_ATTRIBUTE_THUMBNAIL_IS_VALID and don’t update the thumbnail once an item gets updated. This could have still been done using GnomeDesktopThumbnailFactory, but that’s water under the bridge, and should possibly be fixed. Although, images don’t tend to get updated so often, which is probably why nobody notices it.

Related to the above point, currently the modification time of the original doesn’t get stored in the thumbnail. It slipped through the cracks while I was reading the sources of the various modules involved in creating thumbnails in GNOME. However, a versioned cache makes it possible to fix it.

[1] If you are reading between the lines, then you might be thinking that it is serializing and deserializing GeglOperations, and you’d be right.

[2] GEGL might be a generic image processing library with its set of built-in operations, but for various reasons, an application can end up carrying its own custom operations.

[3] The idea of an application storing its edits separately from the original can strike as unusual, but this is how most modern image editors work.

[4] Both Darktable and Shotwell have similar thumbnailing infrastructure. You can read about them here and here respectively.

Written by Debarshi Ray

29 January, 2018 at 17:17

Posted in C, Fedora, Flatpak, GEGL, GNOME, GTK+, GVfs, Photos

Image wrangling with GEGL: an introduction

with 6 comments

One of the core dependencies of GNOME Photos, other than GTK+ and Tracker, is a library called GEGL. It is a GObject-based image processing library primarily developed for GIMP. GEGL is used by Photos to load pixels from files, create thumbnails, edit, share and export images.

Unfortunately, even though GEGL is a powerful and generic image processing framework, it can be hard to find documentation and code samples to refer to, and the pool of people who understand it well enough is relatively small. I am going to do a series of blog posts to address this by feeding the search engines. Hopefully this will be useful for new contributors to GIMP and GNOME Photos, and some of it can be folded back into the reference GEGL documentation; or maybe it will encourage adoption in new and interesting places.

Nodes and operations

Processing images with GEGL requires the creation of a graph, represented by a GeglNode. A GeglNode can either have a number of child nodes connected to each other forming a directed acyclic graph, or it can have a GeglOperation. An operation is where the actual image processing takes place. Multiple operations are chained together in a graph to obtain the desired outcome.

This is enough to get started with some basic effects and enhancements. Here is a snippet that takes a path to an input image, enhances the blacks and saves it as a PNG.

  #include <gegl.h>
  …
  g_autoptr (GeglNode) graph = NULL;
  GeglNode *exposure;
  GeglNode *load;
  GeglNode *sink;

  graph = gegl_node_new ();
  load = gegl_node_new_child (graph,
                              "operation", "gegl:load",
                              "path", /* input path as C string */,
                              NULL);
  exposure = gegl_node_new_child (graph,
                                  "operation", "gegl:exposure",
                                  "black-level", 0.03,
                                  NULL);
  sink = gegl_node_new_child (graph,
                              "operation", "gegl:png-save",
                              "bitdepth", 8,
                              "path", /* output path as C string */,
                              NULL);

  gegl_node_link_many (load, exposure, sink, NULL);
  gegl_node_process (sink);

Notice the many similarities with GStreamer.

There is a whole list of such filters to choose from. Such as gegl:cartoon to simulate a cartoon drawn with a black felt pen, gegl:mosaic to transform an image into a mosaic, gegl:saturation to change the colourfulness of the image, or gegl:posterize, which is used by the similarly named tool in GIMP.

example-00

Buffers

Image pixels are held in a GeglBuffer. Most applications would directly interact with a GeglBuffer at one point or the other. For example, to decode an image file and carry the pixels around instead of repeatedly decoding them off the storage. In the above code sample, the buffers were implicitly created by GEGL unbeknownst to us, but we can use a similar graph to load pixels off a file into a GeglBuffer.

  #include <gegl.h>
  …
  g_autoptr (GeglBuffer) buffer = NULL;
  g_autoptr (GeglNode) graph = NULL;
  GeglNode *load;
  GeglNode *sink;

  graph = gegl_node_new ();
  load = gegl_node_new_child (graph,
                              "operation", "gegl:load",
                              "path", /* input path as C string */,
                              NULL);
  sink = gegl_node_new_child (graph,
                              "operation", "gegl:buffer-sink",
                              "buffer", &buffer,
                              NULL);

  gegl_node_link_many (load, sink, NULL);
  gegl_node_process (sink);

A loaded buffer can be then fed into a graph using a gegl:buffer-source.

As the custodian of pixels, GeglBuffer is similar to the role played by GdkPixbuf, but it has some extra features that are handy for image processing.

Most notably, a GeglBuffer is designed to handle massive images that are larger than the amount of physical RAM available on the system. Instead of holding all the pixels in a linear sequence of bytes, it splits them up into small tiles that can be paged out into a file when not in use. However, if necessary, it is possible to optionally dumb down a GeglBuffer by setting it up to use a single array of bytes, or forcing all tiles to be held in RAM.

A GeglBuffer is not restricted to a single pixel format such as RGB with 8 bits per channel. It can transparently handle a horde of formats — monochrome, Lab, HSL, etc. with different degrees of precision per channel. Finally, it is mipmap-capable.

All these features make GeglBuffer a very sophisticated data structure for storing image pixels. However, they aren’t that important for an introduction to GEGL, so we will save them for a future article.

Happy hacking

This is enough to start playing with GEGL. Here is the code used to create the above image, and is proof that knowing just this much is enough to do practically useful things.

Written by Debarshi Ray

20 November, 2017 at 13:30

Posted in C, GEGL, GIMP, GNOME, Photography, Photos

Stable GNOME Photos Flatpaks moved to Flathub

leave a comment »

Starting from version 3.26, the stable GNOME Photos Flatpaks have been moved to Flathub. They are no longer available from GNOME’s Flatpak repository.

To migrate, first delete the old build:

$ flatpak uninstall org.gnome.Photos/x86_64/stable

Then install it from Flathub:

$ flatpak remote-add --from flathub https://flathub.org/repo/flathub.flatpakrepo
$ flatpak install flathub org.gnome.Photos

Note that this is only about the stable build. The nightly continues to be available from its existing location in GNOME’s repository. You can keep updating it with:

$ flatpak update --user org.gnome.Photos/x86_64/master

Written by Debarshi Ray

10 October, 2017 at 16:20

Posted in Flatpak, GNOME, Photos

GdMainBox — the new content-view widget in libgd

with 3 comments

Now that I have written at length about the new fluid overview grids in GNOME Photos, it is time to talk a bit about the underlying widgets doing the heavy lifting. Hopefully some of my fellow GNOME developers will find this interesting.

Background

Ever since its incubation inside Documents, libgd has had a widget called GdMainView. It is the one which shows the grid or list of items in the new GNOME applications — Boxes, Photos, Videos, etc.. It is where drag-n-drop, rubber band selection and the selection mode pattern are implemented.

However, as an application developer, I think its greatest value is in making it trivial to switch the main content view from a grid to a list and back. No need to worry about the differences in how the data will be modelled or rendered. No need to worry about all the dozens of little details that arise when the main UI of an application is switched like that. For example, this is all that the JavaScript code in Documents does:

  let view = new Gd.MainView({ shadow_type: Gtk.ShadowType.NONE });
  …
  view.view_type = Gd.MainViewType.LIST; // use a list
  …
  view.view_type = Gd.MainViewType.ICON; // use a grid


Unfortunately, GdMainView is based on GtkIconView and GtkTreeView. By this time we all know that GtkIconView has various performance and visual problems. While GtkTreeView might not be slow, the fact that it uses an entirely separate class of visual elements that are not GtkWidgets limits what one can render using it. That’s where GdMainBox comes in.

GdMainBox

GdMainBox is a replacement for GdMainView that is meant to use GtkFlowBox and GtkListBox instead.

GListModel *model;
GtkWidget *view;

model = /* a GListModel containing GdMainBoxItems */
view = gd_main_box_new (GD_MAIN_BOX_ICON);
gd_main_box_set_model (GD_MAIN_BOX (view), model);
g_signal_connect (view,
                  "item-activated",
                  G_CALLBACK (item_activated_cb),
                  data);
g_signal_connect (view,
                  "selection-mode-request",
                  G_CALLBACK (selection_mode_request_cb),
                  data);
g_signal_connect (view,
                  "selection-changed", /* not view-selection-changed */
                  G_CALLBACK (selection_changed_cb),
                  data);


If you are familiar with with old GdMainView widget, you will notice the striking similarity with it. Except one thing. The data model.

GdMainView expected applications to offer a GtkTreeModel with a certain number of columns arranged in a certain order with certain type of values in them. Nothing surprising since both GtkIconView and GtkTreeView rely on the existence of a GtkTreeModel.

In the world of GtkListBoxes and GtkFlowBoxes, the data model is GListModel, a list-like collection of GObjects [*]. Therefore, instead of columns in a table, they need objects with certain properties, and methods to access them. These are codified in the GdMainBoxItem interface which every rendered object needs to implement. You can look at this commit for an example. A nice side-effect is that an interface is inherently more type-safe than a GtkTreeModel whose expected layout is expressed as enumerated types. The compiler can not assert that a certain column does have the expected data type, so it left us vulnerable to bugs caused by inadvertent changes to either libgd or an application.

But why a new widget?

You can definitely use a GtkFlowBox or GtkListBox directly in an application, if that’s what you prefer. However, the vanilla GTK+ widgets don’t offer all the necessary features. I think there is value in consolidating the implementation of those features in a single place that can be shared across modules. It serves as a staging area for prototyping those features in a reasonably generic way so that they can eventually be moved to GTK+ itself. If nothing else, I didn’t want to duplicate the same code across the two applications that I am responsible for — Documents and Photos.

One particularly hairy thing that I encountered was the difference between how selections are handled by the stock GtkFlowBox and the intended behaviour of the content-view. Other niceties on offer are expanding thumbnails, selection mode, and drag-n-drop.

If you do decide to directly use the GTK+ widgets, then I would suggest that you at least use the same CSS style classes as GdMainBox — “content-view” for the entire view and “tile” for each child.

The future

I mentioned changing lists to grids and vice versa. Currently, GdMainBox only offers a grid of icons because Photos is the only user and it doesn’t offer a list view. That’s going to change when I port Documents to it. When that happens, changing the view is going to be just as easy as it used to be.

gd_main_view_set_view_type (GD_MAIN_BOX (view), GD_MAIN_BOX_LIST);



[*] Yes, it’s possible to use them without a model, but having a GListModel affords important future performance optimizations, so we will ignore that possibility.

Written by Debarshi Ray

29 March, 2017 at 00:06

Posted in Blogroll, C, Documents, GNOME, GTK+, Photos

GNOME Photos 3.24.0

with 3 comments

After exploring new territory with sharing and non-destructive editing over the last two releases, it was time for some introspection. We looked at some of the long-standing problems within our existing feature set and tried to iron out a few of them.

Overview Grids

It was high time that we overhauled our old GtkIconView-based overview grids. Their inability to reflow the thumbnails leads to a an ugly vertical gutter of empty space unless the window is just the right size. The other problem was performance. GtkIconView gets extremely slow when the icons are updated, which usually happens when content is detected for the first time and start getting thumbnailed.

gnome-photos-flowbox-1

Fixing this has been a recurrent theme in Photos since the middle of the previous development cycle. The end goal was to use a GtkFlowBox-based grid, but it involved a lot more work than replacing one user interface component with another.
Too many things relied on the existence of a GtkTreeModel, and had to be ported to our custom GListModel implementation before we could achieve any user-visible improvement. Once all those yaks had been shaved, we finally started working on the widget at the Core Apps Hackfest last year.

Anyway, I am happy that all that effort has to come fruition now.

Thumbnails

Closely related to our overview grids are the thumbnails inside them. Photos has perpetually suffered from GIO’s inability to let an application specifically request a high resolution thumbnail. While that is definitely a fixable problem, the fact that we store our edits non-destructively as serialized GEGL graphs makes it very hard to use the desktop-wide infrastructure for thumbnails. One cannot expect a generic thumbnailer to interpret the edits and apply them to the original image because their representation will vary greatly from one application to another. That led to the other problem where the thumbnails wouldn’t reflect the edited state of an image.

Therefore, starting from version 3.24.0, Photos has its own out-of-process thumbnailer and a separate thumbnail cache. They ensure that the thumbnails are of a suitably high resolution, and the edited state of an image is never ignored.

Exposure and Blacks

Personally, I have been a heavy user of Darktable’s exposure and blacks adjustment tool, and I really missed something like that in GNOME Photos. Ultimately, at this year’s WilberWeek I fixed gegl:exposure to imitate its Darktable counterpart, and exposed it as a tool in Photos. I am happy with the outcome and I have so far enjoyed dogfooding this little addition.

Written by Debarshi Ray

21 March, 2017 at 13:13

Posted in GEGL, GNOME, GTK+, Photos

GNOME Photos Flatpaks

with 4 comments

I joined the recent buzz around Flatpak manifests in GNOME, and gave the GNOME Photos builds some routine maintenance. The stable build has been updated to the latest 3.22.x point releases; and the nightly, which I had broken, is again tracking Git master.

To install the stable build:

$ flatpak remote-add --from gnome https://sdk.gnome.org/gnome.flatpakrepo
$ flatpak remote-add --from gnome-apps https://sdk.gnome.org/gnome-apps.flatpakrepo
$ flatpak install gnome-apps org.gnome.Photos

To install the nightly:

$ flatpak remote-add --from gnome-nightly https://sdk.gnome.org/gnome-nightly.flatpakrepo
$ flatpak remote-add --from gnome-apps-nightly https://sdk.gnome.org/gnome-apps-nightly.flatpakrepo
$ flatpak install gnome-apps-nightly org.gnome.Photos

They can be run directly from gnome-shell. However, if you have installed both stable and nightly builds, then you can specifically select one by:

$ flatpak run --branch=stable org.gnome.Photos
$ flatpak run --branch=master org.gnome.Photos

Written by Debarshi Ray

9 March, 2017 at 11:13

Posted in Flatpak, GNOME, Photos