New GNOME API key for Google services

Tl;dr: update evolution-data-server to stop your client from misbehaving; update gnome-online-accounts to shield yourself from other buggy clients.

Recently, a few bugs in evolution-data-server were causing various GNOME components to hit Google’s daily limit for their CalDAV and Tasks APIs. At least evolution, gnome-calendar and gnome-todo were affected. The bugs have since been fixed, but until every single user out there installs the fix, everybody will be susceptible even if they have a fixed copy of evolution-data-server. This is because Google identifies the clients by the OAuth 2.0 API key used to access their services, and not the version of the code running on them.

We are therefore phasing out the old Google API key used by GNOME so that users updating their systems will have no connection to the one that was tainted by these bugs.

Fedora Users

If you are using Fedora 25, make sure you have evolution-data-server-3.22.3-1.fc25 and gnome-online-accounts-3.22.3-1.fc25. For Fedora 24, the versions are evolution-data-server-3.20.6-1.fc24 and gnome-online-accounts-3.20.5-1.fc24.

GNOME Distributors

If you are distributing GNOME, make sure that you are shipping evolution-data-server-3.22.3 and gnome-online-accounts-3.22.3, or evolution-data-server-3.20.6 and gnome-online-accounts-3.20.5.

Debugging GNOME Online Accounts

I spent some time today documenting how to debug various problems with online account integration in GNOME. It is also linked from the main GNOME Online Accounts wiki page. So, you can find it via the usual click-stream and don’t need to rely on this blog.

It is still basic. I want to expand it further to cover how to debug specific integration points. For example, issues with mounting Google Drive in Nautilus, or some breakage involving Kerberos. Anyway, it’s a start and I am quite happy to have one less thing on my TODO list. Sometimes, things in GNOME seem like a maze of hidden D-Bus daemons, known only to the initiated. Hopefully, this will make it more approachable.

Oh, and don’t shy away from editing it. It’s a Wiki!

Google Drive and GNOME — what is a volatile path?

TL;DR: If you are using any GFile API that can create a new file or directory, then please take care of “volatile” paths. Look at standard::is-volatile, g_file_output_stream_query_info and g_file_query_info. Read further if you don’t trust me, or if you develop file management software.

Like other cloud storages, Drive is database-based. Every file or folder is identified by an opaque, server generated, immutable, blob (say “0B9J_DkziBDRFTj1TRWLPam9kbnc”), and has a human-readable title that can be displayed in the user interface (say “Summer Vacation”). This is unlike POSIX filesystems where a file is identified by its path and, barring encoding issues, the basename of that path is its name. ie., the name of the file “/home/alice/Summer\ Vacation” is “Summer Vacation”. Sounds like an innocuous distinction at first, but this is at the heart of the issue.

Let’s look at a few operations to get a feel for this.

Rename

POSIX:

mv /home/alice/foo /home/alice/bar

We need the paths to carry out the operation and once successfully finished, the file’s path changes. Since files are identified by their paths, the operation would fail if there existed another file called “bar” in the same location.

Google Drive:

set_title (id_of_foo, bar)

We only need the identifier (or ID) and the new title for the operation and once successfully finished, the title changes but the ID remains the same. Since IDs are unique and immutable, you can have two files with the same title in the same location. That’s a bit strange, you might think, but not so much.

Creating a New File

POSIX:

touch /home/alice/foo

If you want a file named “foo” in a certain location, barring encoding issues, you ask for a path that has “foo” appended to the path of the location. It will work as long as there isn’t another “foo” in the same location.

Google Drive:

create_file (id_of_parent_folder, foo) → id_of_new_file

Notice that we cannot specify the ID of the new file. We provide a title and the server generates the ID for it. Again, there is no restriction on having two files with the same title in the same location. This is where it starts getting weird.

GIO and GVfs

The file handling APIs in GIO are based on the POSIX model, with some room to accommodate minor diversions. They are ill-equipped to deal with a case like this.

As a quick aside, both MTP and PTP are somewhat similar in this regard. Both use IDs to refer to objects (think of them as files) stored on the phone or camera, and we have backends implementing them in GVfs. These backends construct POSIX-like paths using the human-readable titles of the files, and have an internal cache mapping IDs to paths and vice-versa. Easy enough, but there are two important differences:

• MTP allows no parallelism. You can only perform one operation at a time — a new one won’t start until the earlier has finished. So, the GVfs backend can assume complete control over the storage. Google Drive, on the other hand, is massively parallel. You might be modifying it via GVfs, via a web browser, from a different computer, or someone might be sharing something to you from yet another computer. It is better to rely on the server generated IDs to identify files as much as possible, and let the server worry about keeping the data consistent.
• MTP and PTP are usually backed by a traditional filesystem on the device. eg., FAT12, FAT16, FAT32, exFAT or ext3. Therefore the issue of duplicate titles doesn’t arise.

So, how do we deal with this?

Assuming that we are only doing read-only operations, the URIs used by GVfs’ Google Drive backend look like:

google-drive://foo@gmail.com/0B9J_DkziBDRFTj1TPam9kbnc/0B9J_GlyfBDRBHj1TRWL9khrc

The backend has its own scheme, the account is mentioned, and the path is made up of identifiers. Quite easy. The server doesn’t care about having a POSIX-like path, but we create one to fit into GIO’s POSIX-like world view.

These IDs are decidedly ugly and unreadable, so we use the standard::display-name attribute to tag them with the titles. If you run g_file_query_info on one of these URIs, you will get the ID as the standard::name and the title as the standard::display-name. Thus, the user won’t encounter any ugliness unless she deliberately went looking for it.

It gets complicated when we go beyond read-only access.

Let’s look at creating new files again. Since the GIO and GVfs APIs are designed around the POSIX model, a GVfs backend receives the request as:

create ("/id1/id2/some-title")


Even though the application thinks that it has created a new file named “some-title” inside “/id1/id2” that can be accessed by the path “/id1/id2/some-title”, the new file’s identifier is not “some-title”. It is whatever the server generated for us, say “id3”. If the application has to carry on the illusion of accessing the file as “/id1/id2/some-title”, then the backend has to map this “fake” path back to “/id1/id2/id3”; and this has to happen somewhat recursively because if we are copying a directory, then multiple elements in the file path can be “fake”. eg., “/id1/id2/title-of-subdir/title-of-file”, and so on.

We call these volatile paths. These are identified by the standard::is-volatile attribute and you can use standard::symlink-target to find the actual ID that it maps to. It is recommended that applications map volatile paths to the real IDs as soon as possible because the mapping can break if a parallel operation changes the title of the file.

In the specific case where you have created a new file and writing to it through a GFileOutputStream, then it is better to use g_file_output_stream_query_info instead of g_file_query_info. The output stream knows exactly which file it is writing to, and doesn’t have to do the mapping based on the volatile path, which could fail if the contents of the Drive changed out of band.

For working examples, see Nautilus.

Google Drive and GNOME: 6 years later

GNOME 3.18 is going to be another milestone in our journey to bring various online services closer to the desktop, but this is one that took us 6 years to reach. You can now access your Google Drive through your favourite GNOME applications and the usual GIO APIs

Thibault Saunier started working on this way back in 2009 as a Google Summer of Code project. I picked it up a year ago, and after a few hiccups along the way and some head scratching — more on that later — it is finally here for you to enjoy. You can either wait for distributions like Fedora Workstation 23 to ship GNOME 3.18, or try the 3.17.92 release candidate tarballs that are coming out next week.

Thanks to Alexander Larsson, Philip Withnall and Ondrej Holy for the endless discussions, code reviews and testing for the last one year.

Documents and Photos: a content application update

We have continued our work towards having a nice set of core applications for finding and selecting the user’s content. Documents, which is the eldest, received a round of thorough bug squashing for GNOME 3.12. It is a much more mature and well-behaved application now than it was six months ago, mainly due to the work of our QA teams – both upstream and downstream. Photos received a set of nice new features and is well on its way towards fulfilling the role it was designed for.

Facebook

Álvaro Peña added support for browsing Facebook photos based on his libgfbgraph library.

Search

This was something that was sorely lacking in the previous releases. With the increase in the number of online sources, the inability to be able to filter the content based on some parameters was badly felt. But not any more.

User help

The documentation team has paid a lot of attention to the end-user help for these two applications. Older pages have been refreshed and newer ones written, and I can easily say that GNOME 3.12 will be one of the best in terms of documentation coverage.

Flickring backgrounds

Flickr integration was introduced in GNOME 3.10 when we added support for browsing Flickr content in Photos. In GNOME 3.12 we have taken this one step ahead. The Background Settings panel will now let you choose one of your Flickr pictures as your desktop’s background.

The panel only shows the 50 most recent pictures from each account to avoid overloading it with hundreds of items. If you are looking for that gorgeous photograph that you took a few summers ago, then you might want to use Photos instead. It has better search capability and is meant to handle larger data sets.

GVfs and the GOA volume monitor

Sometime ago I had written about how we integrated ownCloud in GNOME for 3.8. One interesting infrastructural enhancement that was part of this feature is the GOA volume monitor in GVfs.

Volume monitors

A volume monitor is a D-Bus session service that uses the org.gtk.Private namespace (eg., org.gtk.Private.MTPVolumeMonitor, org.gtk.Private.UDisks2VolumeMonitor, etc.), has an entry in /usr/share/gvfs/remote-volume-monitors/, and exposes your disk partitions, iOS devices, MTP or PTP devices as volumes. These volumes eventually show up on the GTK+ places sidebar.

Online accounts

We wrote a volume monitor which exposes your online file storage as a volume. It listens to GNOME Online Accounts and when it notices an account that implements the GoaFiles interface it exposes it as a volume.

Giovanni Campagna extended GVfs’ shadow mount machinery to make it transparently pick up the credentials for accessing the cloud storage from GOA without bothering the user.

The future

Currently this only works for ownCloud because the underlying protocol is WebDAV – something that has a corresponding backend in GVfs.

To unlock support for, say, Google Drive or SkyDrive we would first need to implement the protocol as a GVfs backend, and then add the GoaFiles interface to the corresponding GoaObject.

GNOME Online Accounts and IMAP / SMTP

Starting from GNOME 3.7.91, you can configure your IMAP/SMTP accounts via GNOME’s single sign-on framework and then use them from Evolution.

Many thanks to Matthew Barnes for making the required adjustments in Evolution Data Server and making sure that it landed in time for the 3.8.0 release later this month.

GNOME meets ownCloud

Now that we are in the middle of the various freezes in preparation for GNOME 3.8, it is a good time to talk about some of the things that happened over the last six months. One such thing is the integration of ownCloud in GNOME. Go to the Online Accounts panel in Settings (hit the super key or the Activities button and type “online”) to enable your ownCloud account, and you will be able to access your calendars, contacts and files from GNOME.

Calendars and contacts can be accessed via CalDAV and CardDAV from their respective tabs in Evolution. Files in your ownCloud storage show up as volumes via WebDAV in Nautilus and GtkFileChooser.

Many thanks to Milan Crha for making the necessary changes to hook up Evolution with ownCloud in time for 3.8.0.

Setting up an ownCloud instance on OpenShift

Ever since portknox.net suspended their gratis offering and sereinity (from #fedora-devel) shut down his server, I had run out of instances that I could use for working on bringing ownCloud closer to GNOME.

Few days ago, Michael Scherer pointed me to ownCloud-OpenShift kickstart, and it is really as easy as the README.md claims to be.

Thanks to it I have now got myself owncloud-lostcase.rhcloud.com.