More on synchronization

I started a draft implementation of my identity synchronization idea. In the course of doing this I ran across a minor problem.My initial idea was to keep things very simple: a shared resource would be a collection of bits, and the users of the API would handle all aspects of interpretation. The library would handle bookkeeping details but would defer conflict resolution to the caller — handing back two versions of a file (three-way merge is a possibility, too, but I’m not convinced it is worth the effort) and having the caller respond with a merged version.

Now suppose you log in on your laptop and make some changes while disconnected. When you subsequently reconnect, the ideal would be for all the identity data to automatically resynchronize with the server. This seems like the most useful thing to do — I don’t want to have to actually restart RSSOwl for my saved blog-reading sessions to be available to my other machines.

Unfortunately this throws a wrench into the naive approach outlined above. If the synchronized data is uninterpreted bytes, then any conflict will mess up the idea of generic synchronization without running a particular application to handle a merge.

One way to fix this would be to mandate a file format so that the merge code can be generic. I’m not very fond of this. The other, and I think superior, plan would be to handle conflicts monotone-style: let all commits succeed, and handle merging on demand. In this model we can upload a file (with some ancestry info); when downloading updated files from the server we would simply download all available “head” files and the application would perform multiple merges.

This means having a smart server, but I think that was inevitable anyhow. (Probably Nathaniel is on target, as usual, and I should be reusing monotone’s code for this…)

Another little problem that came up is keyring management. In particular, the gnome keyring will be needed to decrypt data downloaded from the server. But, we want to store the keyring itself as a shared resource. I think the only answer here is a special API used only by gnome-keyring that lets us skip over the download step the first time the keyring file is needed.

Finally, there is a completely different design available. We could use a file-based service like monotone, check things in, and then provide merge utilities for each different file type. One way I don’t like this is that I suspect that current programs don’t differentiate between the different types of stored data; changing the programs themselves ensures a clean separation of identity-related data and transient data. At least for the time being I’m sticking with my first approach.

Gnome 3.0 links

I’ve been reading about Gnome 3.0 lately and I’ve run across some interesting links.

I thought the lazuli mockups looked quite nice. One problem I see in Gnome is that many programs feel quite heavy — e.g., I use evolution only for its calendar, and it seems like an awful lot of UI for something that is conceptually much less important. Programs should have a UI presence about equal to their general importance in my daily life; I only “calendar” occasionally. So, my calendar ought to be invisible most of the time (evo does this well), and quite minimal when I do pull it up (evo fails here). The lazuli screenshots seem to capture this idea pretty well.

This site is sort of a bumptop-y thing written in flash. It is pretty fun to play with. If you click on one link in this post, click on this one.

Gimmie looks pretty cool.

I’m including a link to the Mozilla desktop only because it exists. Mostly I think focusing on the implementation technology is a mistake. Users don’t care — I know I sure don’t. Working on multiple platforms is nice, too, and I can see how that would be useful, but doesn’t fit into my reality.

On the calendaring front, wuja looks like an interesting idea. I’m not a huge fan of web-only software; wuja bridges the gap in a nice way.

Recent Hacking

Lately I’ve been doing some random Gnome hacking. I’ve been a bit bored with what I’ve been working on recently (SRPM hacking — I hate this stuff) so I’ve cast around a bit for other things to do.

At first I was playing with frysk, but I promised myself that I would be a dilettante and hack only the fun bits. Unfortunately this means I’m now stuck waiting for other folks to write some important infrastructure, and also to fix a few mildly contentious bugs. (The bugs would be fun, but I’m also not interested in navigating the controversies in my spare time…)

So, I inconsistently turned to hacking on gnome-session — something both un-fun and contentious. I’m not always sure what I’m up to…

I’d forgotten exactly how awful XSMP is. It pretends to be policy-free, but when you look deeper you see that it makes many implicit assumptions about how things will actually work. It also uses the egregiously nasty ICE. And, of course, gnome-session’s implementation is no great joy, either, which is more than a little my fault.

Anyway, I fixed a number of problems and now I’m waiting for patch review.

Gnome hacking has gotten better since the bad old days. The documentation (glib in particular) has improved, for one thing. The newer APIs are also more well thought out.

I looked a bit more into a minimal implementation of my state-sharing idea. Gnome seems to already have most of the bits I’ll need: I can lift some encryption code from gnome-keyring, and gnome-vfs allows easy access to remote sites (the latter isn’t perfect for my purposes but will do for a proof of concept).

Dancing with myself

Every time I go on a trip I plan to spend half a day “setting up my laptop”. Mostly this means copying files back and forth until it somewhat resembles my main machine, and hoping I haven’t forgotten anything (say, for instance, the time I went to the RH Summit and forgot to bring the VPN key. Sigh).

I’m much too lazy to set up rsync for my desktop configuration for this purpose, and anyway I suspect that I’d spend as much time tweaking any rsync script as I do just copying things around. Instead I’ve spent my time more usefully, thinking about how this could be automated for KDE or Gnome.

What I’m thinking about in particular is a new role for kde.org or gnome.org: give everybody who uses the desktop a little space on the web site, and automatically encrypt and upload certain bits of data; then automatically download and synchronize it when the user logs in and has a network connection.

Initially this could just be a few things relating to my identity: the contents of my keyring (including passwords for web sites), information about my various mail accounts, my gaim accounts, newsgroups I use and information about what articles I’ve read, my blog list in RSSOwl, my calendar, my sticky notes, my .cvspass file, probably even my mozilla bookmarks.

I think that is actually pretty minimal and that over time we’d find more things to add — just go through the list of programs you use daily and it is pretty easy to see bits of configuration that relate to your identity, as opposed to machine- or session-local ephemera. But initially just starting with one or two of these to work out the concept would be fine.

The synchronization phase is the only tricky point. I’m often using both my machines at once, and if I make an account on some website using my main machine, and make another account around the same time using my laptop, I think it is reasonable to expect that the results will be merged — that I won’t lose one or the other.

This means that simply copying config files around in a naive manner is probably not workable. I don’t think there are any big problems solving this; in most cases I care about, it boils down to computing the union of two lists.

In the unusual case where the two configurations refer to one account with different passwords, I suppose we could defer to the user at synchronization time. I doubt this happens often.

In the bigger view I think this is just the first step toward integrating the desktops with their respective web sites. There are plenty more ways this could be done.

Feeding the beast

The other day at the library I realized that, once again, I’ve been missing a lot of good free movies because I never remember to look at the calendar.

So, I decided to feed the beast a little: I sent email to a few organizations I like, asking them to consider publishing their (currently lame) calendars in Google Calendar. Then I could subscribe to them, either in Google Calendar or using ical.

Ideally every place in Boulder — the theaters, art galleries, city council, swimming pools, radio stations, libraries, etc — would publish their calendars this way. It isn’t too much to ask; send email to groups in your city today.

Bumptop again

Up until now I’ve thought that bumptop (and lowfat, and to a lesser extent croquet) was pretty but not very useful. The demos show a lot of nice file manipulation tools, but I use nautilus very rarely; so bumptop has seemed like it put a lot of energy into the wrong place.

I’ve been thinking about it a bit more and I’m not as sure as I once was. It took a while, but I finally realized that my desktop is littered with things that, on a bumptop desktop, would be active objects subject to manipulation — that bumptop actions are not limited to “files” but instead “desktop objects (some of which are files)”.

So, for instance, all the launchers on the gnome menu and on my various panels would fit into this category. So would things like URLs I dragged to the desktop, notification area icons, other panel applets like the mail checker, methusalem-based task trackers, etc.

Something like a panel full of launchers, or the gnome menu itself or that matter, would instead become a bumptop container of some kind. Menu editors would just go away, deferring to a more general (and, if this whole idea makes any sense at all, easier to use) organizational technique.

Technologically it all seems like a good time.. groovy physics-based 3d-ness; downloading hibernating python bits from the web to live like fireflies on the desktop, telling us the state of our imap account or calendar or remote build, until we delete them or bury them under a pile of crumpled PDFs; treating our applications like the edsels they are and, finally, stripping them of their chrome, choosing instead to pull bits from youtube and google video and our mythtv servers to remix them in place in a specialized desktop… a google of hackers could spend years having fun on this.

I’m still skeptical though: the extended definition of “desktop objects” still only covers a small minority of what I do daily. I only occasionally use launchers (and those I have set up nicely on an auto-hide side panel), and the whole point of applets and notifiers is that they are typically passive.

Silly desktop idea

A sillly destop idea: .desktop files should start with “#! some-new-program” and be executable. Then I could run them directly from the command line. Naturally the script interpreter would know how to parse command line arguments and pass them on to the underlying program appropriately.

Gnome Deployment

I was reading about Gnome 3.0 recently (and I’m working on a long post about it), when I ran across a post by Luis Villa about Gnome versus web development. (BTW you should read his blog if you don’t already.)

I found this pretty compelling in general — it is a nice analysis and it is also a list of things that can actually be addressed without going crazy.

I’m interested in the deployment issue, since I’ve done some playing in this area with Java Web Start. And, though I think I don’t agree that this is really a blocker, it is still fun to think about a bit — the technology is fun and cool, I’m just not convinced it solves a pressing problem. (Your response here…)

First, yes, with C or C++ I think deployment is just going to continue to suck.

Java already solved this problem in a pretty nice way. It is no big deal to write a single-click install, with automatic updating, good security (either a sandbox or signatures for apps that need more permissions), etc. This could use a little work for better Gnome integration, and of course we should be shipping this in Fedora, but the amount of needed work here is quite small. (I assume C# has something similar but I really don’t know.)

I looked into this for Python a little this weekend. And, it looks like Python Eggs and the Python cheese shop provide a nice basis to make this work. Finishing this is pretty concrete:

  • Define an XML document describing an application. This would include the same kinds of things that ended up in JNLP: icons, splash screens, the required python version, a list of python egg URLs, etc.
  • Add some kind of signing to python egg downloads. Unfortunately I don’t think sandboxing is an option. One idea would be to restrict eggs to python.org and gnome.org URLs — but that interferes with the idea of easy deployment.
  • Make a new URL type so that Mozilla will know to run appropriate python script to parse the XML, install the eggs, and launch the application.

For best results this would incorporate some things I was looking at for NetX: having the ability to download an application without running it; dbus integration so that a previously cached application won’t try to update itself in a net-less environment; drag-and-drop of entire applications from Mozilla to the panel; and now, integration with mathusalem when downloading eggs.

Strategic Incompetence

I happened across a post about strategic incompetence — the idea that it is better to appear to be incompetent at unpleasant tasks, so that other people will do them for you.

I’m sure we’ve all encountered this in our work. I know I have; I’ve occasionally regretted demonstrating any knowledge of how autoconf and automake work, as I often get stuck with ugly build hacking as a consequence.

That said, I think this strategy is basically flawed. It can only work if you are in an environment full of enablers who don’t mind that you’re dumping on them. It isn’t easy, of course, to confront this kind of behavior; but in a team environment at least it is crucial.

I do wonder if sometimes a project as a whole can use strategic incompetence, in a milder form, as a way to attract developers.

Frysk in the Present and Future

I read Federico’s post about finding poll() wakeups and got really excited about the idea of using frysk to do this.

Unfortunately I looked and frysk doesn’t yet have symbol table lookups, so there’s really no good way to set a breakpoint yet, or to fetch the local callback variables.

What a bummer. frysk offers a couple nice things for this problem: it is very easy to trace many programs (including through forks and multi-threaded programs) at once; this is one of its biggest advantages over gdb for this problem. It is also very easy to write a little custom code to pick exactly what you’re interested in — e.g., any program that calls poll more than 5 times per second — as if you have a scriptable strace.

I didn’t want to leave empty-handed though. So, I refactored ftrace a little and wrote my first actually-functioning frysk-using jython script:

import sys
import frysk
import java

class fdtracer(frysk.ftrace.SyscallHandler):
def handle(self, task, syscall, what):
frysk.ftrace.Util.printStackTrace(java.lang.System.out, task)

tracer = frysk.ftrace.Ftrace()
tracer.setExitHandler(fdtracer())
tracer.addTracePid(int(sys.argv[1]))
tracer.trace()

This will trace a process (identified by pid) and print a stack trace every time a system call in the process exits. (Bah, can’t figure out how to make this blog software not delete leading spaces. So much for posting python code, dammit.)

This is silly of course. Still, writing something like the bad close catcher is an obvious extension of this. And Federico’s case is not really all that far off… I want that to be a nice little script that will automatically find gnome-session, trace all its descendants, notice over-eager polling, and finally print the callback functions that are triggered.