In Montréal for “Boston” Summit

After a crazy Oktoberfest party in Kitchener last night, I woke up at 6:45 this morning, drove back to Toronto, took the ferry to the Island, took a Porter flight to Montréal, took the 747 downtown, took the 165 up to Queen Mary, walked up the hill to Polytechnique to arrive at the GNOME “Boston” Summit. Found Matthias, Owen, Ryan, and Andreas in the hallway, shook hands and received hugs, and felt right at home!

Inside, saw Colin and Marina among a few other familiar faces and many new ones. Marina explained that the reason she looks so sleepy is that she was blogging Ada Lovelace Day last night at 3am. Which of course reminded me that it was Finding Ada yesterday. So I thought I hereby list my own pick of women that I have had the pleasure to work with, and who, in my opinion, have made a lasting contribution to GNOME. Now I don't have to preach these awesome women to this crowd, so I'll just summarize my own experience with them in a two lines. In no particular order:

Marina Zhurakhinskaya has been critical to the Women Outreach Program success and happening in recent years, so for that alone she deserves a special mention. That's independent of she being part of the every-awesome GNOME Shell team. Plus, she's so nice and great to hangout with in person.

Stormy Peters wrote in her job application for the GNOME Executive Director as part of her responsibilities to be the "mom". And she delivered! It was a pleasure being on the board when she was in charge. Plus, she's so energetic she brightens everyone around her whereever she goes!

Karen Sandler is awesome in spite of being a lawyer! I have not had the opportunity to work with her in her new role, but at the Software Freedom Law Center, she was a great resource to the GNOME board, and much easier to get hold of than, well, other Free Software lawyers. Now, I did not actually know about her DJ hobby (check her website!) and wedding invitation until today. Waiting to run into her here to learn more :-D.

Rosanna Yuen is hard to find on Planet GNOME, and that's a shame. Many may not know her, she's sometimes better recognized as zana. Fortunately she's been making more regular appearance at GUADEC. Anyway, it's hard to imagine anything in the Board / Foundation level getting happened without her back-office work. She moves the money, she keeps the book, she knows what happened in the board five years ago! Plus, sometimes researches and books the venue for Boston Summit too.

At GUADEC this year, and at the Summit today I had the opportunity to meet a few young ladies rather new to the GNOME family: Pat, Kat, Meg, and Nohemi: you girls rock! I hope I blog about you for the years to come!

Went for lunch with Marina and Owen, had a great Thai chicken green curry, and talked food. I was thinking about a small project to hack on while at the summit and I thought I pickup rewrapping lines in vte / gnome-terminal upon width change. It's a well-defined well-contained problem, I have a design in mind, and one of the most common requests against vte. I passed my design past Owen, we agreed that it should work, and I hope that's what I'm going to hack on. Stay tuned!

I want to close by a picture of my favorite GNOME artist with the coolest hair style:

[Woah, long post! Been a while since I last did that...]

Labels: Boston Summit, Finding Ada, gnome

¶ 3:58 PM   0 comments

 Arrived at the Desktop Summit

Just got to the Desktop Summit in Berlin. It's lovely seeing everyone after two years.

I'm running Text Layout Summit August 9th to 12th, so come find me for some font and text chat!

Labels: berlin, gnome, harfbuzz, pango

¶ 8:24 AM   2 comments

 GNOME 3.0

GNOME 3.0 released.
Slashdot effect observed.
No Slashdot post in sight.
Great job, everyone!

Labels: gnome, gnome3

¶ 4:19 PM   0 comments

 June

June 18th was my last day at Red Hat.

I spent last week road-tripping to Eastern Canada with my friends.

In a couple of weeks I will start working for Google Canada in the Waterloo office. I will be working for a large part on HarfBuzz as part of the Chrome / ChromeOS team.

Unfortunately the setup also means that I have to skip this year's GUADEC as I can't get a visa on time :(.

I will keep my current GNOME duties, namely maintaining the text stack (fribidi, fontconfig, harfbuzz, pango, etc) as well as vte. The break may give me some time hacking on things I couldn't get the time to hack on before even. We'll see.

That's all for June.

Labels: gnome, pango, vte

¶ 4:39 PM   15 comments

 Stepping down from the GNOME Foundation board

When I decided to run for the foundation board in 2006, many of the old timers where not running again and there was the feeling that new people are needed on the board. The board work has been very educational and rewarding for me, but given other engagements and all the new, capable, people on the board this year, I think it's time for me to step down so I can focus on hacking.

The board has decided to appoint Paul Cutler to take the seat. Paul has been doing wonders on the marketing team, GNOME Journal, and the sysadmin team. I'm sure this opportunity gives him more ways to contribute to GNOME even more.

Labels: gnome

¶ 3:56 PM   5 comments

 Europe, here I come!

I wrote my last exam on Tue night and We afternoon headed to the airport to get to Spain for the WebKitGtk hackfest at the Igalia offices. At this time, stuck in Frankfurt airport after missing my connection.

After the hackfest I'm doing a mini tour of Western Europe, thanks to RailEurope. Mostly visiting friends and family. Currently looking like: Frankfurt -> Kassel -> Paris -> Brussels -> Amsterdam -> Hanover -> Berlin -> (Maybe) Zurich.

Looking forward to meet GNOME guys. And will blog about hackfest progression. Before you ask, my mandate for the event is to port WebKit to use the new HarfBuzz API.

Labels: gnome, harfbuzz, travel, webkit

¶ 5:59 AM   7 comments

 Pango vs HarfBuzz

Since the rewritten HarfBuzz is shaping up fast and getting lots of Buzz these days, I get asked the same question again and again: "Will HarfBuzz replace Pango?" This post tries to answer that.


Short answer: No, not at all! Pango is here to stay. It will change, but only get better.


Long answer:

Pango provides two levels of API: A low-level and a high-level.

Low level API: What I can the "three pillars of pango":

• pango_itemize(): Breaks text into runs that each have the same font, Unicode script, language, direction, and other characteristics.
• pango_shape(): Shapes a single run of text, given the font, script, language, direction, and other properties. Shaping means converting Unicode text to positioned glyphs.
• pango_break(): Does line breaking and other text segmentation (cursor positions, cluster boundaries, word boundaries, and sentence boundaries).

High-level API: Pango's high-level API consists of the PangoLayout object, aka "here's a piece of text render it in this box I don't care what you do."

Of these, HarfBuzz only does shaping. That is, hb_shape() is functionally equivalent to pango_shape().


API implications: Here is how moving to HarfBuzz affects the Pango API:

• Everything in pango-ot.h will be deprecated and be a thin wrapper around hb-ot.h. This is already done in the harfbuzz-ng-external branch of Pango.
• There will be new API in Pango, perhaps in pango-hb.h to help extracting various HarfBuzz structures from their Pango equivalents.
• pango_shape() will be a thin wrapper around hb_shape() (read below).

Pango Modules: pango_shape() calls into Pango shaper modules to get the actual shaping done. There are two kinds Pango shaper modules depending on what they do (the API is the same, so Pango doesn't differentiate between the two classes):

• Bridge modules: The basic-win32.c, basic-atsui.c modules call into another, platform native, shaping system to get the work done. The external (not integrated in Pango yet) modules basic-graphite.c and basic-m17n.c also do the same for the SIL Graphite and m17n shaping libraries.
• On Linux, since there currently is no native shaping engine, Pango has multiple shaping modules, one per script, to do the actual shaping (arabic-fc, syriac-fc, indic-fc, thai-fc, ..., and basic-fc for all the non-complex scripts).

Now, as HarfBuzz becomes the shaping engine on Linux, all those script-specific modules will be removed and basic-fc will simply call into hb_shape(). That's indeed what the basic-fc.c in the harfbuzz-ng-external does.

Later on, when we add support for native win32, CoreText, Graphite, and m17n to HarfBuzz, all those other modules will also be replaced by HarfBuzz-calling equivalents.

Which one to use: Pango or HarfBuzz? Depends.

PangoLayout is designed to be the 'render this text in this box I don't care how' kind of API. That's a perfect fit for GUI toolkits like GTK+, but not suitable for lots of other uses, for example:

• Web browsers
• Word processors
• Designer tools
• Font design tools
• Terminal emulators
• Batch document processors
• TeX engines

while in many of those cases PangoLayout can be made to work (with much pain, mind you), Pango still provides the lower level API and lots of other bits and pieces to get something going. What it doesn't give full control on however is font selection, which happens to be a deal-breaker for many of those usecases (browsers following CSS rules, etc).

So, each of those kinds of applications need to assess the pros and cons of using Pango vs using HarBuzz and providing all the other bits themselves. For example, HarfBuzz doesn't provide:

• An itemizer
• A Unicode Bidirection Algorithm implementation
• A Unicode Line Breaking implementation
• Glyph rasterization
• Glyph metrics information
• etc

There's also a hybrid use possible: to borrow those pieces from Pango on platforms that it's feasable, but drive HarfBuzz directly. It all depends. When in doubt, ask! We have a mailing list.

That said, Firefox will use HarfBuzz as soon as it's ready (there are patches circulating around). Google is using old HarfBuzz for their Webkit and will port to the new one. I'm also attending the Webkit-GTK hackfest in December to port that to the new HarfBuzz. We'll work towards sharing the HarfBuzz-dealing code among Webkit backends.

This is already a long post. Let me finish now. Hope I made it a tiny bit more clear.

Labels: gnome, harfbuzz, pango, textlayout

¶ 8:01 PM   7 comments

 Love GNOME? Show Your Support!

The new Friends of GNOME program that was launched in January have been a great success. I for one have certainly been feeling the love. Stormy will be posting stats this week.

In the mean time, if you use and enjoy GNOME, here's a few different ways you can support it:

• If you have not already, opt in the Adopt a Hacker monthly subscription. It's the easiest way to contribute to GNOME in a sustained and affordable way. Like they say, what are you waiting for? :)


• If you prefer one-time donations, you can do that too. Remember, GNOME is a registered non-profit and donations are tax-deductible in the US.


• You can also support GNOME by letting your friends know about it. Just go to this page and post one of the badges to your blog or website. Thanks in advance!


• Finally, as a special offer, the first person to donate to GNOME in the Philanthropist level ($1200 or more) will receive the following framed art work with an engraved plaque recognizing your contribution:

  
  (Click to enlarge)

  The frame is 13"x18" and comes with over 4GB of ram! Perfect for a GNOME-Friendly office. If you attend GUADEC this year, it will be handed to you by the GNOME Foundation board of directors there. If not, we'll make other arrangements. Contact me at behdad-gnome-org to claim. First come first serve!

Labels: gnome, GNOME foot logo

¶ 10:00 AM   14 comments

 Git Tips and Tricks

Good to see GNOME happily using git. It turned out to be a surprisingly smooth process. Thanks krh and owen!

Some of us have been busy working on the docs. There's still a lot to write, edit, and polish (help appreciated!). My favorite page is the Git Tips and Tricks. I learned a couple new tricks myself, so thought I share. Add yours.

Labels: git, gnome

¶ 3:11 PM   3 comments

 Lots of GNOME happenings

Many good things have been happening around GNOME. Others have already blogged about most of them, so I cheat and link frequently.

A Very Late “Happy New Release”

The release team did it again: GNOME 2.26 was release
d on March 18th, on schedule to the day.

Thanks everyone involved: translators, developers, release engineers, and release note authors.

Humm, darn, I just remember that I didn't hold a release party this time. What a shame. I sure have been busy. More on that later.

Google Summer of Code

It's that time of the year again. GNOME is participating in Google Summer of Code 2009.

The deadline for submitting student applications is April 3rd. For more information (for both students and mentors) as well as project ideas check here.

I decided not to run cairo for GSoC again. And I'm not involved in the GNOME GSoC organization. That makes this year the first GSoC that I have no involvement in. I plan to pick up again next year. That said, I'll be in a GSoC panel at UofT next Tuesday March 31st, in GB244. Students welcome.

GUADEC Call for Participation

The Gran Canaria Desktop Summit website is finally up and running. Which means, you can register for GUADEC now.

The Call for Participation is also up. April 10 is the deadline for submissions. So, hurry up! Bastien, Emmanuele, Paul, Ross, Ryan, as well as myself will review the submissions.

The conference layout is going to be different this year, which makes scheduling harder. Earlier-than-last-night-and-late submissions are highly appreciated. Yes, I'm talking to You ☟.

Next Foundation Elections

The current GNOME Foundation Board of Directors' term ends on June 30th. Which means elections will be coming up in a month or two as Vincent explains.

Take a couple minutes and read his post again if you are passionate about GNOME.

The board work can be frustrating at times, but in the long run it can be quite rewarding. While I miss the days that I could focus on coding and let others worry about organization, money, etc, I'll be running again this year.

Friends of GNOME Love ♥

In January we launched the new Friends of GNOME website. The main feature of the new site has an option for recurring donations, called Adopt a Hacker.

The fun part about the Adopt a Hacker option is that you get to choose your favorite hacker from a list of about a dozen hackers, and the adopted hacker will send you a postcard as a token of appreciation.

One thing about the postcard idea is that since we started the program recently, we are still sorting out the logistics and no postcards have been sent out yet. Which I guess will make receiving it more unexpected for our friends, and hopefully more delightful.

I certainly felt that way when I came back from vacation a few weeks ago and found this in my mail box:

The back of the card reads:

Hi!
I recently joined this
adopt-a-hacker programme
It is something about post
cards. I think I am
supposed to send you
one? See you soon.
R

Thanks you R from Zürich. Now I owe you two. And I feel adopted!

So, what are you waiting for? Adopt a Hacker today!

Labels: elections, gnome, gsoc, guadec

¶ 5:21 PM   0 comments

 GNOME DVCS Survey Results

In December I ran a distributed version control system survey for GNOME. From the survey opening page:

Thank you for taking the GNOME DVCS Survey. This survey is run on behalf of the GNOME Foundation board of directors, release team, and sysadmin team. The GNOME project is planning a possible move from SVN to a distributed version control system in 2009. The contenders for the system to use are bzr, git, and hg. The aim of the survey is to help us better understand familiarity and preferences of our active contributor base regarding the future version control system for GNOME. The survey results will be informational and will be sent to foundation-list and desktop-devel-list upon completion.

GNOME contributors with an SVN account who had an SSH key installed on their account were invited to fill in the survey. A total of 1083 account holders were invited, and 579 filled in the survey. The survey results are now available to the public here. Elijah did an initial analysis of the data. His analysis also includes the survey questions and answers. Find it here. If you analyze the results, please leave a comment on this post linking to your analysis.

Labels: dvcs, gnome, survey

¶ 10:41 PM   6 comments

 Improving Login Time, Part 3: bootcharting

Been a while since last instalment. So, having tightened up gnome-settings-daemon, I headed to get an overview of the entire login process, using the venerable bootchart. Before I jump in, bootchart is cool and everything, but if someone wants a smallish project to hack on, port the bootchart graphing tool from Java to pycairo. It's some 5k lines of code and should shrink considerably. Then we can make an interactive tool based on that and other pycairo scripts we have. I can see how that can turn into a profiler thingy in the long run...

For this assignment I updated my home machine to latest Fedora 10, created a new user, and bootcharted gnome-session in a warm login. I got the chart at right (click to enlarge). Lets see how it looks:

• The CPU becomes idle at 18 seconds in. The little activity lingering after is the bootchart overhead. That's a long time to log in!
• Doh, the CPU becomes idle for a couple of seconds in the middle.
• There's intense CPU activity periods caused by a many processes, namely, gnome-settings-daemon, wnck-applet, clock-applet, gdm-user-switch, and less intense in gnome-panel, nautilus, notification-daemon, and sealert.
• gnome-settings-daemon is terminating!

This doesn't look right. Lets look at inside:

The CPU idle time

Let me quote parts of my first post about how gnome-session works:

To transition to the next phase, the current phase should either complete or time out. A phase is complete when all apps associated with the phase signal completion. An app can signal completion in a variety of ways, the simplest of which being that the process terminates. A phase times out after ten seconds. Description for the phases as well as a more a longer version of this condensed overview is available in gnome-session/gnome-session/README.

With that in mind, the two second idle looks a hell lot like a phase timeout. Inspecting ~/.xsession-errors and searching for gnome-session confirms that:

gnome-session: Application 'libcanberra-login-sound' failed to register before timeout

Oh oh! /usr/share/gnome/autostart/libcanberra-login-sound.desktop is where this application lives, and it contains these lines:

Exec=/usr/bin/canberra-gtk-play --id="desktop-login" --description="GNOME Login"
AutostartCondition=GNOME /desktop/gnome/sound/event_sounds
X-GNOME-Autostart-Phase=Desktop

That is, it runs canberra-gtk-play in the Desktop phase of gnome-session. So what's happening? gnome-session is waiting for canberra-gtk-play to finish playing login sound, how cool is that!? Not only gnome-session waits for it to finish playing, even when it does (after five seconds), gnome-session fails to notice that. So it waits another five seconds until the phase times out and it proceeds to the next phase. To verify this, note that in the graph, canberra-gtk-play starts at about 3.2 seconds in, and the CPU picks up activity again, at 13.2.

Why does it happen? Shouldn't gnome-session at least continue after the five-second playing is over? Well, yes according to the README document, no according to the code:

  if (manager->priv->phase == GSM_MANAGER_PHASE_INITIALIZATION) {
       /* Applications from Initialization phase are considered
        * registered when they exit normally. This is because
        * they are expected to just do "something" and exit */
       app_registered (app, manager);
}

So, only Initialization phase was handled that way. Not hard to fix. Bug filed, patch submitted. Then I also patched libcanberra to add a --background option to canberra-gtk-play. Reported and patch submitted. There is an alternate, much shorter, fix: move login sound to Application phase which is the last phase. Applications in the Application phase are not required to notify the session of their startup. That's a one line fix. The gnome-session fix is still a good thing to do. Anyway, with that fixed I got down from 18 to 13 seconds. Not bad!

The Mess

Lets see what's making the login take that long. The last batch of processes to start are the panel applets (clock-applet, wcnk-applet, trashapplet, gdm-user-switch, and notification-area) that start around 8 seconds in. The panel on the other hand starts at 3. What is the panel doing for those 5 seconds?! Time to switch tools and take a look inside panel. Federico looked at it in February, so I reused his patch and added more annotations as needed. The image at right is the first timeline view of what's happening inside gnome-panel. There are three large gaps there. The first one is the linker working hard. The second one is gnome_program_init(). I have absolutely no idea why those two are taking so long. Those two are very unstable though and go up and down in different runs. I will look into them later.

But the largest gap by far is the third one, which happens inside the first size_request(), and after I added enough annotations, it became clear that it's happening when pango is measuring fonts for the first time. Ouch! Checking the strace log, it becomes clear that fontconfig is opening three East Asian fonts. Why? It's not supposed to. Well, at least not if its cache is up to date. I then run fc-match and observe that it takes over a second. I try fc-cache and it errs about not being able to update cache for two East Asian font directories on my system. Humm, this is a first for me, although I confess I get reports about situations like this all the time... Reading more log, it appeared that opening cache files in ~/.fontconfig was failing as Permission denied! Weird stuff. Inspecting ~/.fontconfig revealed the reason: It had the following permissions: drw-rw-r--. Where did the x permissions go? This was a new user account, so the first process to call into fontconfig would have been gnome-settings-daemon. Oh my! In one of my patches from the last round, I had replaced a call to the glibc function daemon() with custom code that I lifted from preload, which I had lifted from some other daemon back in 2005. Anyway, seems like the daemonizing code there had a umask (0177); call in it, and no one's eyes ever caught the bug there. That mask is fine for files, but directories created under that are absolutely useless. And since Nov 25 we backported those changes into Fedora 10! Chaos! I had to go clean that up. Filed and fixed the bug in g-s-d first, and pushed fixed versions in F10 and rawhide. Then filed bug with fontconfig to fix it there too. Will look into that this week.

Next question was, why wasn't the system font cache updated anyway? Normally the users should not need to cache anything unless they have installed fonts in their home directory. I checked the RPM spec for the offending font packages. Dang, the spec files had a umask 0133 before calling fc-cache. Oops. Removed those and pushed updates out. However, that does *not* justify it as fc-cache doesn't need to create any directory when updating system-wide caches. That's still a mystery. Got to reinstall the broken RPMs to see what's going on.

Anyway, I then looked into gnome-settings-daemon to see why it was crashing. It was gstreamer crashing it. As I found last time, gstreamer initialization forks and tries to update the cache. The fork being there to not crash the parent process in case a plugin goes south. However, if updating the cache fails in the child process, it then retries from the parent. If that fails too, it just exits! The reason it wasn't updating the cache successfully? The same umask(0177). Now I remember, I can actually blame Vincent! When he was releasing GNOME 2.25.1. IRC log from November 5th:

We totally had the bug in our hand and let it go. Looked so innocent for sure. Not very clear who let it go. Vincent says he'll sue me for posting this log anyway.

Later on I also got mail from a Fedora user saying that the umask thing also created problems for him with ORBit. I already reported to gstreamer devs, and will fix the fontconfig one myself. If anyone can look into ORBit, that would be great.

Lesson learned the hard way: In your library code, when you mkdir(), do a chmod() immediately. If you have not been doing it before, you may also want to consider trying a chmod if writing to the directory fails. umask issues have affected many many packages in Fedora before. Users install RPMs with limited umask and all kind of things break. better fix it in the libraries and programs, instead of distros adding umask to their specs and getting it wrong...

Oh, the timing. Fixing these, the huge gap in the panel shrank drastically, from 1.4 seconds to 0.13 seconds. 130ms is still a long time to spend in fontconfig. Seems like about 60% of that is just opening and reading/mmap'ing the config and cache files. That can be improved by reducing the number of files: packaging more font files in the same directory, and avoiding micro config files. Fortunately Nicolas is on the right track to fix these in Fedora. The other 40% is spent on the first FcFontSort() call. I have ideas about how to improve that, and will look into it in the coming days.

Do we need to call into fontconfig there? The panel needs to know the height of the default font to calculate the minimum panel height and enforce that. Sounds like a good idea, but knowing that the panel properties dialog does not let you resize it to below the minimum height, this check is redundant unless you managed to modify the gconf key directly or changed the font size while panel was not running. In other words, the exceptional case. If we can avoid that call, g-s-m can go to next phase some 130ms earlier.

There is another 90ms gap in the graph, and that is GTK+ finding the icon cache to use and loading the panel icon. Right, the panel has an icon! If I remove the gtk_window_set_default_icon_name() call the gap shrinks considerably, but then when you ctrl+alt+tab you won't see the cute gnome-panel icon. Donno, maybe we can set the icon later to avoid holding the entire session back for 90ms.

Lets step back and look though. fontconfig cache, gstreamer plugin cache, icon cache. We've got so many caches, which actually do wonders, but just finding and opening them still takes a lot of time. The gstreamer case is easy to fix since it has a single cache file in a single directory, it's just that it currently is doing unnecessary checks. The fontconfig and icon caches are harder to fix. The libraries have to look into multiple directories and read multiple config files to use the cache. It stinks. There used to be a short period of time that fontconfig generated one fat cache file for all the fonts. That's a terrible idea when you need to regenerate the cache every time a new font is installed. But what if we had a "git gc"-like mode to merge multiple cache files into one? New font dirs will go into new cache files, until next time, either manually or automatically, fc-cache --compact is invoked... I'll think about it.

I said "holding the entire session back" twice in previous paragraphs. This was not actually happening: gnome-panel was registering with the session as first thing in its main() (it's implied in the gnome_program_init() call), and the session was taking that as "I'm ready, go to next phase". Which means that Nautilus may start up and draw icons before the panel has had a chance to set its struts. Net result is that Nautilus icons will jump down when the panel gets to set up the struts. So I wrote patch for that. But that didn't fix the problem, another bug was in the play. Two hours of debugging later, I fixed that too.

The rest

With this mess cleaned, login time is back down to a sane 5.5s. gnome-panel is not taking much CPU time anymore. Nautilus some. Other than those, two things stand out: python and sealert. sealert is an applet to "View & interpret SELinux denials". It's written in python. Worse, when starting, it calls over D-BUS to start a server instance of itself, in another python process, and the two wait for SELinux denials to happen so they can report it to you using notification-daemon bubbles... for you to click and dismiss. Srsly, what the heck is Aunt Tillie to do with them? Forget Aunt Tillie, what is even a tech-savvy user to do with them? We were at the Fedora 10 release party in Seneca a couple weeks ago and a user asked me what these bubbles are. I explained to him, and he said "I just close them because they happen all the time". Which is true. It's great if you are debugging why your HTTP server does not serve your files, but is it good for anything else? Certainly not enough to be enabled by default. Please, disable sealert by default and help save the planet. In the mean time, I also proposed adding an additional phase to gnome-session which would run after the Application phase and is for starting processes that do not render to the desktop immediately, because those can start at the end, to give a better perception of desktop having loaded completely. sealert technically falls into that category. In practice however, it pops up a bubble before the login is done. In many setups at least.

I think the same argument also applies to the kerneloops-applet. Maybe enable these in Rawhide so people report bugs, but disable them in stable release. I'm pretty sure that's the right thing to do. kerneloops-applet is quite light on the CPU though. Removing sealert, kerneloops-applet, and some other lightweight tasks that I don't need, I get down to 4.5 seconds. The biggest CPU consumer now is Nautilus, and it's taking three seconds to settle down. Federico looked into it before, and I will too at some point. But I first want to fix fontconfig. The ROI on that is huge: every application startup can become something between 50ms to 100ms faster if I my ideas work out.

Big picture

One wise Federico once said "Set concrete goals", although people now attribute that wisdom to Arjan these days. Regardless, let's set a concrete goal: I want to make Fedora default install, new user, warm login in three second. That may not happen next year, but that's where we need to be if we ever want to boot, say, in 15 seconds. Why optimize warm login when cold is so much worse? Because to make cold login go under 5 seconds, one first has to make warm do that. My other argument is that ideally people suspend their computer instead of shutting down, so first boot is the wrong thing to optimize. Warm login on the other hand is still common in multi-user systems. Moreover, my hope is that a 3 second warm login plus opportunistic preload'ing at login screen will give us a first login of about 5 seconds. Long road to there; we will see. By the way, 3.5 years later, Federico's GUADEC keynote is still pure gold.

How would we get there? By making fontconfig not suck 100ms for each application. By fixing Nautilus, whatever it is doing that it shouldn't. By getting rid of libgnome dependencies. By figuring out why linking is so slow. And by not running sealert and other unnecessary stuff.

To keep things in perspective, lets have a quick look at cold login to see what's there to optimize. The image at right shows a cold login of the same desktop I used above. Lots of I/O, and 14 seconds login time. The CPU goes way down for extend periods of time, when panel, Nautilus, pulseaudio, and python are all doing heavy I/O. No idea what they are reading, but looks like something that should be possible to fix.

This next one is with preload running. Down to 12 seconds. Not much really. Looks like it failed to preload anything that those four I/O-heavy processes use. No wonder, because preload can only preload stuff that are mmap()ed. That mostly translates to shared libraries. A more advanced prefetching solution, like Harold's readahead_collector that monitors all I/O will be able to solve this. Oh, Harold by the way blogged recently about using systemtap to measure boot-time I/O. He's actually looking at me to fix gconf I/O issues, and that's exactly what I'm going to do with his script when I get to that stage.

If you read this far, I owe you your beverage of choice. Find me at GUADEC+Akademy!

Labels: gnome, gnome-panel, gnome-session, login, performance

¶ 3:13 PM   38 comments

 Improving Login Time, Part 2: gnome-settings-daemon fixed

Thanks everyone for your comments on my previous poinst. Those issues have been keeping me quite busy. And as soon as I thought I'm done, the cleaned up plot showed some new rather empty areas that now were standing out, so I added some new annotations and continued optimizing. Well, I think I'm at a point that I can't make it any faster, so here we go.

Regarding the architecture, I made the parent process wait until the child initializes all the plugins and exit only when the child wants to enter its main loop (bug 559168). This way, plugins can use the already existing g_idle_add() postpone work for after the parent returns. Note that there is no waiting before the idle callbacks are run. It's just a way to decide what has to be run before any other processes in the session start, and what can run in parallel with other processes.

Lets look at the plot from last time again and go over the hot spots I identified previously:


1. linking: g-s-d now doesn't link to libgnome directly, so the gap the has shrank now. However, all the modules still indirectly link to libgnome by way of libgnome-desktop, so the first module loading actually gets the hit now. Well, that was before I learned that Alex already ripped the libgnome dependency from libgnome-desktop. I don't have that installed, so the actual situation is better than what I'll show in my after plot.

Status: Fixed (bug 557808).


2. gtk_init: As I said there's not much to gain here. At some point I want to dig in and reduce roundtrips Gtk+ makes. Owen did this in 2003, bringing the number of round trips from 52 down to 23. I suspect we may have gained some excess fat in the five years since, so an inspection may be in order. No status change.


3. fontconfig_monitor: I moved installing the monitors to an idle callback, but also added a direct call to FcInit() in the startup routine, to make sure the fontconfig cache is up to date before a herd of other session applications start and all try to rebuild the cache if it's indeed out of date.

Status: Fixed (Part of bug 559166 and bug 559550). The total time is no different, it's just partially delayed.


4. mkfontdir: I cleaned up the module to not run mkfontscale or XSetFontPath unless there's any fonts present. This optimizes the common case where users don't have any per-user fonts installed for the legacy X font system.

Status: Fixed (bug 559163). Nothing left of it on the plot.


5. mousetweaks: Fixed it to not try to stop a daemon that we know is not running.

Status: Fixed (bug 559165). Nothing left of it on the plot. Also made it start in an idle callback (part of bug 559166).


6. init_kbd: Removed the trap+XSync around each individual grab request. And added a grand one outside the loop. That did it. Also start this in idle handler as it doesn't have to be started before other apps.

Status: Fixed (bug 559164 and bug 559482).


7. acme_volume_new: The awesome gstreamer hackers are working on it. ensonic told me that they were stat'ing each plugin twice where one would be enough. That is fixed in trunk/master. There are other ways to optimize it (not fork, not stat in the first place, etc) and all should be properly fixed in gstreamer. So I wrote to gstreamer-devel.

Status: Awesome gstreamer hackers working on it.


8. gnome-screensaver: Rodrigo made it start in an idle callback, and I reshuffled it to do its callback after everything else.

Resolution: Fixed.


9. clipboard_manager: Also starts in idle callback now.

Status: Fixed (part of bug 559166).


10. xrdb: Plugin disabled by default.

Status: Fixed (bug 557807).


That was for issues I identified last time. Thanks to the very responsive g-s-d maintainer, Jens Granseuer, these patches were all reviewed and committed in a few hours and made it to the 2.25.1 tarball.


I also identified some other hot spots after that I also fixed:


11. gconf: By auditing the gconf usage of the daemon and the plugins I simply added the most feasible preloading setting on vairous directories. The effect on the plugins is minimal, but this is visible as a clear win around the beginning of the plots where the daemon is building the list of all the plugins (busy blue-and-brown area), even when you add the preloading time.

Status: Fixed (bug 559167).


12. XSync: I noticed that a lot of the error-trap+XSync that the keyboard code across many plugins did were actually not necessary. So a bunch of those were fired too.

Status: Fixed (bug 559346 and bug 559562).


13. more XSync: Seems like much of the remaining XSync's or other synchronous X operations (mostly in xkb code) can also wait until the idle handler is run.

Status: Fixed: a11y-keyboard and media-keys plugins are moved to idle callback (bug 559564).


14. GnomeBg: A byproduct of the fact that I was running g-s-d in foreground causing Nautilus not just starting, and a bug in gnome-screensaver was causing g-s-d to set background after gnome-screensaver startup. Now this is normally not a big deal because when g-s-d detects that Nautilus is running it does not set the background. The fix I committed was needed to make my plot clean though.

Status: Fixed by delaying GnomeBg object creation until needed, essentially ignoring spurious change notifications (bug 559639).


15. status_icon: The a11y-keyboard plugin was showing a large gap that I tracked down to an innocient-looking gtk_status_icon_new_from_icon_name(). Indeed, looking at the strace log, this was causing the Gtk+ theme to be looked up in various places and opened, etc, while I knew on my desktop, and on most desktops, the a11y icon is not enabled to be shown by default. So I made the icon lookup lazy.

Status: Fixed (bug 559558).


That's it. These are all now committed too. And just when I thought I've exhausted my list of things to fix in g-s-d, I noticed that the daemonization logic in it opens the X display and then forks and use that same display from the child. While this seems to have been working fine, it's definitely in the don't-do-it land. So that one has to wait until I'm done with this already-long blog post before being fixed (bug 559695).

So! Lets look at the plot after all the fixes above. The overall time halved. Lets look at the remaining spots quickly (almost none can be fixed in g-s-d):

1. Linking the daemon

2. gtk_init

3. Preloading plugin data gconf dirs

4. linking libgnome and other unneeded libs (fixed in libgnome-desktop trunk already)

5. xrdb: the xrdb plugin is gone. This very small invocation is needed to set font settings on the server. This is used by non-GTK+ cairo applications, so I'm hesitant to kill this just yet

6. fontconfig cache check

7. xrdb for Xcursor. Not sure if this one can go away

8. Linking pulseaudio and other audio libraries

9. Linking keyboard related libraries

10. gstreamer cache check

11. End of daemon initialization. Parent process exits now.

12. Relaxing. X server running mostly.

13. Installing fontconfig inotify monitors.

14. Keyboard initialization.


That's all for tonight, and for g-s-d. Next: overall view of the login process.

Labels: g-s-d, gnome, performance

¶ 12:35 AM   15 comments

 Improving Login Time, Part 1: gnome-settings-daemon

Any true GNOME hacker has to take a shot (multiple shots, mind you) at improving login time. Federico did it a couple years ago, and since I want to be like Federico when I grow up, that's just what I've been doing for the past week, and expect to keep doing for the weeks that come.

How does login work anyway? In short, gnome-session is started and it then in turn reads the list of tasks to start from .desktop files. Each task is marked as belonging to one of a few login phases, in chronological order:

• startup
• initialization
• window-manager
• panel
• desktop
• application

The startup phase belongs to gnome-session itself. Initialization, then, is the first phase application can sign up to be run in. To transition to the next phase, the current phase should either complete or time out. A phase is complete when all apps associated with the phase signal completion. An app can signal completion in a variety of ways, the simplest of which being that the process terminates. A phase times out after ten seconds. Description for the phases as well as a more a longer version of this condensed overview is available in gnome-session/gnome-session/README.

The initialization phase is where actual doing stuff begins, and for this blog entry we will focus on that (to be honest, only part of it). The mentioned README has this to add about the initialization phase:

GSM_SESSION_PHASE_INITIALIZATION is the first phase of "normal" startup (ie, startup controlled by .desktop files rather than hardcoding). It covers low-level stuff like gnome-settings-daemon and at-spi-registryd, that need to be running very early (before any windows are displayed).

Before leaving the quotation, notice the emphasis: "before any windows are displayed". We'll get back to that.

The most prominent process started as part of the initialization phase is gnome-settings-daemon. No wonder, as that single module consists of some fifteen plugins that do all kinds of startup. In other words, initialization is gnome-settings-daemon. Lets dive inside.

When Jon McCann and co refactored gnome-settings-daemon into plugins, they also added profiling annotation hooks that can be used for Federico-style timeline plotting. So what I did was rebuilding g-s-d, hooking up strace, and drawing the results. All this on my home machine that is rather beefy and otherwise underutilized.

I did one thing wrong, I forgot to send the strace background, so my mock g-s-d was not terminating and hence kept gnome-session waiting for it for ten seconds and finally giving up on the phase and moving on. The net result was that g-s-d was run solo with no other process racing with it for the CPU. That, and doing warm logins, meant my timings where very predictable and consistently reproducible. In this scenario g-s-d becomes idle in just short of one second. In a more real scenario of sending the strace to the background, it takes more like 2.5s. But that's beyond the point. By letting g-s-d run as fast as it can, it's easier to spot what's slow, as those are sure to stand out.

Before getting any further, I stopped and asked myself "is this worth optimizing?" A good question to ask before any optimization work is to start. My immediate response was: "sure, if we I can cut that 1s in half, that's about 10% saving on a 5s login time". After five seconds of thinking I wondered: "but isn't g-s-d forking and returning in the parent immediately? A .5s saving in g-s-d idle time means little for the login time as a whole." And that's mostly true, but hey, aren't the initialization phase processes supposed to set things that need to be done before any windows are shown? And by forking and returning early, g-s-d is actually not doing that. I can already see how, for example, the xsettings plugin in g-s-d comes up and sets the font rendering settings of the display, causing a redraw in any Gtk+ applications already started. It would be much better to make g-s-d actually do essential initialization as fast as it can, return in the parent, then take its time doing other work that does not have to be done before windows are shown. That's what I'm planning to make it do. In light of that plan, lets see how tight or loose things currently actually are.

Without any further rambling, here is the plot of gnome-session-daemon as shipped in Fedora rawhide starting up. The only modifications I have made is adding more annotations for the plot.

After studying the plot and the underlying log and code I identified ten hotspots and plans to fix them. The hotspots are named in the plot. The names are not readable in the small version included here (click for full version), so I have also numbered them:


1. linking: This is pretty much the cost of loading 67 shared libraries that g-s-d currently directly or indirectly links to. There's not much we can do immediately to make the linker faster. We can try to link to fewer libraries however. Seems like we don't really need libgnome. Chopping that spares some 20 of that 67.

Resolution: mccann already filed Bug 557808 – don't use libgnome.


2. gtk_init: There's not much to do here right away, except that I want to profile gtk+ initialization sometime to see if we can improve it. It's not long, but any saving benefits every application, so it's worth pursuing. No resolution.


3. fontconfig_monitor: This one is so embarrassing. It's my single commit to g-s-d, and it takes the longest time in the plot. What this code does is to add gio/inotify monitors on all font directories and configuration files known by fontconfig for change notification so it can 1) rebuild the fontconfig cache and 2) signal applications to reload their font configurations.

Now, it's a good idea to make sure fontconfig cache is current before other applications start and each try to rebuild the cache, but installing the monitors can wait. It's one of those things that is equally as good if done 10s into the login.

Resolution: Only check that fontconfig cache is current. Defer installing file monitors to idle time.


4. mkfontdir: This one's so bogus. We scan two directories and cleanup symlinks for "cursor fonts" we may have created before. Then if there's any cursor font set in gconf, we create a symlink for it, then call mkfontdir (that in turn calls mkfontscale, which does a bunch of stats on nonexisting files and directories, etc) and get and set the X server font path. All that work even if there is no cursor fonts set..

Resolution: Skip spawning mkfontdir and setting the server font path if there is no cursor font set.


5. mousetweaks: This one's my favorite. According to the man page, "mousetweaks is a daemon that provides various mouse features for the GNOME desktop. It depends on the Assistive Technology Service Provider Interface (AT-SPI)." What the g-s-d plugin does is to monitor the relevant gconf keys, and start/stop the mousetweaks daemon on demand.

On a typical desktop with no tweaks configured (%99+), it spawns "mousetweaks -s", which means "stop the running daemon, if any". The mousetweaks process then starts up, initializes a bunch of stuff, including the a11y stuff (not the fastest stuff it seems), tries to find a running daemon, fail, and silently exit. So much for so little.

Resolution: Don't spawn mousetweaks if no tweaks configured. In other words, don't spawn "mousetweaks -s" unless we know a daemon is running.


6. init_kbd: This one was harder to figure out. There was no big fat thing going on. Instead, there is a look over some 20 different media keys, for each of them some gconf reading and a grab_key operation.

The gconf stuff as my plot agrees is not the bottleneck as the code already does a one-level-deep preloading on the gconf directory. The grab_key invocations however each take real time, and they add up. Looking into what grab_key does is revealing: for each combination of the ignored modifiers, for each screen, it does the usual "push error handler on display; do something with display; flush the display; pop and see if any errors happen". Multiplied by the number of keys, that's a bunch of X display flushes while we're not really interested in pass/fail status of individual operations.

Resolution: Do one "push; do; flush; pop" instead of many.


7. acme_volume_new: What's happening here is that to be able to control volume and other mixer properties, we end up initializing gstreamer. Which in turn wants to ensure that its binary cache of plugins is up to date, so it forks and stats all the plugins. Ouch!

Now, making sure the gstreamer cache is up to date before every other application starts using it is a good idea, but it doesn't have to be so painful!

Whether it's that no one has got to fix it yet, or if there's good reasons for the binary cache not to simply store the timestamps of the folders and compare that instead of doing stat on every plugin on every startup of every gstreamer-using application, I don't know. I won't judge. Love to hear the issues. But experience with Pango and fontconfig tell me that a more decent cache can be done and indeed should be done.

The fontconfig cache, admittedly, becomes really hairy at times (time skews, anyone?), but much, much, much much, better than if fontconfig stated any and every font on startup!

I also have no idea why it gstreamer forks for the cache check. I could think of not polluting the current process or risk crashing it. BUT! If the forked process fails validating the cache, it then retries in process! Oh well...

Resolution: Awesome gstreamer hackers, please fix your cache! In the mean time, forcing gstreamer to not fork may help (can be done by setting an env var).


8. gnome-screensaver: Not sure why gnome-screensaver is so heavy to start up. That's for another session. But, who cares if gnome-screensaver starts 10s into the session? Right, you got it, it does not have to be started before all other windows.

Resolution: Start at idle time.


9. clipboard_manager: This one also baffles me. It's a bunch of X roundtrips. Shouldn't take that long. Anyway, given how clipboard managers work (they are useful when the app holding the clipboard content is existing), no one would really notice if we started it 10s into the session.

Resolution: Start at idle time.


10. xrdb: The xrdb brokenness (it calls gcc!) is a well-known and well-studied issue. According to mclasen the only reason we kept doing it was xemacs, but allegedly that uses Gtk+ these days. Is there any other reason we should be doing xrdb in g-s-d in 2009?

Resolution: mccann filed Bug 557807 – disable xrdb plugin by default. Isn't he awesome?


That's tonight's ten commandments, err, resolutions. I have already started hacking the new architecture in g-s-d and patching the plugins as described above. There are more plugins, and each can use a quick remove. In general any plugin that does "set XYZ and hook up for change notifications on it" can be split up to do the change-notification hookup at idle time if that part consumes considerable time. That pretty much rounds it up for now.

Labels: g-s-d, gnome, performance

¶ 4:00 AM   23 comments

 GNOME Job Posting Board

Many GNOME-friendly companies have job openings that they announce either on blogs, IRC, or random mailing list.

Many GNOME hackers look for jobs and they crawl blogs, mailing lists, or ask on IRC.

Now there is a central place that community members can post GNOME-related job openings, and job seekers can subscribe to. Nothing fancy, a good old wiki page: Jobs. Populate!

Labels: gnome, jobs

¶ 3:24 PM   2 comments

 Setting the record straight

In Heathrow for another hour, then will arrive in Istanbul for GUADEC. I'm staying at Golden Horn. Guys, lets meet at the lobby around 9PM for mild beer tasting.

Also, it's a shame to read "Istambul" on pgo so frequently. Please, write "Istanbul", read as you wish :).

Can't wait to meet everyone... For those of you not coming this year, sorry guys, have to live with it for a week. I'm talking to you sri :-D.

Update: There are two Golden Horn hotels. I'm in the Sultanahmet one. But gather in the lobby of either one and you'll find enough familiar faces to go out drinking with.

Labels: gnome, guadec, istanbul

¶ 4:09 AM   2 comments

 Akademy+GUADEC *2009* Hosting Proposals

In response to our joint call-for-bids earlier this year, the GNOME Foundation and KDE e.V. boards received three proposals tohost Akademy+GUADEC 2009. The bids are available for review here.

The boards did not receive any separate bids for Akademy-only or GUADEC-only hosting. Which proves again, how excited the community is about the joint conference. Note that we got only one bid for GUADEC 2008, and two bids for the year before that.

At this time we are soliciting comments from the GNOME community and other GUADEC regulars. Please use the thread on foundation-list to submit your comments. The review period closes on July 4th, in preparation for making a decision in Istanbul.


Regards,

Behdad
On behalf of GNOME Foundation and KDE e.V. boards

Labels: 2009, akademy, gnome, guadec, kde

¶ 11:20 AM   0 comments

 Real GNOME Hackers

are older in bugzilla points than in real life. (not me)

Labels: gnome

¶ 3:57 AM   2 comments

 The one with chpe Rocking!

On #gnome-hackers today:

behdad: guys
behdad: if you see chpe
behdad: make sure you hug him
behdad: he pushed more than 100 commits to gnome-terminal today
crevette: yeah
crevette: behdad: near 350
behdad: crevette: yeah, still deleting mail...
***behdad gave up on reading them
crevette: hello btw behdad
behdad: Status Whiteboard| |[decision][chpe:wontfix]
behdad: haha
crevette: :)
behdad: hey crevette
behdad: I have no idea what he's been doing
behdad: but this all looks SO GREAT
bkor: move to gtkbuilder, etc
behdad: and he totally gave up on updating ChangeLog btw
behdad: bkor: much more
behdad: complete code cleanup
bkor: behdad: yeah, didn't read it yet
behdad: removed the terminal widget abstraction
bkor: behdad: seems to have removed vte abstraction
behdad: yeah
bkor: hehe
behdad: which is good
behdad: but this is proof that we need git....
crevette: this is a proof that some people are aliens
crevette: :)
behdad: crevette: that too
bkor: behdad: DVCS
behdad: bkor: political correctness :P
bkor: behdad: I don't agree
behdad: I agree :)
jonner: behdad: what did chpe do?
crevette: jonner: he commited like a mad on gnome-terminal
behdad: crevette: about 500 to this moment
behdad: jonner: ----^
jonner: 500 commits?
behdad: jonner: yes
jonner: holy...
crevette: damn
behdad: we should do a special gnome release just for that
***crevette always considered chpe like the unknown hero of GNOME
crevette: he deserves tens of blog post to tell how he is good
behdad: crevette: he defies fame
crevette: :)
pochu: how can somebody do 500 commits in a single day?
behdad: someone put this in topic :)
behdad: pochu: pushed multiple git branches he had sitting around
jonner: probably git-svn (?)
Leftmost: Wow. gnome-terminal could certainly use some cleanup.
behdad: each having ~50 commits for a cleanup task
pochu: behdad: ah
pochu: cool anyway :)
pochu: sadly I don't use gnome-terminal :(
pochu: he could have done that in GTK+ or something else ;)
behdad has changed the topic to: chpe meter: http://tinyurl.com/5yvgc3

As such I'm giving away the titles of gnome-terminal maintainer and gnome-terminal developer. I certainly don't fit those anymore. It's all chpe's now!

The closest thing to this that ever happened to my modules was what Chris Wilson did to vte. Fortunately I was successful converting him to a cairo hacker and he has made 600 cairo commits since. Lets see where chpe shows up next!

Labels: chpe, gnome, gnome-terminal

¶ 5:19 PM   8 comments

 GUADEC schedule now available

Thomas is still uploading last GUADEC's videos, but...

The schedule for GUADEC is available now. There are still a bunch of slots awaiting confirmation from their speakers before showing up in the schedule, but the core days (9th, 10th, 11th) are pretty much complete now.

Expectnation made it relatively easy to do the schedule after I upgraded to Firefox 3 (it was painfully slow with FF2 under Linux). Thank you guys. Doesn't mean I didn't have to use the pen though. This is how it was done:

Still better than last year.

Labels: gnome, guadec

¶ 2:14 PM   2 comments