Mark Shuttleworth | here be dragons

There’s been a flurry of discussion around the idea of syncronicity in free software projects. I’d like to write up a more comprehensive view, but I’m in Prague prepping for FOSSCamp and the Ubuntu Developer Summit (can’t wait to see everyone again!) so I’ll just contribute a few thoughts and responses to some of the commentary I’ve seen so far.

Robert Knight summarized the arguments I made during a keynote at aKademy last year. I’m really delighted by the recent announcement of that the main GNOME and KDE annual developer conferences (GUADEC and aKademy) will be held at the same time, and in the same place, in 2009. This is an important step towards even better collaboration. Initiatives like FreeDesktop.org have helped tremendously in recent years, and a shared conference venue will accelerate that process of bringing the best ideas to the front across both projects. Getting all of the passionate and committed developers from both of these into the same real-space will pay dividends for both projects.

Aaron Seigo of KDE Plasma has taken a strong position against synchronized release cycles, and his three recent posts on the subject make interesting reading.

Aaron raises concerns about features being “punted” out of a release in order to stick to the release cycle. It’s absolutely true that discipline about “what gets in” is essential in order to maintain a commitment on the release front. It’s unfortunate that features don’t always happen on the schedule we hope they might. But it’s worth thinking a little bit about the importance of a specific feature versus the whole release. When a release happens on time, it builds confidence in the project, and injects a round of fresh testing, publicity, enthusiasm and of course bug reports. Code that is new gets a real kicking, and improves as a result. Free software projects are not like proprietary projects - they don’t have to ship new releases in order to get the money from new licenses and upgrades.  We can choose to slip a particular feature in order to get a new round of testing and feedback on all the code which did make it.

Some developers are passionate about specific features, others are passionate about the project as a whole. There are two specific technologies, or rather methodologies, that have hugely helped to separate those two and empower them both. They are very-good-branching VCS, and test-driven development (TDD).

We have found that the developers who are really focused on a specific feature tend to work on that feature in a branch (or collaborative set of branches), improving it “until it is done” regardless of the project release cycle. They then land the feature as a whole, usually after some review. This of course depends on having a VCS that supports branching and merging very well. You need to be able to merge from trunk continuously, so that your feature branch is always mergeable *back* to trunk. And you need to be able to merge between a number of developers all working on the same features. Of course, my oft-stated preference in VCS is Bazaar, because the developers have thought very carefully about how to support collaborative teams across platforms and projects and different workflows, but any VCS, even a centralised one, that supports good branches will do.

A comprehensive test suite, on the other hand, lets you be more open to big landings on trunk, because you know that the tests protect the functionality that people had *before* the landing. A test suite is like a force-field, protecting the integrity of code that was known to behave in a particular way yesterday, in the face of constant change. Most of the projects I’m funding now have adopted a tests-before-landings approach, where landings are trunk are handled by a robot who refuses to commit the landing unless all tests passed. You can’t argue with the robot! The beauty of this is that your trunk is “always releasable”. That’s not *entirely* true, you always want to do a little more QA before you push bits out the door, but you have the wonderful assurance that the test suite is always passing. Always.

So, branch-friendly VCS’s and test-driven development make all the difference.  Work on your feature till it’s done, then land it on the trunk during the open window. For folks who care about the release, the freeze window can be much narrower if you have great tests.

There’s a lot of discussion about the exact length of cycle that is “optimal”, with some commentary about the windows of development, freeze, QA and so on.  I think that’s a bit of a red herring, when you factor in good branching, because feature development absolutely does not stop when the trunk is frozen in preparation for a release. Those who prefer to keep committing to their branches do so, they scratch the itch that matters most to them.

I do think that cycle lengths matter, though. Aaron speculates that a 4-month cycle might be good for a web site. I agree, and we’ve converged on a 4-month planning cycle for Launchpad after a few variations on the theme. The key difference for me with a web site is that one has only one deployment point of the code in question, so you don’t have to worry as much about update and cross-version compatibility. The Launchpad team has a very cool system, where they roll out fresh code from trunk every day to a set of app servers (called “edge.launchpad.net”), and the beta testers of LP use those servers by default. Once a month, they roll out a fresh drop from tip to all the app servers, which is also when they rev the database and can introduce substantial new features. It’s tight, but it does give the project a lot of rhythm. And we plan in “sets of 4 months”, at least, we are for the next cycle. The last planning cycle was 9 months, which was just way too long.

I think the cycles-within-cycles idea is neat. Aaron talks about how 6 months is too long for quick releases, and too short to avoid having to bump features from one cycle to the next. I’ve already said that a willingness to bump a feature that is not ready is a strength and not a weakness. It would be interesting to see if the Plasma team adopted a shorter “internal” cycle, like 2 months or 3 months, and fit that into a 6 month “external” cycle, whether Aaron’s concerns were addressed.

For large projects, the fact that a year comes around every, well, year, turns out to be quite significant. You really want a cycle that divides neatly into a year, because a lot of external events are going to happen on that basis. And you want some cohesion between the parts. We used to run the Canonical sprints on a 4-month cycle (3 times a year) and the Ubuntu releases on a six month cycle (twice a year) and it was excessively complex. As soon as we all knew each other well enough not to need to meet up every 4 months, we aligned the two and it’s been much smoother ever since.

Some folks feel that distributions aren’t an important factor in choosing an upstream release cycle. And to a certain extent that’s true. There will always be a “next” release of whatever distribution you care about, and hopefully, an upstream release that misses “this” release will make it into the next one. But I think that misses the benefit of getting your work to a wider audience as fast as possible. There’s a great project management methodology called “lean”, which we’ve been working with. And it says that any time that the product of your work sits waiting for someone else to do something, is “waste”. You could have done that work later, and done something else before that generated results sooner. This is based on the amazing results seen in real-world production lines, like cars and electronics.

So while it’s certainly true that you could put out a release that misses the “wave” of distribution releases, but catches the next wave in six months time, you’re missing out on all the bug reports and patches and other opportunities for learning and improvement that would have come if you’d been on the first wave. Nothing morally wrong with that, and there may be other things that are more important for sure, but it’s worth considering, nonetheless.

Some folks have said that my interest in this is “for Canonical”, or “just for Ubuntu”. And that’s really not true. I think it’s a much more productive approach for the whole free software ecosystem, and will help us compete with the proprietary world. That’s good for everyone. And it’s not just Ubuntu that does regular 6-month releases, Fedora has adopted the same cycle, which is great because it improves the opportunities to collaborate across both distributions - we’re more likely to have the same versions of key components at any given time.

Aaron says:

Let’s assume project A depends on B, and B releases at the same time as A. That means that A is either going to be one cycle behind B in using what B provides, or will have to track B’s bleeding edge for the latter part of their cycle allowing some usage. What you really want is a staggered approach where B releases right about when A starts to work on things.

This goes completely counter to the “everyone on the same month, every 6 months” doctrine Mark preaches, of course.

I have never suggested that *everyone* should release at the same time. In fact, at Ubuntu we have converged around the idea of releasing about one month *after* our biggest predictable upstream, which happens to be GNOME. And similarly, the fact that the kernel has their own relatively predictable cycle is very useful. We don’t release Ubuntu on the same day as a kernel release that we will ship, of course, but we are able to plan and communicate meaningfully with the folks at kernel.org as to which version makes sense for us to collaborate around.

Rather than try and release the entire stack all at the same time, it makes sense to me to offset the releases based on a rough sense of dependencies.

Just to be clear, I’m not asking the projects I’ll mention below to change anything, I’m painting a picture or a scenario for the purposes of the discussion. Each project should find their own pace and scratch their itch in whatever way makes them happiest. I think there are strong itch-scratching benefits to syncronicity, however, so I’ll sketch out a scenario.

Imagine we aimed to have three waves of releases, about a month apart.

In the first wave, we’d have the kernel, toolchain, languages and system libraries, and possibly components which are performance- and security-critical. Linux, GCC, Python, Java, Apache, Tomcat… these are items which likely need the most stabilisation and testing before they ship to the innocent, and they are also pieces which need to be relatively static so that other pieces can settle down themselves. I might also include things like Gtk in there.

In the second wave, we’d have applications, the desktop environments and other utilities. AbiWord and KOffice, Gnumeric and possibly even Firefox (though some would say Firefox is a kernel and window manager so… ;-)).

And in the third wave, we’d have the distributions - Ubuntu, Fedora, Gentoo, possibly Debian, OpenSolaris. The aim would be to encourage as much collaboration and discussion around component versions in the distributions, so that they can effectively exchange information and patches and bug reports.

I’ll continue to feel strongly that there is value to projects in getting their code to a wider audience than those who will check it out of VCS-du-jour, keep it up to date and build it. And the distributions are the best way to get your code… distributed! So the fact that both Fedora and Ubuntu have converged on a rhythm bodes very well for upstreams who can take advantage of that to get wider testing, more often, earlier after their releases. I know every project will do what suits it, and I hope that projects will feel it suits them to get their code onto servers and desktops faster so that the bug fixes can come faster, too.

Stepping back from the six month view, it’s clear that there’s a slower rhythm of “enterprise”, “LTS” or “major” releases. These are the ones that people end up supporting for years and years. They are also the ones that hardware vendors want to write drivers for, more often than not. And a big problem for them is still “which version of X, kernel, libC, GCC” etc should we support? If the distributions can articulate, both to upstreams and to the rest of the ecosystem, some clear guidance in that regard then I have every reason to believe people would respond to it appropriates. I’ve talked with kernel developers who have said they would LOVE to know which kernel version is going to turn into RHEL or an Ubuntu LTS release, and ideally, they would LOVE it if those were the same versions, because it would enable them to plan their own work accordingly. So let’s do it!

Finally, in the comments on Russell Coker’s thoughtful commentary there’s a suggestion that I really like - that it’s coordinated freeze dates more than coordinated release dates that would make all the difference. Different distributions do take different views on how they integrate, test and deploy new code, and fixing the release dates suggests a reduction in the flexibility that they would have to position themselves differently. I think this is a great point. I’m primarily focused on creating a pulse in the free software community, and encouraging more collaboration. If an Ubuntu LTS release, and a Debian release, and a RHEL release, used the same major kernel version, GCC version and X version, we would be able to improve greatly ALL of their support for today’s hardware. They still wouldn’t ship on the same date, but they would all be better off than they would be going it alone. And the broader ecosystem would feel that an investment in code targeting those key versions would be justified much more easily.

An update on the long term plans for Ubuntu release management. 8.04 LTS represented a very significant step forward in our release management thinking. To the best of my knowledge there has never been an “enterprise platform” release delivered exactly on schedule, to the day, in any proprietary or Linux OS. Not only did it prove that we could execute an LTS release in the standard 6-month timeframe, but it showed that we could commit to such an LTS the cycle beforehand. Kudos to the technical decision-makers, the release managers, and the whole community who aligned our efforts with that goal.

As a result, we can commit that the next LTS release of Ubuntu will be 10.04 LTS, in April 2010.

This represents one of the most extraordinary, and to me somewhat unexpected, benefits of free software to those who deploy it. Most people would assume that precise release management would depend on having total control of all the moving parts - and hence only be possible in a proprietary setting. Microsoft writes (almost) every line of code in Windows, so you would think they would be able to set, and hit, a precise target date for delivery. But in fact the reverse is true -  free software distributions or OSV’s can provide much better assurances with regard to delivery dates than proprietary OSV’s, because we can focus on the critical role of component selection, integration, testing, patch management and distribution rather than the pieces which upstream projects are better able to handle - core component feature development. This is in my mind a very compelling reason for distributions to focus on distribution - that’s the one thing they do which the upstreams don’t, so they need to invest heavily in that in order to serve as the most efficient conduit of upstream’s work.

We also committed, for the first time, to a regular set of point releases for 8.04 LTS. These will start three months after the LTS, and be repeated every six months until the next LTS is out. These point releases will include support for new hardware as well as rolling up all the updates published in that series to date. So a fresh install of a point release will work on newer hardware and will also not require a big download of additional updates.

Gerry Carr at Canonical put together this diagram which describes the release management plan very nicely:

The Ubuntu team does an amazing job of ensuring that one can update from release to release, and from LTS release to LTS release directly, too. I’m very proud to be part of this community! With the addition of some capability to support newer hardware in LTS releases, I think we are doing our part in the free software community - helping to deliver the excellent work of thousands of other teams, from kernel.org to GNOME and KDE, safely to a huge audience.

There’s one thing that could convince me to change the date of the next Ubuntu LTS: the opportunity to collaborate with the other, large distributions on a coordinated major / minor release cycle. If two out of three of Red Hat (RHEL), Novell (SLES) and Debian are willing to agree in advance on a date to the nearest month, and thereby on a combination of kernel, compiler toolchain, GNOME/KDE, X and OpenOffice versions, and agree to a six-month and 2-3 year long term cycle, then I would happily realign Ubuntu’s short and long-term cycles around that. I think the benefits of this sort of alignment to users, upstreams and the distributions themselves would be enormous. I’ll write more about this idea in due course, for now let’s just call it my dream of true free software syncronicity.

Hearty congratulations to the entire Ubuntu community on the successful launch of 8.04 LTS. This was our best release cycle ever, from the planning at UDS-Boston last year, at which we had many different teams and companies, to the beta process which attracted so much in the way of testing and patches. I think we can be justifiably proud of the quality of 8.04 LTS. From the code to the documentation, from translations to advocacy, this has been a team effort with the shared goal of delivering the very best free software experience to the very widest possible audience. May Hardy be both enduring and endearing.

I’m very conscious of the fact that Ubuntu is the pointy edge of a very large wedge - we are the conduit, but we exist only because of the extraordinary dedication and effort of thousands of other communities and projects. We all owe a great deal to the team who make Debian’s “unstable” repository possible, and of course to the upstream projects from GNOME and KDE through to the Linux kernel. We hope you will be proud of the condition in which we have carried your excellent work through to the users of Ubuntu.

So, well done everybody! I hope that friends, family, colleagues and others will have the opportunity to try it out and understand why we have all devoted so much to this project. Our work is deeply important - we are helping to bring free software to a new level of acceptance and adoption in the wider world.  Ubuntu’s success adds to the success of free software. So as much as it is fun, challenging, the opportunity of a lifetime, a profession for some and a passion for others, it’s also changing the world. I don’t exactly want to shout “Save the Cheerleader, Save the World” but to me you are all Heroes.

Mark

Windows is a very important platform, and our justifiable pride in Linux and the GNU stack shouldn’t blind us to the importance of delivering software that is widely useful. I believe in bringing free software to people in a way that is exciting and empowering to them, and one of the key ways to do that is to show them amazing free software running on their familiar platform, whether that’s Windows or the MacOS.

Firefox, for example, is an inspiring free software success story, and I’m certain that a key driver of that success is their excellent support for the Windows environment. It’s a quick download and an easy install that Just Works, after which people can actually FEEL that free software delivers an innovative and powerful browsing experience that is plainly better than the proprietary alternatives. I’ve noticed that many of the best free software projects have a good Windows story. MySQL and PostgreSQL both do. Bazaar works well too. And users love it - users that may then be willing to take a step closer to living in the GNU world entirely.

So, I was absolutely delighted with the way Agostino Russo and Evan Dandrea steered the Windows-native installer for Ubuntu into 8.04 LTS. What I think is really classy about it is the way it uses the Windows Boot Manager sensibly to offer you the Ubuntu option. If I was a Windows user who was intrigued but nervous about Linux, this would be a really great way to get a taste of it, at low risk. Being able to install and uninstall a Linux OS as if it were a Windows app is a brilliant innovation. Kudos to Agostino and Evan, and of course also to the guys who pioneered this sort of thinking (it’s been done in a number of different ways). It looks crisp, clean and very professional:

I’m a little daunted at something as new as WUBI being the very first experience that people have of Linux, free software and Ubuntu, but initial reports are positive.  I did have a question from the media that started with “it didn’t work for me but…” which makes me a wee bit nervous.

So - yesterday I suggested folks hammer on the Heron for servers, today, here’s a call for folks who have a Windows machine and would like to see WUBI in action to test it out and let the developers know if there are any last-minute gotchas. Happy hunting!

Reports of beta testing for 8.04 LTS seem very positive all round, to the great credit of the desktop and server teams who have been working so hard to make Hardy Heron rock. I have been running Hardy on my laptop through most of the cycle, but took the plunge on my home firewall and desktop (Kubuntu) machine this weekend.

The coolest part of the firewall upgrade is the fact that Michael has made the release upgrade tool independent of the GUI, so you can use it for server upgrades too.

So, now would be a great time to test the upgrade! File bugs if you run into any issues with your particular configuration. Apparently, this is upported on both Ubuntu 6.06 LTS (Dapper) and Ubuntu 7.10 (Gutsy), so feedback on either upgrade path would be most welcome.

sudo aptitude install update-manager-core

That should make sure you have the release upgrade tool installed. Now you can trigger the upgrade process to the current beta:

sudo do-release-upgrade --devel-release

This should fetch the latest version of the update tool, which knows about various transitions in library versions etc so that it can attempt to update your machine smoothly without leaving large amounts of dangling packages. You can say “no” if you don’t like the proposed package install and removal plan (in which case, your feedback would be very valuable!).

For fresh installs, 8.04 LTS should be good to go on any high-volume server platform available in the market today - let the server team know if you run into any problems at all. They are hoping to meet the desktop team’s “Just Works” standard, so the bar is set pretty high. From my perspective, the upgrade was smooth - full marks and my thanks to everyone involved.

The tragedy unfolding in Kenya is a reminder of the fact that a flawed election leaves the “winner” worse off than he would be losing a fair contest.

Whoever is President at the conclusion of this increasingly nasty standoff inherits an economy that is wounded, a parliament that is angry and divided, and a populace that know their will has been disregarded. And he will face a much increased risk of personal harm at the hands of those who see assassination as no worse a crime than electoral fraud. That is at best a Pyrrhic victory. It will be extremely difficult to get anything done under those circumstances.

There is, however, some cause for optimism amidst all the gloom. It seems that many Kenyan MP’s who were fingered for corruption during their previous terms were summarily dismissed by their constituencies, despite tribal affiliations. In other words, if your constituents think you’re a crook, they will vote you out even if you share their ethnicity.

That shows the beginnings of independent-minded political accountability - it shows that voting citizens in Kenya want leaders who are not tainted with corruption, even if that means giving someone from a different tribe their vote. And that is the key shift that is needed in African countries, to give democracy teeth. Ousted MP’s and former presidents are subject to investigation and trial, and no amount of ill-gotten loot in the bank is worth the indignity of a stint in jail at the hands of your successor. As Frederick Chiluba has learned, there’s no such thing as an easy retirement from a corrupt administration.

Of course, that makes it likely that those with skeletons in their closets will try even harder to cling to power, for fear of the consequences if they lose their grip on it. Robert Mugabe is no doubt of the opinion that a bitter time in power is preferable to a bitter time after power. But increasingly, voters in Africa are learning that they really can vote for change. And neighboring countries are learning that it hurts their own investment and economic profiles to certify elections as free and fair when they are far from it. It would be much harder for Robert Mugabe to stay in power illegally if he didn’t have *nearly* enough votes to stay there legally. You can fudge an election a little, but it’s very difficult to fudge it when the whole electorate abandons you, and when nobody will lend your their credibility.

The best hope a current president has of a happy retirement is to ensure that the institutions which will pass judgement on him (or her) in future are independent and competent, to ensure that they will stay that way, and to keep their hands clean. It will take time, but I think we are on track to see healthy changes in governance becoming the norm and not the exception in Africa.

Yesterday, we saw the most extraordinary failure of economic leadership in recent years, when the US Federal Reserve pressed the “emergency morphine” button and cut Federal Reserve rates by 0.75%. It will not help.

These are extremely testing times, and thus far, the US Fed under Bernanke has been found wanting. Historians may well lay the real blame for current distress at the door of Alan Greenspan, who pioneered the use of morphine to dull economic pain, but they will probably also credit him with a certain level of discretion in its prescription. During Greenspan’s tenure at the Fed, economic leaders became convinced that the solution to market distress was to ensure that the financial system had access to easy money.

This proved effective in the short term. When LTCM looked set to explode (private investments, leveraged up dramatically, managed by Nobel prize-winning financial theorists, placed a bet on a sure thing which didn’t pan out quite as expected) Greenspan engineered an orderly unwinding of its affairs. When the dot com bubble burst, Greenspan kept the financial system energised by lowering rates so far that they were, for a substantial period, at negative levels.

A negative real interest rate means we are effectively paid to take out loans. That might sound good, but how would you feel if I used the words “paid to take a few more hits of crack cocaine”? The underlying problem was that people had become accustomed to high rates of return and did not want to accept that real rates of return in the US were moving down. They had become accustomed to easy money, and Greenspan’s policy ensured that money remained accessible at a time when people had demonstrated a low ability to invest that easy money well.

Low rates give people an incentive to invest in stocks, even if those stocks are not earning very much. This meant stock prices recovered quickly, and the effect was amplified by the fact that low rates increased corporate earnings. This was a so-called “soft landing” - disaster averted. He must have known the risks, but the one big warning sign that would likely have convinced Greenspan to return to normal rates was missing: inflation. Low rates, and especially negative rates, have historically always resulted in inflation. Greenspan kept rates low because there were no signs of inflation. It seemed as if the US had entered a new era where the correlation of rates and inflation no long held true. People explained it by saying that the US was increasing its productivity dramatically (productivity increases are like anti-inflation medicine). Now, with hindsight, it appears that the real reason for the absence of inflation was that the Chinese were increasing their productivity dramatically, and that US consumers were spending so much on Chinese goods that Chinese productivity growth, not US productivity growth, was keeping US prices low.

When tech came off the boil and people should have been using the pause to clean up their affairs, Greenspan made it easy for people to get themselves into a worse position. Easy money made stock market prices artificially high, so stock market investors felt rich. Worse, easy money made house prices artificially high (by about 45%), so everybody felt wealthier than they had planned or expected to.

To make matters worse, a series of financial innovations created a whole industry designed to help people go back into debt on their houses. I remember trying to watch TV in the US and being amazed at the number of advertisements for “home equity withdrawals”. They made it sound like turning your major personal financial asset - your paid-off house - into an ATM machine was a good thing. In fact, it was a means to spend all of your primary store of wealth. And with inflated house prices, it was a way to spend money that you did not really have. A convenient way to get into a deep, dark hole of family debt. The result? The average American owns less of her home today than  she did 30 years ago - 55% as opposed to 68%. Easy money makes people poorer.The company with the most irritating ads, Ditech (and I feel ashamed to be contributing to their website search ranking with the mention, perhaps it will help instead to link to their customer feedback), has a tagline “People are smart” and a business model built on the idea that “People are dumb”. Their “most popular” product strikes me as being tailor-made to make it easy to turn home equity - an asset - into new debt.

Why did Greenspan do it? I think he genuinely believed that there was something different about the modern world that had altered the laws of economic gravity. I suspect he no longer feels that way.

But Greenspan is no longer Chairman of the Fed. Ben Bernanke blinked, yesterday, and in that blink we have the measure of the man.

Greenspan acted carefully, logically, and basically prudently. Several years of anomalous economic data are a reasonable basis to think that the rules have evolved. You would have to have a Swiss (700 years of stability) or Chinese (”we think it’s too early to tell if the French Revolution was a good idea”) approach to stick with economic theories that are at odds with the facts for very long. Greenspan made a mistake, and it will have huge consequences for the US for a generation, but he had reasons for that mistake. Bernanke just blinked, he panicked, despite knowing better.

We now have rigorous economic explanations for all that is happening. We have come to understand, quite clearly, what is going on in the world. The deflationary Eastern wind has been identified. We know there is no productivity miracle in the US, no change in the laws of physics or economics. So we know that the US patient is addicted to easy money morphine, medicine that was prescribed with good intentions by Dr Greenspan, medicine that has in the last 7 years made the patient more ill and not less. More morphine today constitutes malpractice, not economic innovation. We know the consequences of more morphine - stock prices will rise artificially (4% yesterday, on the news of the shot), house prices will stumble along, companies will take longer to default on their loans.

Bernanke might be hoping to do what Greenspan did - retire before the addiction becomes entirely obvious. Too late. While the Fed is clearly not willing to admit it, the markets have just as clearly taken their own view, that the prognosis is not good. They are smart enough to see that all Bernanke has done is cover up the symptoms of malaise, and many are using the temporary pain relief to head for safer territory. I expect that any relief will be brief, market recoveries will  fade, the rout has been deferred but not averted.

I started out by describing the Fed’s actions as a failure of economic leadership. Some folks are lucky enough to lead from the bottom of the cycle, up - they take over when things are miserable and can only really get better. They look like heroes even if their voodoo has no mojo, so to speak. Others are less lucky, they get handed custodianship of an asset that is at the peak. As for Bernanke, he’s in that latter category. He needs to be able to speak clearly and frankly about the hard work that lies ahead in the US. He needs to appeal to the very best of American industriousness - a traditional willingness to work hard, be smart, and accept the consequences of refusing to do so. He needs to lead under the most difficult circumstances. But that’s what leadership is about.

Fortunately for Bernanke, central bank independence is widely believed to be the only credible approach to economic governance. That independence gives Bernanke the right to stand at odds with political leaders if needed. Given the recent White House announcements - more morphine, further indebtedness for the worlds most indebted country - there’s no stomache for a real program of rehabilitation in the Bush Administration. Bernanke will have to lead without political support, a very difficult task indeed. Our greatest and most memorable leaders are those who lead through difficult times. The same is true of failures of leadership. Appeasement, or rehabilitation. Chamberlain, or Churchill. Thus far, Chamberlain.

O’Reilly, the organisers of Ubuntu Live, have just issued the call for papers for Ubuntu Live 2008. The theme of the event is “Taking it Further”, which I think is perfect for Ubuntu this year!

The subtitle to the conference should probably be “Going into production at scale”, because it seems everywhere I look these days people are taking Ubuntu into production. Perhaps it’s the preparation for the April LTS release, perhaps its that more and more of their apps and solutions are certified on Ubuntu, or perhaps its just that confidence in Ubuntu for large-scale deployments on the server and the desktop has reached a tipping point, but either way I’m delighted with the ramping up of heavy-duty adoption of the platform that our community delivers with such metronomic precision.

So Ubuntu Live 2008 promises to be informative, as we start to reap the benefits of that experience. If you have interesting deployments or projects that you would like to share, UL2008 would be the right platform to do it! I’d be particularly interested in talks that describe:

• large-scale government deployments of Ubuntu on the desktop (there have now been several)
• specialist deployments, for example high-performance computing clusters, or vertical market solutions
• virtualisation-based deployments where Ubuntu is the host or the guest platform
• large-scale server farms for hosting or web edge-of-the-network deployments
• appliances based on Ubuntu

You can submit a proposal directly or read more about the conference. Hope to see you there!

I completely failed to blog the release of Bzr 1.0 last year, but it was an excellent milestone and by all accounts, very well received. Congratulations to the Bazaar community on their momentum! I believe that the freeze for 1.1 is in place now so it’s great to see that they are going to continue to deliver regular releases.

I’ve observed a surge in the number of contributors to Bazaar recently, which has resulted in a lot of small but useful branches with bugfixes for various corner cases, operating systems and integrations with other tools. One of the most interesting projects that’s getting more attention is BzrEclipse, integrating Bzr into the Eclipse IDE in a natural fashion.

I think open source projects go through an initial phase where they work best with a tight group of core contributors who get the basics laid out to the point where the tool or application is usable by a wider audience. Then, they need to make the transition from being “closely held” to being open to drive-by contributions from folks who just want to fix a small bug or add a small feature. That’s quite a difficult transition, because the social skills required to run the project are quite different in those two modes. It’s not only about having good social skills, but also about having good processes that support the flow of new, small contributions from new, unproven contributors into the code-base.

It seems that one of the key “best practices” that has emerged is the idea of plug-in architectures, that allow new developers to contribute an extension, plug-in or add-on to the codebase without having to learn too much about the guts of the project, or participate in too many heavyweight processes. I would generalize that and say that good design, with clearly though-through and pragmatic layers, allow new contributors to make useful contributions to the code-base quickly because they present useful abstractions early on.

Firefox really benefited from their decision to support cross-platform add-ons. I’m delighted to hear that OpenOffice is headed in the same direction.

Bazaar is very nicely architected. Not only is there a well-defined plug-in system, but there’s also a very useful and pragmatic layered architecture which keeps the various bits of complexity contained for those who really need to know. I’ve observed how different teams of contributors, or individuals, have introduced whole new on-disk formats with new performance characteristics, completely orthogonally to the rest of the code. So if you are interested in the performance of status and diff, you can delve into working tree state code without having to worry about long-term revision storage or branch history mappings.

Layering can also cause problems, when the layers are designed too early and don’t reflect the pragmatic reality of the code. For example, witness the “exchange of views” between the ZFS folks and the Linux filesystem community, who have very different opinions on the importance and benefits of layering.

Anyhow, kudos to the Bazaar guys for the imminent 1.1, and for adopting an architecture that makes it easier for contributors to get going.

I’m absolutely thrilled to see this chart of untriaged bugs in Inkscape since the project moved to Launchpad:

As you can see, the Inkscape community has been busy triaging and closing bugs, radically reducing the “new and unknown” bug count and giving the developers a tighter, more focused idea of where the important issues are that need to be addressed.

A lot of my personal interest in free software is motivated by the idea that we can be more efficient if we collaborate better. If we want free software to be the norm for personal computing software, then we have to show, among other things, that the open, free software approach taps into the global talent pool in a healthier, more dynamic way than the old proprietary approach to building software does. We don’t have money on our side, but we do have the power of collaboration.

I put a lot of personal effort into Launchpad because I love the idea that it can help lead the way to better collaboration across the whole ecosystem of free software development. I look for the practices which the best-run projects follow, and encourage the Launchpad guys to make it easy for everyone to do those things. These improvements and efficiencies will help each project individually, but it also helps every Linux distribution as well. This sort of picture gives me a sense of real accomplishment in that regard.

Bryce Harrington, who happens to work for Canonical and is a member of the Inkscape team, told me about this and blogged the experience. I’ve asked a few other Inkscape folks, and they seem genuinely thrilled at the result. I’m delighted. Thank you!