Search This Blog


Sunday, December 23, 2007

Customers May Not Want Frequent Releases

Damon Poole made arguments on why software needs to be released often.

Product quality is an ongoing effort. No matter how much testing you do, users will find ways to use your product that find holes in your test plan. For any change, whether it is a bug fix or a new feature, you don't really know the impact until customers start using it. That means that the maturation of a change doesn't really start until the change ships. If you have a long release cycle, you will have changes that are made early that are basically gathering dust until you ship. You can't really say that they are tested until you test the whole product as it will ship, so again, true maturation doesn't start until you ship.

your customers would still be exposed to the same number of bugs, just not all at the same time (note: meaning the shorter release cycle, the shorter the exposure time).
Indeed, frequent release is good for software developers, and the software development process precisely because it produces more more frequent feedback to the developers so that they can further fine-tune their applications.

But what is good for the developers may not be good for the customers, if you are working on desktop applications or installable web applications.

This is because the benefits of frequent updates come at the customer's expense.

Before your app is useful, it must be
  1. downloaded
  2. Installed
  3. troubleshooted if installation went wrong.
All of these steps take time. Users are very impatient over the download and installation time. And new versions incur risks as well. What if the downloads are not successfully? What if anything goes wrong with the installation? What if after the installation the app mysteriously stop working properly?

The last question deserves some elaboration. By saying that an app stop working properly I do not mean that there is a hang or crash: that would be easy to fix and should have been weeded out before the app is released. What I mean was that suddenly the app consumes too much memory, or too much CPU power to a point that your computer are barely crawling when you are using the app, or that the app is working fine but gradually becomes slower and slower until a point when you have to restart your pc before it starts working again. These problems are very tricky, and on a large scale application it is not easy to diagnose the problem. Are you, as a system administrator or a plain user willing to give up a working version in favor of a new one in the face of these risks?

We the users want to do our jobs with an app. We don't want all the associative cost of it. The more frequent your updates are, the more trouble we have to take to upgrade. And if the updates are too frequent or the costs are too high, we will simply just give up the updates.

That was my experience with Adobe Reader. Starting from a version ( I can't remember which), Adobe Reader implemented a feature called automatic update notification. You can choose whether to turn on the notification or not. So, every time a new version was released, my computer would get the notification and the download silently. By the time it finished, the Adobe Reader downloader would prompt me for an install.

After the installation, a reboot.

I hated it so much because I didn't just work on one Personal Computer. I worked on multiple machines and some of them were build machines that couldn't be shut down at will. The updates were more of a burden than a blessing.

And what were the benefits? Only fanciful features that I didn't need. The product was stable enough and I didn't need bug fixes at all. From economical sense, the upgrades did not increase my utility at all, in fact, they decreased it.

There is a point of diminishing return ( before negative return) with frequent releases as far as the users are concerned.

So software release management is tricky.

You should not release too often if your software is not stable, in fact you shouldn't release at all until you can have a working version.

You cannot release too often if your software is stable, this is because downloading and installing it takes away customer's time and energy. Eventually they will simply skip releases and defeat the purpose of frequent releases.

But you should release often because it will encourage the feedback loop and makes your app better.

To release or not to release, that is the question.


Adrian Tarau said...

That's why my framework provides automatic updates, it doesn't matter if is a desktop, client/server(actually this one is updated outside the framework, with Java Web Start) or a web application.

You can read about my project here :

I'm working on documentation, I hope I will have something usable in a few weeks.

Jonas said...

I think you're missing the point. You're talking about updates being applied/installed at a certain customer, but the "frequency" discussed is that of releases from the producer.
The key point is that the time between releases is the minimum time you can react to new requests (the agile way) so you should make production ready releases often. There is no need for any customer to use every release, it's up to them to determine the ROI on installation (as it does entail the steps you list and must provide enough benefits to motivate that cost).

BTrey said...

Obviously, the more people who download and use your release, the more eyes you have looking for bugs and issues. But it doesn't follow that the release is useless if not everyone using your product upgrades. Many shops routinely run a few versions behind current. They upgrade at their own schedule, not at the schedule you set by your releases. There's no negative to your users that arises from you making updates available. You make them available as often as is useful to you, and allow them to schedule their upgrades when it's most useful to them. So long as at least one of your customers upgrade and use your new release, you gain benefit even if any particular customer only upgrades once every few releases.

Jeff Santini said...

BTrey's statement is related to the "constant Beta" concept. JetBrains for example almost always has an EAP active for Intellij or Resharper. That way people who are interested in the latest and greatest can have it, but those who want to work on production releases only can stay away from EAP(Early Access Program) and cruise on the stable releases. This gives the best of both worlds.

By the way I was uncomfortable with the idea implied in your post that developers release to please developers. If you are not releasing for the benefit of your customers then there is a problem more fundamental than the release schedule itself.