How do you cope with change in open source frameworks that you use for your projects?

Question

It may be a personal quirk of mine, but I like keeping code in living projects up to date - including the libraries/frameworks that they use. Part of it is that I believe a web app is more secure if it is fully patched and up to date. Part of it is just a touch of obsessive compulsiveness on my part.

Over the past seven months, we have done a major rewrite of our software. We dropped the Xaraya framework, which was slow and essentially dead as a product, and converted to Cake PHP. (We chose Cake because it gave us the chance to do a very rapid rewrite of our software, and enough of a performance boost over Xaraya to make it worth our while.)

We implemented unit testing with SimpleTest, and followed all the file and database naming conventions, etc.

Cake is now being updated to 2.0. And, there doesn't seem to be a viable migration path for an upgrade. The naming conventions for files have radically changed, and they dropped SimpleTest in favor of PHPUnit.

This is pretty much going to force us to stay on the 1.3 branch because, unless there is some sort of conversion tool, it's not going to be possible to update Cake and then gradually improve our legacy code to reap the benefits of the new Cake framework. So, as usual, we are going to end up with an old framework in our Subversion repository and just patch it ourselves as needed.

And this is what gets me every time. So many open source products don't make it easy enough to keep projects based on them up to date. When the devs start playing with a new shiny toy, a few critical patches will be done to older branches, but most of their focus is going to be on the new code base.

How do you deal with radical changes in the open source projects that you use? And, if you are developing an open source product, do you keep upgrade paths in mind when you develop new versions?

Answer 1

The issue isn't unique to open source. The same issues occur with commercial projects. Maybe even more so because you don't have source you can maintain yourself if the company drops support.

End user type open source projects have rapid upgrade cycles and old versions lose support very quickly. On the other hand, projects that are designed for users to put significant effort coding on top of tend to have long support cycles of the old branch after a major upgrade. Long enough for you to take your time first waiting for it to stabilize and mature, then migrating and thoroughly testing your own code.

It can be difficult resisting the temptation to have the latest and greatest, but realize there is a fundamental tradeoff in software between stability and new features. You can't have one without sacrificing the other. That's why bugfix branches exist, so you can choose what side of the spectrum best fits your needs.

Answer 2

We've dealt with this problem by modularising our code.

Our primary website was built around eight years ago, and we were lucky enough for it to be built by one of the early contributors to the Spring Framework so it was a pretty well-built foundation. But unfortunately we were also unlucky enough that this developer decided to fork Spring after an argument about the direction it was going, so this codebase is now stuck on an unmaintained fork of Spring 1.1.4 (or something of that vintage).

We tried to rewrite the code to move to new versions of Spring, but that proved extremely difficult. So our solution was to start building new features of the site as modules in a completely separate application, using the latest frameworks such as Spring 3.x, Hibernate 3.6, etc.

Now we have two web applications running on the same base URL, and we use Apache's mod_proxy module to allocate each path to the appropriate application. For example, "/members" goes to the old application and "/directories" goes to the new application. However a visitor to the site would have no idea that they're using different applications; they look exactly the same to the user.

The two applications need to communicate, for example when a member logs in to the site (using our new application) we need to notify the old application that they're now logged in. We do this using a simple web service, exposed only to localhost. The amount of integration needed between the two apps turned out to be minimal.

An important part of making this work was to identify and package together shared assets. So all static text, graphics, stylesheets, Javascript and templates were moved out of the original application into a separate - third - project that both the new and old applications use. This ensures that both web applications look and act the same way even though they're completely separate.

I imagine that over time we'll replace more and more of the original application until eventually it's completely unneeded. The nice thing about this technique is that we can do it slowly via a series of small, low-risk changes -- there is no need for a massive and risky sudden switchover to a new system.

Answer 3

And this is what gets me every time.

Every time? You must be making remarkably bad choices. There must be one example where it didn't happen.

When the devs start playing with a new shiny toy

That's a hint. Avoid shiny new toys when using open source projects. Avoid release 1.0.

How do you deal with radical changes in the open source projects that you use?

Step 1. Choose open source projects very, very carefully. Always compare two or more competing projects.

Step 2. When comparing two competing projects, try to understand the "essential" feature that is offered and how they both approach it. Avoid "marrying" one specific API too early.

Step 3. Embrace the design principles of "Late Binding" and "Loose Coupling". Try to insulate from open source project changes.

Step 4. Explicitly make cost/benefit comparisons between open source projects and "rolling your own". Once in a while, creating your own solution may be better than coping with an open source solution.

Changing the file names shouldn't be too hard. Yes, it's a big, ugly script. Yes, it has to be run for several weeks while doing the conversion. But it's a finite cost.

If if happens every time, then develop better coping strategies. Since your observation of the real world is that it's always going to happen, then hoping for the real world to change isn't really going to help much. You have some hard-won experience in change. Leverage that. Consider it like an infection and develop your own immune response.

Answer 4

I don't always do it. A lot of open source projects have active maintenance branches maintained for previous major releases. Sometimes those are maintained by people who need that version to be maintained. A lot of projects stay on a major release until they themselves are ready for a major release.