I was interested to see the post about Flickr’s developing new code on the mainline and using flags and flippers to control it.

Obviously an alternative to this approach is to use feature branches – so developing longer lived features off the mainline and only merging it back in when there you want to “turn the feature on”.

As Ross Harmes says though, “This style of development isn’t all rainbows and sunshine.”

So what are some of the tradeoffs requiring consideration before adopting this approach?

Pros

• Avoids multiple branches and thus merging – makes developer’s lives easier in that they know they are only in a single branch (mainline) and they need to keep that working
• Single codebase – force people to “fix forwards” rather than “roll back” changes.
• According to Ross,  ”Deploys become smaller and more frequent; this leads to bugs that are easier to fix, since we can catch them earlier and the amount of changed code is minimized.”

Cons

• Lots of “if … else …” constructs complicate the code base. As he mentions “after launching a feature, we have to go back in the code base and remove the old version (maintaining separate versions of all features on Flickr would be a nightmare)“
• Makes testing more complex with all these configurations needing to be tested
• If you are not careful, can lead to a certain amount of “copy and paste” coding as opposed to DRY style. This can require significant discipline to avoid.

As regards the “Deploys become smaller and more frequent”, this is possible to achieve with other methods, including task branches or the equivalent. Indeed I would say that making deploys smaller and more frequent is a desirable thing to do in any case.

The phrase “remove the old version” is very important – and implies a suitable amount of refactoring post release (and the discipline to make sure that happens). One of the things that I have learnt to appreciate more and more is the ability and the willingness to delete unneeded code (with the safety net of old versions being stored in my repository). You may comment it out line by line (a quick /* … */ around a block, while easy to do is very easy to miss when reading the code), but it’s amazing how quickly redundant code starts to slow you down and consume energy and brain cycles – especially for new people to that code. Keeping a clean code base through good refactoring has proven to be a very effective agile development practice for many people.

I have seen this flag approach used successfully in the past, but also seen it start to become a morass of spaghetti code when there are too many options, or when new features interact with each other (which is where “copy and paste” starts to become attractive, inspite of all its myriad failings and creation of maintenance problems). This is particularly true of “products” where there are lots of customisable options which need to be kept available to customers in the code base for years. Addressing this particular problem in a maintainable way requires a lot of thought and effort. Codebases in C/C++ with multiple #ifdefs can become a big problem if not very carefully managed. I remember seeing an excellent internal paper within Symbian on the half dozen potential approaches to introducing variability within their codebase for mobile phone systems (a harder problem than for many due to needing to provide source to manafacturers – so code changes are visible to others). The options ranged from #ifdef (very seldom considered as valid) to modular components to ordinary conditional statements.

A similar danger story was a company producing financial software for 15 different customers, with multiple different options, all from the same “codebase”, and indeed from the same workspace/development environment  (it had a database and 4GL involved which made it hard to create separate workspaces). If a customer wanted a new feature, they got it, but they were also forced to take 20 other features or bug fixes that had been checked in on the mainline and were requested by other customers. It often took weeks to stabilise things. Luckily things have improved substantially since then for that particular company.

This whole area has resulted in approaches and tools to address the problem – search for “software product lines” and “variant management” for some links. A friend of mine, Mark Dalgarno, runs Software Acumen who consults and sells products in this arena.

In the Flickr case, it would be interesting to get some metrics as to how many “features in progress” they have on the go at any one time, and how their development team tends to be split amongst features  (how many people per feature).

In the absence of such metrics to help people really understand the issues fully, I hope that the Flickr post won’t be a siren call which lures unsuspecting software developers on to a new set of rocks. As always, learn from others but make sure they are solving the same problem that you have to solve.