One of the things that I do quite frequently is download new packages and bits and pieces of code to play with on my local machine. I frequently start making local edits to perform local configuration changes, or perhaps try out the odd idea. Quite often these packages are downloaded, played with, and then discarded if they don’t meet my needs. But of course some of them end up with a permanent place in my tool chest, or installation.

Exploring with Bread Crumbs

I start getting twitchy if I start making changes to anything without checking them in somewhere as I go. It is all too easy otherwise to lose track of the changes you have made and quickly get yourself into a mess, losing lots of time and effort.

So, like Hansel and Gretel, I prefer to leave a trail of bread crumbs along the way so that I can explore safely (avoiding if possible my breadcrumbs being eaten by the birds!).

I view this as being able to explore while having the safety net of my saved versions stored away. Advantages:

• I know what I have changed at any point (can show what’s changed and do diffs)
• I have a bread crumb trail leading me back home to my initial clean configuration
• I can revert to known working states if the current experiment goes wrong (e.g. accidentally delete chunks of text in the editor, or change several settings at once without appreciating their interactions)

Bread Crumb Trail Tools

My tool of choice for SCM is Perforce which is fast, effective and has lots of client tools. Having used it for many years as a consultant and trainer it is second nature to me and quick and easy to use, and rock solid.

And yet, for this particular type of work, I find it not always  ideal. Why?

• I need to setup a client workspace – not difficult, but enough steps to still be annoying and to often result in me not doing it (requires making decisions on naming, default paths in the repository etc – seems like it could be easily automated, but just slightly too variable in requirements to make this easy)
• The changes are “permanent” in my repository – even if the whole experiment turns out to have been a red herring (and yes of course I can obliterate stuff, but that’s another extra step)

Please note that if I really want to keep all my work – for example the package turns out to be of long term use, then I make the small extra effort and import into Perforce (together with third party codeline pattern etc to be able to track future changes as new release are made etc.). At this point, it turns out that saving it “centrally” is very beneficial.

Bread Crumb Tool of Choice

My favourite current tool for this “temporary” bread crumb saving is Mercurial (hg).

The advantages for me:

• It is very quick and easy (but also a full featured system should I ever need it)
• The “repository” is saved in the .hg directory at the root
• If I remove the whole tree then repository goes too

I use the command line client to save an initial snapshot (from the root of the tree where I have extracted the package):

hg init .
hg add .
hg commit -m "Initial version of XXX as downloaded"

Subsequently I typically use the following subset of commands (from “hg help”):

• add add the specified files on the next commit
• commit       commit the specified files or all outstanding changes
• copy         mark files as copied for the next commit
• diff         diff repository (or selected files)
• help         show help for a given topic or a help overview
• log          show revision history of entire repository or files
• remove       remove the specified files on the next commit
• revert       restore individual files or directories to an earlier state
• status       show changed files in the working directory

TortoiseHg is also useful.

Conclusion

Perforce remains my tool of choice for most SCM related activity, but Mercurial is a very useful addition to my personal tool chest, and in particular in this type of scenario.

The main thing I would always encourage people: Whatever you do, get into a habit of using a version control tool as often as you can! You will seldom regret it.

I was interested to see the post about Flickr’s developing new code on the mainline and using flags and flippers to control it.

Obviously an alternative to this approach is to use feature branches – so developing longer lived features off the mainline and only merging it back in when there you want to “turn the feature on”.

As Ross Harmes says though, “This style of development isn’t all rainbows and sunshine.”

So what are some of the tradeoffs requiring consideration before adopting this approach?

Pros

• Avoids multiple branches and thus merging – makes developer’s lives easier in that they know they are only in a single branch (mainline) and they need to keep that working
• Single codebase – force people to “fix forwards” rather than “roll back” changes.
• According to Ross,  ”Deploys become smaller and more frequent; this leads to bugs that are easier to fix, since we can catch them earlier and the amount of changed code is minimized.”

Cons

• Lots of “if … else …” constructs complicate the code base. As he mentions “after launching a feature, we have to go back in the code base and remove the old version (maintaining separate versions of all features on Flickr would be a nightmare)“
• Makes testing more complex with all these configurations needing to be tested
• If you are not careful, can lead to a certain amount of “copy and paste” coding as opposed to DRY style. This can require significant discipline to avoid.

As regards the “Deploys become smaller and more frequent”, this is possible to achieve with other methods, including task branches or the equivalent. Indeed I would say that making deploys smaller and more frequent is a desirable thing to do in any case.

The phrase “remove the old version” is very important – and implies a suitable amount of refactoring post release (and the discipline to make sure that happens). One of the things that I have learnt to appreciate more and more is the ability and the willingness to delete unneeded code (with the safety net of old versions being stored in my repository). You may comment it out line by line (a quick /* … */ around a block, while easy to do is very easy to miss when reading the code), but it’s amazing how quickly redundant code starts to slow you down and consume energy and brain cycles – especially for new people to that code. Keeping a clean code base through good refactoring has proven to be a very effective agile development practice for many people.

I have seen this flag approach used successfully in the past, but also seen it start to become a morass of spaghetti code when there are too many options, or when new features interact with each other (which is where “copy and paste” starts to become attractive, inspite of all its myriad failings and creation of maintenance problems). This is particularly true of “products” where there are lots of customisable options which need to be kept available to customers in the code base for years. Addressing this particular problem in a maintainable way requires a lot of thought and effort. Codebases in C/C++ with multiple #ifdefs can become a big problem if not very carefully managed. I remember seeing an excellent internal paper within Symbian on the half dozen potential approaches to introducing variability within their codebase for mobile phone systems (a harder problem than for many due to needing to provide source to manafacturers – so code changes are visible to others). The options ranged from #ifdef (very seldom considered as valid) to modular components to ordinary conditional statements.

A similar danger story was a company producing financial software for 15 different customers, with multiple different options, all from the same “codebase”, and indeed from the same workspace/development environment  (it had a database and 4GL involved which made it hard to create separate workspaces). If a customer wanted a new feature, they got it, but they were also forced to take 20 other features or bug fixes that had been checked in on the mainline and were requested by other customers. It often took weeks to stabilise things. Luckily things have improved substantially since then for that particular company.

This whole area has resulted in approaches and tools to address the problem – search for “software product lines” and “variant management” for some links. A friend of mine, Mark Dalgarno, runs Software Acumen who consults and sells products in this arena.

In the Flickr case, it would be interesting to get some metrics as to how many “features in progress” they have on the go at any one time, and how their development team tends to be split amongst features  (how many people per feature).

In the absence of such metrics to help people really understand the issues fully, I hope that the Flickr post won’t be a siren call which lures unsuspecting software developers on to a new set of rocks. As always, learn from others but make sure they are solving the same problem that you have to solve.

So, time to turn an early new year’s leaf over on the blogging front!

I have been struggling with some motivational issues as to making sure to keep updating the blog. Part of the reason for that is the tools I was using for maintaining the blog. The problem was that the tools I had and the interface to them were such that it became a bit of a chore to write new articles, upload them etc. Net result – I didn’t feel the temptation to blog much!

One of the reasons I went with Rublog (original blogging framework) is that I wanted to ensure that all my posts were appropriate version controlled – it is against my principles to have things “only” stored in a database when I am used to full version control (and various tools for seeing history of articles being stored).

Various other issues came to the fore as regards new features, comments etc. Having looked around, I realised that I had also been suffering from some code snobbery – my personal lack of enthusiasm for PHP (and preference for Ruby/Python), leading me to look askance at PHP based solutions. But tools are tools, and market share is as important in the blogging tool market as the SCM market – the more people using your tool, the greater the enthusiasm and availability of good options, addins, plugins etc.

So, the result is a conversion to WordPress. A bit of help from the redirection plugin and all the older articles should still be available!

The proof of the pudding as regards updates and the benefits of the new tool is in the eating, so watch this space for regular updates – keep me honest!!