Robert Cowham's Musings

One of the the things that comes up just about every time I go in to do some Perforce related consulting for a client is that of repository structure – how do you map some particular branching scheme (combination of branching patterns) onto the repository?

This of course necessary since Perforce uses the repository structure to represent branches as well as other directory structures. This way of recording branches is feature one might occasionally wish were implemented otherwise, but it does have its attractions, and is the same model as used in Subversion and also Microsoft’s new Team Foundation Server product (as an aside, it is perhaps not surprising that Microsoft chose this model since it is a not very well kept “secret” that their in house SCM tool called SourceDepot, introduced during the development of Windows 2000, was a re-badged copy of Perforce. Team Foundation Server is obviously a total rewrite on top of SQL Server, but they seem to be comfortable with the “branch in path space” model).

Some clients need help working out a good branching scheme, and some have relevant experience with other tools so are comfortable with that part, but many need help with the mapping. I have covered a little of this in my article Introduction to Perforce Branching, but this article expands on that.

For their flexibility, I would generally recommend people use branch specs (e.g. created by “p4 branch” command) for branching since you can add view mappings to address not propagating renames, and branch extra components. However, if you can keep the branch spec to a single view mapping then that is great! (Note that the P4V GUI allows integration via drag-and-drop – sometimes convenient, but can be somewhat dangerous…).

Guidelines

A repository branch structure is about creating an appropriate information hierarchy (designing an information architecture). So here are some guidelines that I have found useful over the years and seen successfully used in many organisations (and I certainly didn’t invent any of them!). Maybe in the future I can come up with some appropriate pattern like descriptions, but for now, here they are:

• Standardise naming conventions – make branch directories obvious
• Keep branch spec mappings and client workspace spec mappings as simple as possible (and permissions)
• Make it as easy as possible for users (new and experienced) to visualise the branching scheme as they click down through the hierarchy
• Find the right balance between broad & shallow vs. narrow & deep hierarchies
• Keep branches of a similar type at similar depths in the structure
• Make branch directories sort sensibly
• Plan for the future: 2, 3, 5 years down the line – include things like the year (and optionally month) somewhere in the path to provide a natural sorting.
• Provide Guidelines and Delegate!
• Remember – there is no one best solution, but there are many good enough solutions!

Some of what comes below is also covered by Laura Wingerd in her recent book “Practical Perforce” (O’Reilly) – you really need to get this book if you are responsible in any way for Perforce usage at your company! Her quote which sums things up nicely is:

“There is no reason that the repository structure should match the release lineage, but when it doesn’t it causes confusion to new users [and cognitive dissonance to all users].”

Now while I agree with the intent of what Laura has written, I find her suggested option is perhaps a little too restrictive, and it doesn’t take into account organisations with many branches at a particular level (and indeed many different types of branches).

Standardise Naming Conventions – Make Branch Directories Obvious

Laura gives a very simple example in her book of naming convention – just use uppercase for the branch component. Thus she has:

//depot/MAIN/...
//depot/REL1/...
//depot/SOME_PROJECT/...
//depot/ANOTHER_PROJECT/...

This is simple and can work very well (although it rather suggests that you don’t use uppercase for any other paths which might be unnatural in certain companies for certain items). Another alternative is to append (or prepend) something to the name indicating that it is a branch directory, e.g. “.branch” or or “-branch”, or “.br” for those who don’t like typing! Thus the above becomes:

//depot/main.br/...
//depot/rel1.br/...
//depot/some_project.br/...

I am happy to have other suggestions recommended!

Sensible Sorting

Directory (and thus branch) names should sort sensibly at any point in the hierarchy, thus

• use leading zeros for numeric components, such as 01 instead of 1 if you are going to have more than 10, or indeed 4, 5 or 6 leading zeroes where necessary
• reverse the order of dates using yyyy-mm rather than mm-yyyy

Examples:

//depot/main.br/...
//depot/rel_01_00.br/...
//depot/rel_02_00.br/...
//depot/rel_02_01.br/...
//depot/rel_02_02.br/...

//depot/dev/2007_06_some_project.br/...

Broad & Shallow vs. Narrow & Deep

At any one level the number of choices (sub-directories/branches) should be appropriate and naturally limiting (note that the structure needs to be as useable in several years time as it is now). For example, a structure which ends up with thousands of branches in a single directory will become unwieldy over time. Divide these branches into sub-directories using appropriate naming (and sorting). For example if you wish to create lots of branches for individual change requests then the obvious structure is very wide and shallow at some point in your repository:

~/rel 	/CR000001/...
	/CR000002/...
	:
	/CR000099/...
	/CR000100/...
	:
	/CR009100/...
	:

Instead of that, try:

~/rel 	/CR0000xx	/01/...
			/02/...
			:
			/99/...
	/CR0001xx	/01/...
			:
	/CR0091xx/	/01/...
			:

This naturally splits things up and also sorts at all levels.

As a corollary of the above you should split things up even if they don’t have a “natural” dividing component. One alternative is to use the date as an element of the path (or even a prefix) as YYYY or YYYY-MM to give some sort of natural sorting by date. Examples being:

~/2006-01/some_project/...

Or without introducing another level but using a prefix (which is often just a matter of personal preference):

~/2006-01-another_project/...

In 2 or 5 years time it should be obvious to any user which are the which are the “new” ones where they are likely to be spending most of their time, and which are the “old” branches which can typically be ignored.

Hiding or Retiring Old Branches

It is possible “hide” old branches by removing read permissions on those repository paths (leaving them visible to superusers or perhaps an “archive” user). There are some potential performance implications to doing a lot of this (the more lines you add to the protections table the more work the server has to do for commands).

Often the easiest way is just to make them easily ignored via an appropriate hierarchical naming convention as shown above.

Provide Guidelines and Delegate Responsibility

Larger organisations may have a number of teams or units each of which is responsible for products with sometimes quite different lifecycles and thus branching schemes. In these cases it makes sense to set the guidelines and principles and delegate to the teams the work of defining the precise conventions to be used.

Automation

If your repository structure is simple enough and regular enough then it is easy for tools or scripts to:

• create them (perhaps with input from a user), including adding a mapping to the current client workspace view
• have triggers that validate that they are being created in the correct place (e.g. don’t allow a branch with wrong naming convention to be created in certain directories or levels of the repository). This prevents user error.
• work out the relationship between branches automatically and thus be able to produce simple reports for things like changes that need to be propagated etc. While this is often done by means of some configuration file (stored in the repository), and indeed this is sometimes necessary anyway, it is easier if it can be automatically deduced from the structure (so that a new branch “popping up” is automatically included into the reporting mechanism).
• make your protections table easier to maintain

There is No One Best Design!

And remember that this is always true – there are always many ways that are good enough, and the final choice often comes down to personal preference.

However, every type of branch should have a “home”, so that when a user creates a type of branch your CM Plan should have simple guidelines so the user is never at a loss as to where to create one. This requires identifying the types of branches likely to be used.

You need to do enough thinking and planning to get at least 80% of it right. It is of course possible to re-structure your repository (by branching from old name to new name and deleting the old within the same atomic changelist). However, this can cause you quite a few problems if the branch you are re-structuring has relationships with lots of other branches (e.g. release branches being maintained).

People who are aware of the issues and have experience, can slot in a new branch type fairly easily (though they can still end up making sub-optimal choices that they regret).

People ask from time to time how to browse a Perforce repository, and the easiest answer is to use p4web.

Some advantages:

• It’s free, and in browse-only mode doesn’t consume extra licenses – which means that you can “publish” changes via your Perforce server to managers and other people who may not normally have a Perforce license.
• It will “serve up” things like Word documents such that they are typically opened in Word by your browser
• It can give “canoical” or fixed URLs for documents or other configuration items stored in Perforce, so that fixed references always get the latest versions (as opposed to a file share where it is often unclear which is the latest and if there are any other versions etc)
• The default URL to get the head revision is very simple:
  • <p4web-base-url>/<document path within perforce without double slash at start>, e.g.
  • http://p4web-machine/depot/some/path/to/document.doc

Example of Accessing a Document

A test document has been setup in the Perforce Public Depot.

The direct URL to load the document is http://public.perforce.com/guest/robert_cowham/test/testdoc.doc

Alternatively it can be accessed for example with the history via p4web as:

http://public.perforce.com:8080/guest/robert_cowham/test/testdoc.doc?ac=22

Bear Security in Mind

If you setup in browse only mode as a user with access to the whole repository, you may be bypassing other security requirements and protections.

Consider setting up different p4web instances with different access permissions. Putting a separate layer on top of p4web with usernames and passwords, while possible, is likely to lead to maintenance problems.

DNS Alias to keep the URL fixed

If you start referring to a Canonical URL which then becomes used inside documents, make sure that the machine where p4web is running is accessed via a DNS alias so that it can be moved easily without breaking the references.

Also consider having a setup (e.g. via Apache or IIS virtual server settings) where the default web port of 80 is used to avoid ugly port components of the path such as :8080 or some other port  being required.

Parameters

The following were culled from personal experience and postings on perforce-user (by Noah Salzman and Nick Levine )

Action Codes

10	Changelist Detail
http://p4web.example.com:8080/12345?ac=10

14	Branch Specification
http://p4web.example.com:8080/foo_branch?ac=14

16	Label Spec
http://p4web.example.com:8080/build_tag_123?ac=16

22	Revision History
http://p4web.example.com:8080//depot/main/foo.c?ac=22

64	View Head Revision
http://p4web.example.com:8080//depot/main/rel1/foo.c?ac=64

69	Submitted Changelists
http://p4web.example.com:8080//depot/main/rel1/foo.c?ac=69
http://p4web.example.com:8080//depot/main/rel1/...@build_tag_123?ac=69

142	View Annotated
http://p4web.example.com:8080//depot/main/rel1/foo.c?ac=142

143	View Fully Annotated
http://p4web.example.com:8080//depot/main/rel1/foo.c?ac=143

Note: the single forward slash (versus a double forward slash) is
important in URLs that do not include depot paths.

107	List Jobs  (p4 jobs)
http://clrfi.alu.org:8080/?ac=107
http://clrfi.alu.org:8080/?ac=107&jsf=Job&jsf=Status&jsf=Date&jsf=Title&jsf=Area&jsf=Priority&ft=status%3Dopen

111	Describe Job (p4 job)
http://clrfi.alu.org:8080/job000005?ac=111

URLEncode the "@" (%40) or "#" (%23), for example, show contents of head rev

//public/jam/src/Build.com#2 is
http://public.perforce.com:8080/public/jam/src/Build.com%232?ac=64 

For #1
http://public.perforce.com:8080/public/jam/src/Build.com%231?ac=64 

For label @jam2-2-0 it is:
http://public.perforce.com:8080/public/jam/src/Build.com%40jam2-2-0?ac=64

This is a personal review of the Norwegian Aikido Federation (NAF) Vinterleir (winter school), 15 – 18 March in Tromso. Jessica Loeb and myself formed the London contingent – unfortunately Francis and Craig who were due to come too were not able to in the end.

Tromso is a very nice small town in the north of Norway, well inside the arctic circle. At this time of year there is plenty of snow and yet the daylight is a good 12 hours (it is far enough north to range from almost no daylight mid-winter to no real darkness in summer). There is not a vast amount to Tromso, but one of its attractions is that it seems to take about 10 minutes by car to go anywhere, including being out of the town and in some lovely nature.

The Vinterleir is traditionally taught by Bjorn Eirik Olsen (6th dan), Technical Director of the NAF, but this year, in honour of 30 years of aikido in Tromso (which was the first club in Norway at all), they invited Suganuma Sensei (8th dan). He has been 11 times to Norway but always in the south for the summer camp, and I think he very much enjoyed the different surroundings of snow and nature.

This also worked out very well as they have the Doshu Moriteru Ueshiba leading the summer camp in July this year. That promises to be a rather larger affair than usual!

I have actually been half a dozen times to the Vinterleir – there was a period when I went every year, but the last time was 5 years ago – Benedict who is now 4 and a half years old changed life a little!

As ever, the welcome in Norway is always extremely friendly and the atmosphere is a huge part of the success of the camp. Most people stayed with dojo members which was very nice, and in my case, together with Geirr from Oslo, we were kindly looked after by Ingelill.

It was great to see that Suganuma sensei had brought a large party from Fukuoka – nine in all. As was mentioned at one point in the speeches, it was also very nice that the Japanese group were not just fairly senior teachers, but ranged from a 17 year old to a 70 year old, and beginners to a 6th dan.

Suganuma Sensei

The training itself was excellent as usual – very clear and precise. Suganuma sensei’s teaching seems to vary very little from year to year, and is very unflashy on the surface and always seems to be oriented towards the basics in these courses. He is always very clear about techniques and gives lots of smaller pointers to particular things to watch out for ranging from use of atemi to positioning for maximum safety during nikkyo.

There are times when this can almost seem a little boring – and it certainly contrasts with Inaba sensei who spends quite a lot of time talking/lecturing during courses can also give some very interesting food for later thought and reflection (although some people find that very boring!).

Suganuma sensei is very attentive during the practice and frequently helps beginners with great patience, including taking ukemi for them to make sure they have understood.

Over the years it has become an interesting exercise to tease out more from what Suganuma sensei is demonstrating, and I was starting to get a better sense of how he uses his body and energy, and feels the connection with his partner. I still have lots of questions about the “shape” of some of his techniques which I can not replicate without expending a lot more muscle in comparison with other ways of doing the same techniques. Something to work on next time.

One very interesting point for me was when he mentioned how often we tend to provoke a negative response from our partners by inappropriate or hard techniques. This is something I am finding very fruitful for my current practice.

I will certainly be back again, although am interested to know if his teaching varies for example in his home dojo(s), although an interview with him implies that it doesn’t. It was very useful to practice a little with Tokuda sensei (6th dan) and get an alternative feel from someone who has trained with Suganuma sensei for many years. A very clear use of energy and sense of sticking and controlling in some techniques. I would love to arrange a visit on a future trip to Japan. For me a lot of the interest is because of the contrast with Inaba sensei – two very inspiring, but very different, teachers.

My Class

Due to the number of people on the course and the size of the main dojo not being sufficient to accommodate everyone, people were divided up into groups and roughly a third went off for a separate class at various points. I was asked to teach one of these and had some fun doing it.

I taught a somewhat relaxed and very much feeling based class rather than technique based, mainly because this is what I am finding most interesting in my personal practice at the moment. We started with some breathing exercises, breathing out and sucking the belly in, and then rotating the hips to drop the tanden/belly forward and using this to allow a fast in-breath, but with no hint of force or strain. It is important to feel as if breathing first into the belly and not fill the upper chest, or at least not let stiffness in there. Also there can be a temptation is to “sniff” which tends to stiffen the body – I find it much easier to do with an open mouth. This exercise certainly seemed a little strange at first, but I then built on it to show some simple techniques, e.g. from katatedori, where I find that an out-breath to absorb your partner’s attack/energy, followed by a quick in-breath seems to release some energy upwards which makes it much easier to lift their arm and indeed their energy.

I then focussed on keeping very soft and relaxed arms and shoulders while performing simple koky-ho from ryotedori (2 hand grab) or tenchinage, and allowing the arms to drop in a very relaxed manner without any sense of strain or pushing.

People seemed to enjoy it (after the initial confusion perhaps on the breathing and where I might be going with it!) and I had several appreciative comments afterwards which is always nice. My sources are mainly Inaba sensei’s teaching regarding the breathing, and Peter Ralston’s book “Zen Body-Being” for a fascinating approach to relaxation and effortless power which I am finding very rewarding to study.

Grading

I missed the kyu grading, but watched the dan grading on Saturday with interest: three shodans and two nidans, of whom I knew 4: Vegard, Steinar, Haakon and Kai Hare. The shodan grading took a good 20 minutes or more – any number of techniques being required to be demonstrated. Somewhat unnecessary in my opinion, but then Inaba sensei does rather tend to an alternative approach others find not rigorous enough. Interestingly the nidan gradings were shorter, since the requirement to work through the basic techniques was dropped in favour of a more free style approach. The candidates all did very well and deservedly passed. A nice surprise at the end was when Stein-Are Engstad was called out – he seemed very surprised and didn’t know quite what was going on only to be awarded his yondan.

The Party

As always the party on Saturday night was a great success, and this year was held in Polaria (the Polar Centre) – one of the main tourist attractions in Tromso. The food was delicious and there were some very good speeches around the celebration of 30 years. Birger Sorenson was present who started the club in Tromso as a 4th kyu back in 1977. In that first group were Bjorn Eirik and Kore (both present) so we were treated to a few reminiscences of life back then (including some slightly alternative views of exactly what did go on!), and a little earlier in the evening a slide show of some photos over the years.

Suganuma sensei also spoke, and offered his congratulations. He also mentioned that in the summer camps a few years ago I had asked him the question “What was the most important thing in Aikido?” He answered that it was musubi or connection. A couple of years later, I asked him the same question, not wishing to catch him out, but genuinely interested if he had changed his mind or not. When this was pointed out to him at the time, he saw the funny side, and this year saved me the trouble of asking it again by saying that he still thought musubi the most important thing!

I realised that he had demonstrated the importance of “connections” off the mat as well – it is one of the qualities that make him the very impressive man that he is. You don’t build an organisation of 100 or so dojos over 30 years if you can’t establish and more importantly maintain good connections with people. There are lots of aikido teachers who are technically very capable, but very few who have the personal qualities and lack of ego of Suganuma sensei. I am sure that his many years of study of zen alongside his aikido has had a large influence.

The dancing afterwards was ably lead as ever by Bjorn Eirik and Birger Sorenson! Having been kicked out at 1am those of us who were left walked through the centre of a still buzzing Tromso, and while the main diehards went to the “Fun Pub”, a smaller group of us enjoyed a beer at a slightly quieter local. The morning did however roll around rather early!

Touristy Things

The hosts had arranged a visit on Sunday morning for the Japanese, and Jessica and I tagged along. We took the cable car up the mountain on the mainland (to 420m), and were met by a couple of Sami (Lapp) hosts who gave us a reindeer sleigh ride and some coffee in a “lava” or tepee like tent. It seemed rather touristy and kitsch at first, but thanks to the really friendly and genuine guides turned out to be a lot of fun. The reindeer sleigh rides were only a quick circuit of perhaps 200m, but my ride became a touch more interesting when in response to my question “do you ever race?”, much encouragement was given to the reindeer who promptly got over excited and veered off course heading for his mates down the hill. It turns out that reindeer don’t like to be touched, and aren’t terribly easy to control – starting out consisted of pulling the reindeer forward with them initially resisting, and then turned into a quick sprint and a leap on to the sledge as the reindeer suddenly bounded off!

Our hosts were young guys whose family apparently has some thousands of reindeer in their herd who spend the summer around Tromso (from April onwards), but they winter about 100km further south and inland where the climate is drier and the snow easier to dig for lichen/moss in. They were very friendly and enthusiastic with their explanations on details of dress and life, and it was very enjoyable. The only problem was that our time was a little limited and it would have been great to have a walk after lunch given the great weather up there.

Conclusion

A great course, leaving me feeling rather well exercised, and with some more happy memories to add to the store associated with Norway. Thanks to Bjorn Eirik and all the other members of Reimeikan.

The 3rd BCS CMSG Conference is now open for registrations with a good lineup of speakers, and an excellent looking venue.

A number of people commented on problems getting to Cambridge (not so convenient from various parts of the UK), so hopefully Oxford will fit the bill a little better. In addition, we can up the budget a little for a slightly more commercial venue.

Some other differences this time around:

• PB Projects is doing the bulk of the heavy lifting in terms of organising details etc – yes it costs us, but it seems to be working fairly well. and takes some (though not all – e.g. sorting out timetable/speakers) of the load off the committee.
• Only just realised that offering a group discount might help bring in a few more people – we more interested in having more people there than making a vast profit (haven’t yet managed to get the committee fact finding trip to the Caribbean off the ground to spend our current surplus, but am working on it…)

So, the first registrations are already in, and look forward to plenty more!

Meanwhile, am hopefully getting a little breather to allow rather more blogging than of late – have a stack of topics piled up awaiting that spare moment!

There have been some interesting posts recently on the challenges faced by
Microsoft in developing Vista. It has of course fairly recently made it out of
the door which in itself is a major achievement. However, it has become clear
that many features originally scheduled for inclusion have been dropped (WinFS
for example). The sheer challenge of developing such a vast system is immense,
and a key part of any such development effort is the change, configuration and
release management.

Some fairly recent blog postings have highlighted some of the issues involved,
and seeing as SCM is right in the mix, I though it worth a review.

Overview of Vista – Paul Thurrott’s
Supersite for Windows

The sheer size and scope of Windows Vista makes it difficult to review, to
digest, and to understand. If you step back too far, it doesn’t look very
impressive at all: It’s like XP with a spit-shine. But if you get too close,
it’s easy to get lost in the seemingly never-ending lists of new features.

A previous article by Paul
bemoaned various lost features.

Joel Spolsky wrote an article
How many Microsofties does it take to implement the Off menu? reflecting
on UI choices and how less is usually more.

This attracted a comment (or was it coincidental):
Moishe Lettvin’s Windows Shutdown Crapfest posting:

The most frustrating year of those seven was the year I spent working on Windows
Vista, which was called Longhorn at the time. I spent a full year working on a
feature which should’ve been designed, implemented and tested in a week..

The key SCM-related part of Moishe’s post is:

In small programming projects, there’s a central repository of code. Builds are
produced, generally daily, from this central repository. Programmers add their
changes to this central repository as they go, so the daily build is a pretty
good snapshot of the current state of the product..

In Windows, this model breaks down simply because there are far too many
developers to access one central repository. So Windows has a tree of
repositories: developers check in to the nodes, and periodically the changes in
the nodes are integrated up one level in the hierarchy. At a different
periodicity, changes are integrated down the tree from the root to the nodes.
In Windows, the node I was working on was 4 levels removed from the root. The
periodicity of integration decayed exponentially and unpredictably as you
approached the root so it ended up that it took between 1 and 3 months for my
code to get to the root node, and some multiple of that for it to reach the
other nodes. It should be noted too that the only common ancestor that my team,
the shell team, and the kernel team shared was the root.

Joel Spolsky then
chimed in:

Of all the things broken at Microsoft, the way they use source control on the
Windows team is not one of them…

When you’re working with source control on a huge team, the best way to
organize things is to create branches and sub-branches that correspond to your
individual feature teams, down to a high level of granularity. If your tools
support it, you can even have private branches for every developer. So they can
check in as often as they want, only merging up when they feel that their code
is stable. Your QA department owns the “junction points” above each merge. That
is, as soon as a developer merges their private branch with their team branch,
QA gets to look at it and they only merge it up if it meets their quality bar.

I think there are various interesting things about this discussion.

Managing Complexity and Reducing Dependencies

From the days of The
Mythical Man Month: “Adding manpower to a late software project makes
it later”, it has been clear that the challenges of managing large scale
systems are many. The whole book is a joy to read, and the Wikipedia articles
referenced give a good highlight.

I particularly like the idea of Conceptual Integrity to help keep things simpler
both in development and control.

Microsoft’s products would appear almost by design to have a very high level of
interdependencies between them (although
this is changing:
“And what happened is as the
projects got larger and larger, we introduced too many complex interdependencies
on early software, more so than we could really digest throughout the system,”
said David Treadwell, corporate vice president of the .Net Developer Platform
group).

One advantage of the dependencies is that if your customer buys one
product then they pretty much have to take a slew of accompanying
products too.

Branches and Sub-Branches – Your SCM Tool

Back to Joel’s comment that the best way to organise things is branches and
sub-branches. This is a fairly classical SCM approach to the problem which has
many benefits.

It is interesting to note that Joel mentions SourceDepot which Microsoft uses
internally, and which is a re-badged version of Perforce circa 1999 (when win2k
was in development), with some Windows specific improvements such as memory
usage, but from what I can gather no fundamental algorithm improvements.
Perforce of that era only had the ability to easily propagate changes between
directly related parent-child branches. Things like grand-parent <->
grand-child propagation skipping over intervening parent were possible but
tricky – the tool didn’t default to handling it. (Note that I am
interested in the original comment about there being a tree of repositories,
as opposed to a tree of branches – if the former, then it
makes things much more difficult to improve merging across separate repositories).

Microsoft’s new Team Foundation Server tool (uses a similar branching model and
terminology but totally re-architected it would appear), does not
yet support good common ancestor detection either.

Over the years Perforce has
addressed this in various ways, and from 2005 it is working well and
with reasonable performance at large sites. Thus in Perforce you can pull
changes from one sub-branch to another without going via the parent, and
changes propagated back to the common parent will not cause major problems for
either sub-branch. There still remains a very large question as to whether
“uncontrolled” propagation of changes is a good idea (as Laura
Wingerd puts it -

“why we don’t allow driving through hedges” – Ch7), and my advice is
certainly to think carefully and plan your normal strategy (with tightly locked
down exceptions maybe permissible). It still comes back to managing complexity:
the more you allow changes to be propagated all over the place in an ad-hoc
fashion, the harder it is to track what has gone one.

As mentioned by Joel, Accurev is a very interesting looking tools and their
stream browser certainly looks attractive, and has raised the bar in terms of
features for the SCM vendors. My impression is that Perforce still has the
advantage in
terms of scalability (see Google paper) and performance, but I would
certainly put Accurev on my evaluation list if advising a client in this area.

I had some fun with the Agile SCM Column at CM Crossroads this month:

A Christmas (Configuration) Carol (Abridged)

We have endeavored in this Ghostly little tale, to raise the Ghost of an Idea, which shall not put our readers out of humour with themselves, with each other, with the season, or with us. May it haunt their houses pleasingly, and no one wish to lay it.

Their faithful Friends and Servants

C.D., R.C., B.A & S.B.

Charlie’s GhostIn which the ghost of (Configuration) Charlie appears to our heroine SCM Sally to help her in her dealings with VP of Development Ben (’Benezer)Scrooge.

Continued…

Sometimes inspirations strikes, though it is interesting what turns out in the end – had a couple of different ideas (and a lot more quotations from the original), but it evolved into what you see. Not quite as provocative as a previous column:

National Treasures of Agile Development

Introduction

Recent research has discovered a very interesting cache of papers about a little known Tribe called the Agile Developers. The first document is a draft dated July 4th (the year is illegible).

(Document starts)

When in the Course of human events, it becomes necessary for some developers to dissolve the bands which have connected them with others, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature’s God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation.

We hold these truths to be self evident,

Continued…

Have been meaning to do some work on P4Python recently, and the first thing I realised I should do is to update the test harness.

This is based around the unittest module so is fairly standard Python. Does some fairly standard things with test suites etc, and provides a reasonably good example of how to use the code itself, so acting as some level of documentation of P4Python.

The old version assumed a pre-existing Perforce server installed and running with some known content. This was fine for my own personal testing but had a couple of problems:

• it assumed the training repository which is fine for Perforce Consulting Partners but which isn’t available to ordinary mortals (so they couldn’t run the test harness locally)
• was a snapshot with existing users and client workspaces so required a license to use – not good for everyone
• it required too much manual setup before running

Obviously a candidate for tidying. So recent work done:

• Change to use the new Perforce Sample Repository which anyone can download and install (works without a license too)
• Change to automatically create and run a new server instance on a fresh install from the sample repository download (automatically unzip etc)

The end effect is fairly nice and automated, and provides a much better ongoing resource to anyone wishing to do work on P4Python. Please note that it currently assumes Windows environment, but will insert a few checks to make platform independent shortly.

To have a look you can either:

• download the complete P4Python sources
• browse the test module
• just browse the Perforce Public Depot using your favourite method: //guest/robert_cowham/perforce/API/python

Note that credit for various aspects should go to Ravenbrook from their work on P4DTI – I stole various techniques from their implementation of a test harness.

It also serves as a reasonable example of a test harness for Perforce scripting, and am very happy to receive comments and improvements (there are no doubt some Python gurus out there who can suggest some improvements at points in the code). I use something fairly similar as a test harness for the VSStoP4 scripts which are actually written in Perl (too horrible to write such a thing in Perl was my feeling)!

Would like to encourage people to take their own copies and give back some improvements in the general framework at least…

This is a personal review of the week in Wales with Inaba sensei and other teachers organised by Tetsushinkan dojo. It is not a blow by blow account of everything that took place, but a personal summary with various highlights of things that were important for me.

First a quick overview:

• An advanced guard of the Japanese contingent arrived on Thursday (not including Inaba sensei)
• Udegawa san taught the normal Friday evening class at Tetsushinkan
• Saturday everyone decamped the 4-6 hours to Wales (this included Inaba sensei and Endo san who only arrived that day from Poland where they had been teaching the previous week). Various other overseas visitors flew in that day and also drove down.
• Sunday was the official opening and first practice
• Standard schedule during the week
  • Morning practice 9.30 – 12.30
  • Afternoon activities in groups ranging from canoeing to horse riding to cliff walks
  • A couple of Instructor’s sessions 6-7.30pm for invited people (others free to watch)
  • A couple of meetings with instructors to review links with Shiseikan, including future courses
• On Tuesday evening we greeted Toyama guji (the Chief Priest of the Meji Jingu shrine who was “on tour” in Europe) with his entourage to Wales. He came to watch practice on Wednesday and kindly hosted a meal on the Wednesday evening for an invited group.
• Final practice on Friday, with an excellent party that evening in a restaurant in Newport
• Return to London on Saturday
• Japanese all left Sunday

Summary

It was a truly excellent week. I came away with masses of food for thought and further reflection regarding both life and budo studies. It was a great pleasure to renew old friendships and to make some new friends. I also feel a renewed personal challenge to do more with these insights!

The teaching started with a formal opening ceremony conducted by Mori san, a priest from the Meiji Jungu shrine and included prayers and harai tachi – cleansing of the dojo and the spirits of those present. Inaba sensei talked about his wish to demonstrate the importance of this aspect as part of his teaching. Shinto is a relatively recent name for practices which go back thousands of years. The gods and spirits come and sit in the branches of the himorogi, part of the small shrine set up during practice. He does not view this as a religious practice and certainly has no wish to convert everyone to Shintoism! However, he wished to demonstrate the importance of setting the tone for the week – in a way a sort of misogi for all those present.

Those of you who have seen Inaba sensei will know that talking and lecturing form quite a part of his teaching. This can be a bit frustrating at first for those used to other styles of teaching where much more emphasis is placed on practice. Many of Inaba sensei’s messages keep recurring over the years, but as often happens, various things made more sense to me this time – maybe I have changed in the meantime and so now hear them differently!

Why Are you Practicing?

As with the initial ceremony, this was something Inaba sensei returned to at several points. We should understand why we are practising – what do we wish to protect or defend? What are we prepared to do in the face of attack, for example on our loved ones? Answers to this question inform all the rest of our studies, and thus really are fundamental. Study and thought will be well repaid.

Relationship with Kunii Sensei

Inaba sensei gave some more details on his relationship with Kunii Sensei (the 18th generation Headmaster of the Kashima Shinryu school). This relationship was fairly short – 17 months I believe as a direct student before his death in 1966. Inaba sensei started studying aikido in the early 1960s at Hombu (and with Yamaguchi sensei). Being allowed to study with Kunii sensei was not easy, but was enabled by an introduction from Ashizu sensei, a Shinto philosopher that Inaba sensei was also studying with (and indeed that association went on for nearly 30 years). The relationship with Kunii sensei was obviously very close and Inaba sensei mentioned being asked by Kunii sensei to pass what he learnt through direct transmission and also the results of his own independent study. Inaba sensei obviously feels a very strong obligation as a result.

Developing the Body

This has long been a key concern – developing our bodies as a foundation for studies in budo. This time around there were copies of Inaba sensei’s book on the subject (English and Japanese in the one volume). The English title is “Researching Japanese Budo”. It originated in a series of articles for a Japanese magazine in the early ’90s, and I was involved in helping revise the translation – I hasten to add that this meant reviewing the resulting English and offering some suggestions for clarifying unclear points or reworking phrases which appeared a little clumsy. The real work was done by Endo san and Annika Hansen. The original translation was done by Yamada sensei and Diane Zingale, of which a number of people received photo copies nearly 10 years ago.

Anyway, pressure of time meant that it still isn’t quite where it might be in the English translation, but certainly full of very useful and interesting information – perhaps will review separately. I suggest contacting Shiseikan directly if you wish to get a copy.

Energy Inherited from your Ancestors

This was a new concept for me, and something I did not find easy to get a handle on in a physical sense. Intellectually to me it makes sense to talk about what we inherit from our direct ancestors and also the energy of the place we are in, but how to do anything physical with such energy was not clear. The exercise demonstrated was basically kokyu-ho (e.g. both hands grasped and lifting your hands and thus your partner in front of you). The difference as shown between this and normal ki energy was not clear to me in any practical way, so very much one for further research.

Kenjutsu Kata

I was very interested to see demonstrated some more dynamic ways of studying kenjutsu. Inaba sensei discussed the various kata and went through the details of the Uradachi (second set of 10 techniques) in some detail. However, as well as the classic form of the kata with uke and nage, we were also shown more dynamic forms, for example where uke more or less attacked in a kendo style and the response need to be much more freeform than in the set version of the kata. I was particularly interested as I had been thinking prior to the course that my own kenjutsu risked being a little formulaic – falling into the pattern of anticipating the technique and only being able to respond in the fixed form – even getting annoyed when the attack wasn’t “just right”! Some teaching of the ukemi side were also very useful for me and others who are in positions to teach. It is important to ensure that you are applying appropriate pressure to your partner to draw their best out from them.

He did mention the requirements to study each set of kata (Kihon, Ura, Aishin, Jissen, Kassen) properly before moving on to the next, but also to place them in a context of being able to react flexibly.

Inaba sensei went through the Uradachi in some detail as part of one of the instructor’s courses. Particular points of note included the requirement for both sides to treat the shinai as a real sword and thus heavy contact by uke is not appropriate – you can’t lean on the blade of a katana! Normally nage leads by taking the first step, although it may appear that both people start together. It is uke’s responsibility to respond to nage.

Kesa Giri Footwork

I don’t think anyone present will forget the outdoors practice of kesa giri – the location was at Clydey Cottages – a place with a great lawn and stunning view over the valley. Inaba sensei really demonstrated the difference between short, restricted, inhibited kesa giri and the more expansive cuts which really started to connect with the nature all around us. The difference that this made in everyone’s kesa giri was very marked, partiularly the following day back inside – you could see the expansiveness still present.

This moved on also to the use of the feet. For practical reasons of the ground being perhaps unstable, or uneven, or on a slope, he emphasized the need to step with the feet fully before completing the cut. It is easy in the clean confines of the dojo to get used to cutting with the landing of the foot and the focus point of the sword cut occurring at the same point. Outdoors however, this doesn’t work – as soon as you step you risk slipping and sliding, so it is important to have fairly firm footing before completing the cut.

Battojutsu (Sword drawing)

This was another memorable day outside (though slightly more rainy) focussing on the drawing of the sword. We were all using bokuto (wooden swords) for practice but he did discuss the use of katana (live blades). He does not recommend practice with iaito (metal swords with blunt blades), as these tend to give one a false sense of security. A katana will “teach” you if you let it. It requires much more focus and attention which is good for practice and concentration. In a way, you should not use a katana too much since familiarity may breed contempt and thus lack of focus.

Various exercises and cutting to different sides (front, back and both sides) were demonstrated and practiced. In practice you shouldn’t just get into the habit of thinking “OK, now I will pretend someone is attacking from behind, so let’s react”. It should be a much more instinctual response – wait and be alert, and the right moment will arise. Remember also flexible footwork – react off either foot.

Stepping off the Line

This was another exercise completed outside. It is one I have seen and practiced several times over the years, and yet realised this time around that I really need to focus much more on it for my own practice.

You start a couple of paces away from uke, and then step in so as to “invite” an attach (standard punch to the stomach). As the punch comes in you need to step off the line so that you end up in a triangle with feet pointing at your opponents centre. If uke attacks right hand (and foot), then it is quite normal to initially half step with the right foot to “invite”, and then step off the line with the left foot closest to uke. See Figure 1.

The alternative which we practised involves “inviting” with a half step forward of the left foot. The result is similar, but left and right feet are reversed.

This appears quite simple but is difficult to do well, and in particular getting the timing correct and being able to move your body appropriately is key. Watching Inaba sensei, he doesn’t necessarily appear to move particularly fast, but it is at just the right moment and with appropriate speed.

Further developments of this exercise shorten the distance to one step, then half a step, then no steps (with uke’s hand on your belly). Challenging stuff!

The Grading

This happened at the end of the week during the last session ranging from shodan to yondan (errrr that would be me…). Most people were from Tetsushinkan but some others from London and Greece, and everyone generally did very well. Results are on the Tetsushinkan events page.

As is typical with Inaba sensei’s gradings they were relatively short. In each case various attacks were specified and the actual technique was not specified – indeed it was perfectly acceptable to perform the same technique several times in a row to the same attack. The important factors are spirit, body movement and effectiveness, rather than necessarily technical beauty. People also demonstrated some of the Kashima Shinryu kenjutsu kata according to the knowledge and experience of the person being graded (e.g. Kihondachi or Uradachi). This approach is certainly different to what is normally done within many standard aikikai  dojos. I did discuss this afterwards over a beer with a couple of people – “but how do you know they know all the appropriate techniques?” was one question. I obviously can’t speak for Inaba sensei but I think that it is usually clear from a demonstration the sort of level a person has reached, and it is perfectly in tune with his general approach of studying relatively few techniques but in great depth. In addition, I think that this sort of approach works well in smaller organisations where the people being graded are known to the examiners – I think there might be more problems with larger scale organisations.

Instructor’s Meeting – The Future

There were a couple of interesting discussions with vairous instructors and countries present, e.g. Bjorn Eirik Olsen and others from Norway, Pascal Durchon and Joel Roche from France, Anita Kohler from Germany and various others from UK etc. As with all good discussions, some heat and light at times was eventually turned into some good results and conclusions.

Inaba sensei was keen to get an understanding of how he could best support those present, and what sort of future courses and training was both sought and might be possible. He did mention that his preferred method is to teach a small group of people at the Shiseikan where he could give a lot of time and attention to them, and he made a generous offer for those prepared to arrange visits. Pascal and Anita in particular expressed a desire to give more people the opportunity to get to know Inaba sensei and experience his teaching. The rough plan for 2007 is for a seminar in France to be organised primarily by Pascal, followed by Norway in 2008 and Germany in 2009. Visits to Shiseikan are left to individual groups to arrange. I think the intention for Paris is to perhaps offer open sessions where anyone can attend, but also a more restricted session for those with previous experience and who have already made contact with Inaba sensei and are already studying his methods. This would allow for more advanced training, in particular for kenjutsu. I think it will be very interesting to see how this works, and certainly look forward to Paris next summer!

Figure 1 – Stepping off the line. Step1 in is the “invitation” (or sassoi) – a half step. Then you step 2 to move left foot around an finally step 3 moves the right foot again. Note that both feet are now off the line but the focus (shown by heavy dashed arrow) is on a line pointing at uke’s centre which forms a triangle with the initial focus line.

Conclusion

The course was excellent – lovely surroundings and a great atmosphere. Serious study and yet not in a heavy atmosphere. The number of seniors around (from Japan and elsewhere) made for some excellent training.

Inaba sensei is a fascinating and inspiring teacher on many levels, and it is worth taking any opportunity to get around him and learn from him. Some of his methods are a little extreme perhaps, and I am not clear in my own mind how easy it is to combine what he teaches with classic aikikai aikido. The differences go to quite deep levels. This is fine for those of us who primarily follow his teaching. Others who think it might be nice to spice up their traditional methods may find it distinctly more challenging. How much integration is possible and how much is it two separate approaches – I suspect it is going to take me at least quite a few more years to decide. In the meantime he is planning for the future. He is not going to go on for ever – seek him out and take advantage of him!

Life has been just a touch busy recently having been flat out on various client projects pretty much over the whole summer (managed a week away but only just!). All grist to the mill for future blogging, so hopefully a variety of articles to come!

Meanwhile one of the things I was doing was preparing and then giving a presentation for the (first) Perforce European Conference on 19th September in central London.

I think the papers will be out pretty shortly on the Perforce site, but meanwhile a few highlights and personal notes. There were some big names present and it was good to hear about various practices and principles in operation.

Keynote

Christopher Seiwald did a variation on his slightly “aw shucks” style keynote. Some key points:

• Perforce doing fine: 200,000 users and 4,000 companies
• Company motto: “Aim low and hit!” (do one thing well, remain best of breed and wait for the analyst pendulum to swing back to best of breed rather than suite integration, which it seems to do on a regular basis)
• Working on a variety of things for future world domination, but don’t want to pre-announce as usual
• Very pleased with the way things are happening in Europe, and obviously at the response to this event.
• Next US Conference 9 – 11 May 2007, Las Vegas.
• Sydney office now opened to give global timezone support coverage!

Symbian

Deepak Modgill did a nice presentation on the challanges faced by Symbian for their offshoring. Another in the Symbian series of how their business and vairous configuration management practices have evolved. Not deeply technical but interesting never-the-less.

SAP

Obviously a flagship site for Perforce. Thomas Kroll and Claudia Loff did a good presentation. Interesting how much process and tools they had wrapped around Perforce. A few key stats:

• 4,800 users
• 80+ Perforce servers (but all on same cluster hardware)
• Fujitsu Siemens clusters with 32Gb RAM running SunOS 9
• SAN (mirrored) for main data

They use a very structured process (repository structure and branching scheme) and a parallel (P4SAP) system with its own database to record things like changes and migrations (they call them transports) of releases between different servers. There is also a layer P4MS (Management System) to handle users etc.

Quite impressive.

Process Automation

Obviously my talk was wonderful! I was thought fairly pleased with how it went down and got some good comments afterwards. For anyone interested, the Ruby triggers framework and a couple of utilities are in my area of the Perforce Public Depot.

I will no doubt be blogging on various related aspects (that I haven’t already touched on).

Bank of America

Good talk by Sean Cody and Kevin Breidenbach about different approaches with the bank. They have been replacing ClearCase with Perforce in various groups, mainly due to the performance for shared development between US, UK and India. Experience of Multisite sometimes taking hours to “sync up”, vs. 10-20 minutes max in Perforce.

Another feature of the talk was the power of continuous integration.

Google

Dan Bloch discussed Google’s use of Perforce and in particular how they manage issues around Perforce database locking and identifying and bumping off rogue commands.

Some more stats:

• 3,000+ users
• Single Perforce Repository
• HP DL585 4-way Opeteron with 128Gb RAM
• Linux 2.4 and NetApp filer

Sounds like it wins the contest for largest number of users against a single server!

The details of the lock identification was very interesting and Dan said he would be releasing the lock.pl script and some docs on the Public Depot real soon now!

Perforce 2006.1 Update

A very interesting and technical talk by Michael Shields regarding a variety of performance optimisations made between 2005.2 and 2006.1.

Summary: 2006.1 is quite a bit faster!

Read the slides for more details.

Laura Wingerd

Laura did another fairly technical talk on what has happened to the branching/merging algorithm, and more particularly common ancestor detection algorithm used in various releases of 2006.1. In her usual inimitable style she came up with some very useful ways of explaining things like convergence and divergence of branches over time. Things got decidedly more technical with discussions on common ancestors and I was left knowing I have to go through some of this in detail in a quiet moment just to make sure I really do understand it! The changes with 2006.1 look good, but I did get the impression some edge cases could give some slightly surprising results if you don’t know what’s going on behind the covers (and indeed the driving intentions behind the algorithm).

Summary

Venue worked very well for location. Networking with both Perforce people and various other delegates was as ever a highlight.

Unfortunately the room booked was not huge which meant the event sold out well ahead of time – a shame a more flexible venue wasn’t chosen, but that was only quibble. Organisation well run.

An excellent day!

There are times when you receive a third party code drop which you wish to import. The classic method is documented in Tech Note 15 and its reference to working disconnected (Tech Note 2). The techniques mentioned work very well to find new files, deleted files and changed files.

There is sometimes a fly in the ointment to do with keyword expansion. This is things like a CVS code drop containing expanded keywords:

$Id: //depot/robertcowham.com/main/blog/data/scm/p4_handling_keywords.html#1 $

The Perforce equivalent of this might be:

$Id: //depot/robertcowham.com/main/blog/data/scm/p4_handling_keywords.html#1 $

The simple command to find differences is “p4 diff -se”. If your local version has Perforce keyword expansion turned on then you will get a load of files spuriously identified as having changed where the only real change is in the keywords.

Thus we want a simple script to run through the diffs and exclude any diffs where only keywords are found (note that this includes where the keyword is embedded, such as in a static variable assignment).

The following simple script is a good base for this. It does the job, and performs pretty well, handling thousands of files in a few minutes. It makes use of unified diff format where changed lines have a prefix in the first character of the output.

# Script to import a set of changed files with existing keywords already expanded
# (either Perforce or CVS).
# Does "diff -se" and processes the output

# Args: current directory to check

require 'P4'

p4 = P4.new
p4.tagged
p4.connect

def process_file(p4, f)
  diffs = p4.run_diff("-f", "-du", f)
  real_diffs = Array.new
  diffs.each { |line|
    case line
    when /^====/
    when /^\@\@/
    when /^ /
    else
      if line !~ /\$Id|\$DateTime|\$Revision|\$Date|\$Author|\$Name|\$RCSfile|\$Source/
        real_diffs << line
        # puts f, line
      end
    end
  }
  if real_diffs.size > 0
    print "Editing #{f}\n"
    p4.run_edit(f)
  end
end

all_files = p4.run_diff("-se", ARGV[0])
print "Processing #{all_files.size}\n"
i = 0
all_files.each{|f|
  i += 1
  # print"Processing #{i}\r" if i % 10 == 0
  # print"Processing #{f['depotFile']}\n"
  process_file(p4, f)
}

It is pretty easy to run, e.g.:

diff_se.rb ...

The net result will be a list of files checked out (p4 edit) in the default changelist.

Note that one of the big advantages of Perforce branching and merging is that it handles merges neatly when keyword expansion is used between branches (and thus you don’t get spurious conflicts).

CVS Imports

If you use the cvs2p4 scripts to import a CVS repository you can end up with a slight problem since the conversion copies the CVS archive files (in RCS format) and Perforce uses them unchanged. The problem comes about because CVS stores the keywords already expanded in the RCS archive. Perforce stores its RCS files with the keywords not expanded, which makes it easier for it to do the merging between branches (without keyword conflicts). While Perforce can handle a CVS archive with the the keywords “pre-expanded”, it does lead to spurious merge conflicts. Note that this problem is only really present during the early merges after the CVS import. It will no longer be present as soon as the base file for any merge is fully in Perforce format (i.e. after at least one merge has been done).