Entries Tagged 'SCM' ↓

People ask from time to time how to browse a Perforce repository, and the easiest answer is to use p4web.

Some advantages:

• It’s free, and in browse-only mode doesn’t consume extra licenses – which means that you can “publish” changes via your Perforce server to managers and other people who may not normally have a Perforce license.
• It will “serve up” things like Word documents such that they are typically opened in Word by your browser
• It can give “canoical” or fixed URLs for documents or other configuration items stored in Perforce, so that fixed references always get the latest versions (as opposed to a file share where it is often unclear which is the latest and if there are any other versions etc)
• The default URL to get the head revision is very simple:
  • <p4web-base-url>/<document path within perforce without double slash at start>, e.g.
  • http://p4web-machine/depot/some/path/to/document.doc

Example of Accessing a Document

A test document has been setup in the Perforce Public Depot.

The direct URL to load the document is http://public.perforce.com/guest/robert_cowham/test/testdoc.doc

Alternatively it can be accessed for example with the history via p4web as:

http://public.perforce.com:8080/guest/robert_cowham/test/testdoc.doc?ac=22

Bear Security in Mind

If you setup in browse only mode as a user with access to the whole repository, you may be bypassing other security requirements and protections.

Consider setting up different p4web instances with different access permissions. Putting a separate layer on top of p4web with usernames and passwords, while possible, is likely to lead to maintenance problems.

DNS Alias to keep the URL fixed

If you start referring to a Canonical URL which then becomes used inside documents, make sure that the machine where p4web is running is accessed via a DNS alias so that it can be moved easily without breaking the references.

Also consider having a setup (e.g. via Apache or IIS virtual server settings) where the default web port of 80 is used to avoid ugly port components of the path such as :8080 or some other port  being required.

Parameters

The following were culled from personal experience and postings on perforce-user (by Noah Salzman and Nick Levine )

Action Codes

10	Changelist Detail
http://p4web.example.com:8080/12345?ac=10

14	Branch Specification
http://p4web.example.com:8080/foo_branch?ac=14

16	Label Spec
http://p4web.example.com:8080/build_tag_123?ac=16

22	Revision History
http://p4web.example.com:8080//depot/main/foo.c?ac=22

64	View Head Revision
http://p4web.example.com:8080//depot/main/rel1/foo.c?ac=64

69	Submitted Changelists
http://p4web.example.com:8080//depot/main/rel1/foo.c?ac=69
http://p4web.example.com:8080//depot/main/rel1/...@build_tag_123?ac=69

142	View Annotated
http://p4web.example.com:8080//depot/main/rel1/foo.c?ac=142

143	View Fully Annotated
http://p4web.example.com:8080//depot/main/rel1/foo.c?ac=143

Note: the single forward slash (versus a double forward slash) is
important in URLs that do not include depot paths.

107	List Jobs  (p4 jobs)
http://clrfi.alu.org:8080/?ac=107
http://clrfi.alu.org:8080/?ac=107&jsf=Job&jsf=Status&jsf=Date&jsf=Title&jsf=Area&jsf=Priority&ft=status%3Dopen

111	Describe Job (p4 job)
http://clrfi.alu.org:8080/job000005?ac=111

URLEncode the "@" (%40) or "#" (%23), for example, show contents of head rev

//public/jam/src/Build.com#2 is
http://public.perforce.com:8080/public/jam/src/Build.com%232?ac=64 

For #1
http://public.perforce.com:8080/public/jam/src/Build.com%231?ac=64 

For label @jam2-2-0 it is:
http://public.perforce.com:8080/public/jam/src/Build.com%40jam2-2-0?ac=64

There have been some interesting posts recently on the challenges faced by
Microsoft in developing Vista. It has of course fairly recently made it out of
the door which in itself is a major achievement. However, it has become clear
that many features originally scheduled for inclusion have been dropped (WinFS
for example). The sheer challenge of developing such a vast system is immense,
and a key part of any such development effort is the change, configuration and
release management.

Some fairly recent blog postings have highlighted some of the issues involved,
and seeing as SCM is right in the mix, I though it worth a review.

Overview of Vista – Paul Thurrott’s
Supersite for Windows

The sheer size and scope of Windows Vista makes it difficult to review, to
digest, and to understand. If you step back too far, it doesn’t look very
impressive at all: It’s like XP with a spit-shine. But if you get too close,
it’s easy to get lost in the seemingly never-ending lists of new features.

A previous article by Paul
bemoaned various lost features.

Joel Spolsky wrote an article
How many Microsofties does it take to implement the Off menu? reflecting
on UI choices and how less is usually more.

This attracted a comment (or was it coincidental):
Moishe Lettvin’s Windows Shutdown Crapfest posting:

The most frustrating year of those seven was the year I spent working on Windows
Vista, which was called Longhorn at the time. I spent a full year working on a
feature which should’ve been designed, implemented and tested in a week..

The key SCM-related part of Moishe’s post is:

In small programming projects, there’s a central repository of code. Builds are
produced, generally daily, from this central repository. Programmers add their
changes to this central repository as they go, so the daily build is a pretty
good snapshot of the current state of the product..

In Windows, this model breaks down simply because there are far too many
developers to access one central repository. So Windows has a tree of
repositories: developers check in to the nodes, and periodically the changes in
the nodes are integrated up one level in the hierarchy. At a different
periodicity, changes are integrated down the tree from the root to the nodes.
In Windows, the node I was working on was 4 levels removed from the root. The
periodicity of integration decayed exponentially and unpredictably as you
approached the root so it ended up that it took between 1 and 3 months for my
code to get to the root node, and some multiple of that for it to reach the
other nodes. It should be noted too that the only common ancestor that my team,
the shell team, and the kernel team shared was the root.

Joel Spolsky then
chimed in:

Of all the things broken at Microsoft, the way they use source control on the
Windows team is not one of them…

When you’re working with source control on a huge team, the best way to
organize things is to create branches and sub-branches that correspond to your
individual feature teams, down to a high level of granularity. If your tools
support it, you can even have private branches for every developer. So they can
check in as often as they want, only merging up when they feel that their code
is stable. Your QA department owns the “junction points” above each merge. That
is, as soon as a developer merges their private branch with their team branch,
QA gets to look at it and they only merge it up if it meets their quality bar.

I think there are various interesting things about this discussion.

Managing Complexity and Reducing Dependencies

From the days of The
Mythical Man Month: “Adding manpower to a late software project makes
it later”, it has been clear that the challenges of managing large scale
systems are many. The whole book is a joy to read, and the Wikipedia articles
referenced give a good highlight.

I particularly like the idea of Conceptual Integrity to help keep things simpler
both in development and control.

Microsoft’s products would appear almost by design to have a very high level of
interdependencies between them (although
this is changing:
“And what happened is as the
projects got larger and larger, we introduced too many complex interdependencies
on early software, more so than we could really digest throughout the system,”
said David Treadwell, corporate vice president of the .Net Developer Platform
group).

One advantage of the dependencies is that if your customer buys one
product then they pretty much have to take a slew of accompanying
products too.

Branches and Sub-Branches – Your SCM Tool

Back to Joel’s comment that the best way to organise things is branches and
sub-branches. This is a fairly classical SCM approach to the problem which has
many benefits.

It is interesting to note that Joel mentions SourceDepot which Microsoft uses
internally, and which is a re-badged version of Perforce circa 1999 (when win2k
was in development), with some Windows specific improvements such as memory
usage, but from what I can gather no fundamental algorithm improvements.
Perforce of that era only had the ability to easily propagate changes between
directly related parent-child branches. Things like grand-parent <->
grand-child propagation skipping over intervening parent were possible but
tricky – the tool didn’t default to handling it. (Note that I am
interested in the original comment about there being a tree of repositories,
as opposed to a tree of branches – if the former, then it
makes things much more difficult to improve merging across separate repositories).

Microsoft’s new Team Foundation Server tool (uses a similar branching model and
terminology but totally re-architected it would appear), does not
yet support good common ancestor detection either.

Over the years Perforce has
addressed this in various ways, and from 2005 it is working well and
with reasonable performance at large sites. Thus in Perforce you can pull
changes from one sub-branch to another without going via the parent, and
changes propagated back to the common parent will not cause major problems for
either sub-branch. There still remains a very large question as to whether
“uncontrolled” propagation of changes is a good idea (as Laura
Wingerd puts it -

“why we don’t allow driving through hedges” – Ch7), and my advice is
certainly to think carefully and plan your normal strategy (with tightly locked
down exceptions maybe permissible). It still comes back to managing complexity:
the more you allow changes to be propagated all over the place in an ad-hoc
fashion, the harder it is to track what has gone one.

As mentioned by Joel, Accurev is a very interesting looking tools and their
stream browser certainly looks attractive, and has raised the bar in terms of
features for the SCM vendors. My impression is that Perforce still has the
advantage in
terms of scalability (see Google paper) and performance, but I would
certainly put Accurev on my evaluation list if advising a client in this area.

I had some fun with the Agile SCM Column at CM Crossroads this month:

A Christmas (Configuration) Carol (Abridged)

We have endeavored in this Ghostly little tale, to raise the Ghost of an Idea, which shall not put our readers out of humour with themselves, with each other, with the season, or with us. May it haunt their houses pleasingly, and no one wish to lay it.

Their faithful Friends and Servants

C.D., R.C., B.A & S.B.

Charlie’s GhostIn which the ghost of (Configuration) Charlie appears to our heroine SCM Sally to help her in her dealings with VP of Development Ben (’Benezer)Scrooge.

Continued…

Sometimes inspirations strikes, though it is interesting what turns out in the end – had a couple of different ideas (and a lot more quotations from the original), but it evolved into what you see. Not quite as provocative as a previous column:

National Treasures of Agile Development

Introduction

Recent research has discovered a very interesting cache of papers about a little known Tribe called the Agile Developers. The first document is a draft dated July 4th (the year is illegible).

(Document starts)

When in the Course of human events, it becomes necessary for some developers to dissolve the bands which have connected them with others, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature’s God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation.

We hold these truths to be self evident,

Continued…

Have been meaning to do some work on P4Python recently, and the first thing I realised I should do is to update the test harness.

This is based around the unittest module so is fairly standard Python. Does some fairly standard things with test suites etc, and provides a reasonably good example of how to use the code itself, so acting as some level of documentation of P4Python.

The old version assumed a pre-existing Perforce server installed and running with some known content. This was fine for my own personal testing but had a couple of problems:

• it assumed the training repository which is fine for Perforce Consulting Partners but which isn’t available to ordinary mortals (so they couldn’t run the test harness locally)
• was a snapshot with existing users and client workspaces so required a license to use – not good for everyone
• it required too much manual setup before running

Obviously a candidate for tidying. So recent work done:

• Change to use the new Perforce Sample Repository which anyone can download and install (works without a license too)
• Change to automatically create and run a new server instance on a fresh install from the sample repository download (automatically unzip etc)

The end effect is fairly nice and automated, and provides a much better ongoing resource to anyone wishing to do work on P4Python. Please note that it currently assumes Windows environment, but will insert a few checks to make platform independent shortly.

To have a look you can either:

• download the complete P4Python sources
• browse the test module
• just browse the Perforce Public Depot using your favourite method: //guest/robert_cowham/perforce/API/python

Note that credit for various aspects should go to Ravenbrook from their work on P4DTI – I stole various techniques from their implementation of a test harness.

It also serves as a reasonable example of a test harness for Perforce scripting, and am very happy to receive comments and improvements (there are no doubt some Python gurus out there who can suggest some improvements at points in the code). I use something fairly similar as a test harness for the VSStoP4 scripts which are actually written in Perl (too horrible to write such a thing in Perl was my feeling)!

Would like to encourage people to take their own copies and give back some improvements in the general framework at least…

Life has been just a touch busy recently having been flat out on various client projects pretty much over the whole summer (managed a week away but only just!). All grist to the mill for future blogging, so hopefully a variety of articles to come!

Meanwhile one of the things I was doing was preparing and then giving a presentation for the (first) Perforce European Conference on 19th September in central London.

I think the papers will be out pretty shortly on the Perforce site, but meanwhile a few highlights and personal notes. There were some big names present and it was good to hear about various practices and principles in operation.

Keynote

Christopher Seiwald did a variation on his slightly “aw shucks” style keynote. Some key points:

• Perforce doing fine: 200,000 users and 4,000 companies
• Company motto: “Aim low and hit!” (do one thing well, remain best of breed and wait for the analyst pendulum to swing back to best of breed rather than suite integration, which it seems to do on a regular basis)
• Working on a variety of things for future world domination, but don’t want to pre-announce as usual
• Very pleased with the way things are happening in Europe, and obviously at the response to this event.
• Next US Conference 9 – 11 May 2007, Las Vegas.
• Sydney office now opened to give global timezone support coverage!

Symbian

Deepak Modgill did a nice presentation on the challanges faced by Symbian for their offshoring. Another in the Symbian series of how their business and vairous configuration management practices have evolved. Not deeply technical but interesting never-the-less.

SAP

Obviously a flagship site for Perforce. Thomas Kroll and Claudia Loff did a good presentation. Interesting how much process and tools they had wrapped around Perforce. A few key stats:

• 4,800 users
• 80+ Perforce servers (but all on same cluster hardware)
• Fujitsu Siemens clusters with 32Gb RAM running SunOS 9
• SAN (mirrored) for main data

They use a very structured process (repository structure and branching scheme) and a parallel (P4SAP) system with its own database to record things like changes and migrations (they call them transports) of releases between different servers. There is also a layer P4MS (Management System) to handle users etc.

Quite impressive.

Process Automation

Obviously my talk was wonderful! I was thought fairly pleased with how it went down and got some good comments afterwards. For anyone interested, the Ruby triggers framework and a couple of utilities are in my area of the Perforce Public Depot.

I will no doubt be blogging on various related aspects (that I haven’t already touched on).

Bank of America

Good talk by Sean Cody and Kevin Breidenbach about different approaches with the bank. They have been replacing ClearCase with Perforce in various groups, mainly due to the performance for shared development between US, UK and India. Experience of Multisite sometimes taking hours to “sync up”, vs. 10-20 minutes max in Perforce.

Another feature of the talk was the power of continuous integration.

Google

Dan Bloch discussed Google’s use of Perforce and in particular how they manage issues around Perforce database locking and identifying and bumping off rogue commands.

Some more stats:

• 3,000+ users
• Single Perforce Repository
• HP DL585 4-way Opeteron with 128Gb RAM
• Linux 2.4 and NetApp filer

Sounds like it wins the contest for largest number of users against a single server!

The details of the lock identification was very interesting and Dan said he would be releasing the lock.pl script and some docs on the Public Depot real soon now!

Perforce 2006.1 Update

A very interesting and technical talk by Michael Shields regarding a variety of performance optimisations made between 2005.2 and 2006.1.

Summary: 2006.1 is quite a bit faster!

Read the slides for more details.

Laura Wingerd

Laura did another fairly technical talk on what has happened to the branching/merging algorithm, and more particularly common ancestor detection algorithm used in various releases of 2006.1. In her usual inimitable style she came up with some very useful ways of explaining things like convergence and divergence of branches over time. Things got decidedly more technical with discussions on common ancestors and I was left knowing I have to go through some of this in detail in a quiet moment just to make sure I really do understand it! The changes with 2006.1 look good, but I did get the impression some edge cases could give some slightly surprising results if you don’t know what’s going on behind the covers (and indeed the driving intentions behind the algorithm).

Summary

Venue worked very well for location. Networking with both Perforce people and various other delegates was as ever a highlight.

Unfortunately the room booked was not huge which meant the event sold out well ahead of time – a shame a more flexible venue wasn’t chosen, but that was only quibble. Organisation well run.

An excellent day!

There are times when you receive a third party code drop which you wish to import. The classic method is documented in Tech Note 15 and its reference to working disconnected (Tech Note 2). The techniques mentioned work very well to find new files, deleted files and changed files.

There is sometimes a fly in the ointment to do with keyword expansion. This is things like a CVS code drop containing expanded keywords:

$Id: //depot/robertcowham.com/main/blog/data/scm/p4_handling_keywords.html#1 $

The Perforce equivalent of this might be:

$Id: //depot/robertcowham.com/main/blog/data/scm/p4_handling_keywords.html#1 $

The simple command to find differences is “p4 diff -se”. If your local version has Perforce keyword expansion turned on then you will get a load of files spuriously identified as having changed where the only real change is in the keywords.

Thus we want a simple script to run through the diffs and exclude any diffs where only keywords are found (note that this includes where the keyword is embedded, such as in a static variable assignment).

The following simple script is a good base for this. It does the job, and performs pretty well, handling thousands of files in a few minutes. It makes use of unified diff format where changed lines have a prefix in the first character of the output.

# Script to import a set of changed files with existing keywords already expanded
# (either Perforce or CVS).
# Does "diff -se" and processes the output

# Args: current directory to check

require 'P4'

p4 = P4.new
p4.tagged
p4.connect

def process_file(p4, f)
  diffs = p4.run_diff("-f", "-du", f)
  real_diffs = Array.new
  diffs.each { |line|
    case line
    when /^====/
    when /^\@\@/
    when /^ /
    else
      if line !~ /\$Id|\$DateTime|\$Revision|\$Date|\$Author|\$Name|\$RCSfile|\$Source/
        real_diffs << line
        # puts f, line
      end
    end
  }
  if real_diffs.size > 0
    print "Editing #{f}\n"
    p4.run_edit(f)
  end
end

all_files = p4.run_diff("-se", ARGV[0])
print "Processing #{all_files.size}\n"
i = 0
all_files.each{|f|
  i += 1
  # print"Processing #{i}\r" if i % 10 == 0
  # print"Processing #{f['depotFile']}\n"
  process_file(p4, f)
}

It is pretty easy to run, e.g.:

diff_se.rb ...

The net result will be a list of files checked out (p4 edit) in the default changelist.

Note that one of the big advantages of Perforce branching and merging is that it handles merges neatly when keyword expansion is used between branches (and thus you don’t get spurious conflicts).

CVS Imports

If you use the cvs2p4 scripts to import a CVS repository you can end up with a slight problem since the conversion copies the CVS archive files (in RCS format) and Perforce uses them unchanged. The problem comes about because CVS stores the keywords already expanded in the RCS archive. Perforce stores its RCS files with the keywords not expanded, which makes it easier for it to do the merging between branches (without keyword conflicts). While Perforce can handle a CVS archive with the the keywords “pre-expanded”, it does lead to spurious merge conflicts. Note that this problem is only really present during the early merges after the CVS import. It will no longer be present as soon as the base file for any merge is fully in Perforce format (i.e. after at least one merge has been done).

Writing good Perforce triggers, and, more importantly, debugging them in live use, turns out to be one of those things that seems simple but has lots of tricky issues that can lead to lots of time being wasted.

In spite of thinking that I understood lots of the issues, I still spent a couple of hours recently debugging a problem that turned out to be a combination of environment and password issues. This was particularly annoying as I had rather though I knew about this stuff (and indeed have advised people over the years about it!), and yet was blindsided and caught out by some issues I had forgotten about or not thought through deeply enough.

I reserve the right to revisit this subject more than once in the future with further insights and news…

Assume Nothing About The Environment!

The classic approach to triggers is to write a nice script (Python or Ruby for me these days – no Perl, though just occasionally I miss it!) and debug it by running with the appropriate parameters from the command line (e.g. create a pending changelist and pass in the pending changelist number). This does indeed tend to turn up a number of issues, but the good thing is you can usually debug them with the appropriate command (<rant> why does python require you to execute pdb.py which isn’t by default put in the path on Windows machines, and why does Ruby not learn from Perl and for example use -d as a parameter to debug things instead of “-rdebug” – very unobvious!</rant>).

The major problem turns out to be the fact that the trigger is executed by the Perforce server process and may have a very different environment to what you might think as you run a “login” session. One sort of expects this on Unix, but on Windows it can be particularly surprising how little is in the environment due to the username that the Perforce process is running in when it is running as a service (default installation on Windows).

Thus the first rule of trigger writing is “assume nothing about the environment!“.

It is very easy to forget this and assume very simple things, like:

• P4PORT is always defined
• P4USER is always defined
• failures of individual p4 commands within the trigger will be obvious

Thus immediate recommendations are:

• Give full pathnames to executables. For example, “/usr/bin/ruby” or “C:\ruby\bin ruby.exe” as the initial parameter for the ruby script, rather than assuming that “ruby” or “python” or whatever will always be in the PATH of the user executing the command.
• When in doubt (I’m generally always in doubt) give full pathnames to scripts too.
• Pass in as parameters the p4port and any other parameters to be used rather than expecting them to be already present in the environment.
• Within the script, explicitly add any extra directories to the search path for commands such as “import p4″ in Python or “require ‘P4′ ” in Ruby or any equivalent import-type statement, unless you are absolutely sure that the imported libraries are globally installed on the machine your are working with. Don’t assume the same directory as the trigger script itself is in is in the path unless you can prove it.
• Trap and print to stdout (or stderr which goes to the p4d server log file) any errors/stack traces including exceptions from your p4 interface to aid hunting out problems. This is much easier to say than to do!

Passwords Cause Problems

In the good old days, before “p4 login” was even a twinkle in Christopher’s eye, you could write your trigger assuming super user privileges (says in Yorkshire accent “we had it tough – could only dream of admin privileges in those days”) and everything would work.

Life became substantially more complicated with security level 3 and login being required. Commands failed due to not being logged in, and this turned out to be a bit of a bugger (’scuse my French) to work out (why it had failed that is).

Received wisdom is “run your triggers as a special trigger/admin user, put that user in a special group with timeout of some very large number, log them in manually and all will be sweetness and light”.

The interesting thing about this approach is that it often works, but as I discovered recently, can flatter to deceive. The problem I had was that the super user was indeed in a special “long timeout” group, and logged in on the same box (generating a suitable ticket). However, as I discovered only after some hair was torn out, the P4PORT that the user was logged in under was different to that used by trigger and thus the P4TICKET file entry was also different and the existing “login” had had no effect and my trigger was unfortunately failing silently.

Thus P4PORT=localhost:1666 where localhost=some_server.some_company.com will not work if the superuser is logged in using P4PORT=some_server.some_company.com:1666, since the latter is what will be in P4TICKET and the former will not be found and thus commands will fail. Be warned and expect/check for this! [Note: this was fixed in p4d 2007.2]

When in doubt print out the environment within your script (via some sort of debug parameter).

Belt and Braces

My current intentions on this front are to produce a trigger framework that helps detect the above problems, and helps both avoid them and, when necessary, debug them in a (relatively) painless manner. This, at the moment of writing, is a work in progress, but I hope to be able to share it with the wider Perforce community as it emerges into the glare of publicity. I do reserve the right to retain the right of surprise to add some slight spice to my upcoming presentation at the European Perforce User Conference on the 19th September (in London).

Update: hopefully will be able to share a rework/expansion of Tony Smith’s P4Trigger.rb framework which addresses some of the above issues fairly shortly – seems to be working at a client – time will tell – but fairly quickly.

Future topics will include ideas on test frameworks etc.

This is a partial review of Practical Perforce by Laura Wingerd, published by O’Reilly (ISBN 0-596-10185-6). The reason it is partial is that I intend to comment in more detail in future blog articles on some parts of the book, and wanted to post this without waiting for the whole thing!

As Laura mentions in the preface, the book is not intended for complete beginners, but more for readers with experience in other SCM (software configuration management) tools who are looking to understand how Perforce works.

To quote the introduction, there are two parts to this book:

• Part I (Chapters 1-6) is a whirlwind technical tour of Perforce commands and concepts. It’s not a tutorial, nor a reference, but helpful nonetheless.
• Part II (Chapters 7-11) describes the big picture, using Perforce in a collaborative software development environment. It is particularly strong on branching patterns, how to structure codelines and tips and tricks in this area.

The real meat of the book for most Perforce sites is thus Part II, but there are definitely some goodies in Part I.

Chapter 1 presents some fundamentals about Perforce syntax and concepts. The diagrams on pages 6 &7 explain the relationship between revisions and changelists very well.

In Chapter 2, Laura discusses client workspaces and things like view syntax. She also describes basic check outs (open for edit in Perforce command line parlance), and introduces branching when she refers to cloning of files. She includes concepts of renaming and replacing content in files, reconciling changes made offline, and even introduces a couple of bits of undocumented syntax such as “p4 files @=1452″. Quite a chunk of information in this chapter.

Resolving and Merging are the subject of chapter 3 and includes some very useful diagrams showing various scenarios. If you have ever had any questions about 3-way merging in Perforce – read this! On pp68-69 she gives examples of reconciling changes you have made to a file someone else renamed using the undocumented merge3 command – interesting if a touch esoteric (also referred to in “How to undo a merge” on p80. The recommendation on p74 to sync and resolve one changelist at a time is certainly worth considering, although I think it will depend on your environment as to how necessary that is.

The basics of branching are covered in chapter 4 including initial scenarios and how to track merge requirements across branches. She makes quite a lot of use of the interchanges command (not yet exposed in the GUIs) and explains the gory details of “yours”, “theirs” and “base” nicely. Her approach of using filespec integrations for the initial examples is nice and simple, but I suspect more people are likely to use branch specs in real life. On p111 she gives a useful couple of commands to show how to find which changes have been merged in (more likely to be automated in scripts for most sites I would expect). Other subjects ocvered include all the gory details of what integrate actually does, as well as some very useful details as to what the interchanges command can tell us, particular with respect to cherry picked integrations.

Chapter 5 is quite short on labels and jobs and shows all the basics. A quick note on the final section where a job is used as a reference for a changelist – as of release 2005.2 there is an undocumented “dynamic label” option where a label can have an attached revision which probably makes the job trick unnecessary.

Chapter 6 gets into the subject of remote depots and proxies and also mentions the very useful spec depot option (automatic versioning of all spec objects). There is also some good advice on using p4web in browse mode to access your repository. The section on triggers and automation is a little light, but understandable.

Part II starts with Chapter 7 “How software evolves”. This chapter is perhaps the highlight of the book, and introduces concepts that are totally independent of Perforce and apply to many SCM tools. Fortunately the chapter is currently available as a free PDF document from the O’Reilly website for the book. A firm understanding of the concepts introduced here will make it much easier for you to come up with suitable branching patterns for use in your organisations, and also, perhaps more importantly, give you some incredibly useful concepts for explaining your structure to other people within the organisation. Most SCM problems are due to poor communication rather than poor tools, or poor ideas. Laura relates the problems in the real world prevent us from an overly simplistic ideal world, and yet how some simple concepts allow us to manage this real world complexity. The “flow of change” and the “tofu scale” are classic concepts which should be in everyone’s SCM vocabulary.

Summary

I am going to stop this post here, and will get to further chapters and some detailed comments on them as I have time.

But I will finish with the recommendation - buy this book!!

Tools Fair on 15th June – Cradle to Grave Support

As Chair of the BCS CMSG now (Shirley Lacy passed on the baton in November but fortunately for me remains in the background as vice chair), I have been feeling a little concerned about responsibility for the BCS CMSG Tools Fair – Cradle to Grave Support on 15th June. I am relieved that things are looking good now with good selection of sponsors supporting us:

Some newish names to me not being hugely up on the service management/ITIL side, but change, configuration and release management are a major focus in that field so look forward to meeting and hearing about them and the issues and solutions available.

A couple of big names missing on the software side which is a shame. A couple of organisations undergoing a re-org and thus no marketing budget to spend (well not now anyway). IBM/Rational aren’t there because I can’t find anyone to talk to that would appear to be able to make a decision. We used to have good relations with Rational but since they were acquired by IBM can’t get anything out of them (I wonder what their responsiveness is if I were looking for support or trying to buy a produce?!). If anyone knows who to contact in the UK on marketing/events I’d be grateful for a heads up. The other one I have failed to get to anyone appropriate is Microsoft – would be good to find out about Team System etc, but all enquiries get passed from pillar to post and a deafening silence is the result.

Show me the money – identifying the value of Configuration Management

The other event we have on the 27th April is an evening event which is a relatively new departure for us. This will be at London South Bank University in their conference centre (which we have used before), and is in the form of a workshop lead by David Cuthbertson. This is going to focus on selling the benefits of CM and we hope will be a useful way of bringing together both new and more experienced people in the field to share experiences. Details to go up on the web site very shortly! David has spoken several times in the past and always gotten excellent reviews. He is also skilled at running workshops and getting contributions from others present. Should be a great evening. Apologies to those not close enough to London who want to attend, but we are looking at running (in particular evening) events elsewhere in the country – get in touch if you have any ideas.

Updated: 2005-11-29 – see link to scripts.
Updated: 2006-03-16 – some clarifications and link to Miki Tebeka’s scripts

A common branching pattern is to have mainline and then task branches where work is done and then “published” by merging to the mainline as shown in Figure 3 of the article Building for Success.

In perforce the “publish” and the “catchup” are both performed by using the integrate command, typically with a branch spec. For example, you might have a branch spec

Branch:	task/fred
View:
	//depot/main/... //depot/task/fred/...

The key thing is that Fred should “catchup” before doing the “publish”. This is so that the more risky merging is done in his task branch and not in the mainline. When doing the merge Fred should be bringing all changes by other people in the project into his branch and with the publish he should just be able to copy his code into the mainline.

There are several ways to achieve this:

• Education – tell Fred what to do and rely on him doing it – so what happens if he doesn’t – can you “persuade” him not to?!
• Don’t give Fred write access to the mainline (e.g. via protections or a trigger), and instead have the integration team do it. The problem then being that the integration team may not know the code as well as Fred and are perhaps more likely to make a mistake.

The basic way of detecting in Perforce whether a catchup has been done is to do a preview integrate from main to task/fred and check that nothing needs to be done, so check for no results from:

p4 integrate -n -b task/fred

If the above produces no results then proceed to do the publish:

p4 integrate -r -b task/fred
p4 resolve -as

The key step is that the “resolve -as” (safe automatic merge), resolves as many files as it can. It looks to see if their are only “theirs” (source) diff chunks or “yours” (target) diff chunks and will select the theirs or yours file appropriately. (Note that “theirs” and “both” or “yours” and “both” are also processed in the same way). The key point is that if there are “theirs” and “yours” (or “conflict”) diff chunks, then safe automatic resolve will not process that file.

Of course having done the automatic resolve and with the changed files sitting in our client workspace it is usually a good idea to do things like a build and smoke test – there’s not a whole heap of trust otherwise…

Thus, in our script we can check for anything not safely resolved automatically (resolve -n shows what still needs to be resolved).

p4 integrate -r -b task/fred
p4 resolve -as
p4 resolve -n
if any results from above command then exit with error

build
if any problems then exit with error

run smoke tests
if any problems then exit with error

If no errors at this point we are ready to submit.

There are some extra wrinkles to this:

• the “resolve -as” may validly not work if you have done a merge with edit during the catchup (”edit from” in the terminology from p4 integrated). You need to detect such situations and do a “resolve -at” to copy them over.
• as soon as one person has done a publish then all other branches will require to do a catchup to pull it in – this means publishes are going to become serial
• there is a window during which the publish is being performed when someone else might sneak in and do a publish thus meaning you need to do a catchup – consider a simple “locking” strategy to prevent this.

Note that the build and smoke test steps need to take an appropriate amount of time. If it takes hours to do them then the process is unlikely to work. Thus a few minutes or tens of minutes is likely to be the limit – this may mean cutting down on the number of tests that are executed – but that is usually OK. Apply judgement!

Although at the last point you might think it would be a brave person to do the submit automatically in a script! It’s probably a good idea to do things like run diffs and give a visual once over before finally checking in, but it should be a pretty easy decision at this point.

In terms of automating the above, I can heartily recommend the various scripting languages with built-in calls to the Perforce API: Ruby, Python and Perl. Getting the results of a command is easy, and exceptions allow you to make error handling pretty easy as well.

Very brief examples of scripts designed to perform the above check in Python and Ruby are now available online. Note they are designed to be run from Custom Tools menu in p4v/p4win. Please note that Miki Tebeka has kindly published more production quality script implementations in python including a GUI front end. He also includes an example of an installer to automate installation of tools for p4win and p4v – check them out. Thanks Miki!

p.s. The above works for any codeline for which you have responsibility for accepting changes – it doesn’t have to be a mainline – it can be a subsidiary integration line with third parties contributing, or team members “proposing” changes. You could automate the script and put it on an intranet page – let people try it out, and if successful then have changelists auto-checked in.