Three Phantastic Tales

When people work together (and most software development involves people working together) they are often not pulling in the same direction. When you notice that others are pulling in a different direction it is natural to assume that they are the cause of the problem. After all you know that you are not doing anything stupid. But in reality the behaviour of your colleagues isn't stupid, it is just strange, because you don't understand what they are trying to achieve. And you can't fix what you don't understand.

Strange behaviour requires explanation, and the form the explanation takes reflects the prevailing cultural context. Behaviourists will talk of "conditioned responses", psychologists of "archetypes", evolutionists of "memes", and I'm going to talk of ghosts. On one level these particular ghosts are a narrative device but, on another, they are very real and pose a danger to any project that is visited by them. Anyone experienced in software development will recognise the spirits in the stories that follow. I have met them many times with many names but, to protect the people who have been possessed by them, I have chosen to use names that reflect their essence.

These shades are Mr Deadline, Seymour Checks and Noah Shortcut; the first of these is a project manager and the others are developers. Each of them contributes to the failure of a project, although each is working in a way that is a rational response to the way they see events unfolding.

Like most project managers Mr Deadline has a lot of demands on his time. He keeps the customer and management informed and contented with progress of the project. He ensures that the equipment, software and developers required for the project are available when they are needed. And he ensures that work is allocated to and completed by developers.

With so many demands upon his time he seeks simple strategies for satisfying them: create a plan against which he marks off progress, predicts when resources will be needed, and records the allocation and completion of development tasks.

At the beginning of the project he gives out the first tasks to Seymour Checks and Noah Shortcut and, happily, both tasks are completed on time. But, as the project progresses, he finds that although Noah continues to complete his work on time Seymour takes longer and longer to complete his work and the project falls behind the planned schedule.

Adding developers to the project helps a little, but none of them are as productive as Noah Shortcut (or as slow as Seymour). Mr Deadline ensures that all the critical elements of the system are completed on time by giving them to Noah, while anything less urgent (or unplanned) is passed to the less reliable developers (Seymour and the others).

The project continues to fall further behind schedule and, additionally, some serious bugs are detected during testing and acceptance testing leading to significant delays through rework. Eventually, the planned delivery date is reached without all the work being completed to an acceptable standard. The project has failed to deliver (and, because of the extra staff, is also over budget).

Sometimes a project will be cancelled at this point, but in this case the project continues. Mr Deadline is required to fix the problems as soon as possible but also comes under pressure to release staff to the next project. In the hope of avoiding a repeat of these problems the next project is staffed with the most productive developers. After a while Seymour is left as the sole developer on the project to fix the remaining problems.

Eventually, all the problems are fixed and the project brought to an (unsuccessful) conclusion.

Noah is bright, eager and understands the need to minimise the amount of time and effort spent on the project. From the moment he receives his first piece of work he is trying to avoid any activity that would delay the delivery of that work. When he looks through the documentation for that work there may be a few things that he doesn't fully understand, but he can see what classes and functions he needs to write - which is all he needs to start cutting code.

Once the code is done the job is almost over - he just needs to integrate it and check that it is working. He spends some time exercising his code through the debugger "to make sure it works", fixing any problems he encounters, and he can soon announce that he's finished.

As time passes his confidence grows: he always finishes on time and is trusted with all the important new stuff to write. He's also very aware that the project is behind schedule - and he does his best to catch up by finding new shortcuts through the project processes. In particular, he finds that he can reduce the time spent checking his work: if the testers find problems he can fix them easy enough; and, since they don't find many, this approach avoids a duplication of effort.

When the last piece of functionality is handed over Noah feels a sense of triumph - there are probably a few bugs to fix, but the hard part is over. And, in recognition of this achievement, Noah and the other great programmers are moved onto the next project to work their magic there!

Despite the impression one may gain from Mr Deadline's story, Seymour writes reliable code quickly. Why then does he take so long completing his tasks?

When Seymour receives his first piece of work he reads through the documentation and makes notes on anything that isn't clear and on how he will prove the code that he writes (to be specific, he does this by writing automated tests). Then he seeks clarification on all the issues, writes the code and checks it works (by running his tests) before announcing he's finished.

As the project progresses he finds that more and more of his work relies on existing code. Where this is code Seymour wrote himself it is clear what the code should be doing and there are tests that demonstrate that this is indeed what it does. Where another developer wrote the code this is not the case and it is frequently unclear whether the code achieves its intent. At first Seymour assumes that his colleagues have validated their code. But after repeatedly finding that his code is failing because of errors in the existing codebase Seymour becomes disillusioned with the slapdash work of his colleagues.

Because in addition to the work assigned to him, Seymour is fixing problems in existing code, he begins to fall behind Mr Deadline's schedule. Seymour is conscious of these delays and especially of the length of time it takes to prove that the problem isn't in his new code and to locate it. So, in an effort to find and correct these problems effectively, he takes to writing tests for any existing code he uses that is missing tests and fixing the problems he discovers. However, as more and more code is added to the project (and as changes are made to the production code without updating the tests) the effort of doing this leads to an even greater overhead to Seymour's activities.

Only when the codebase in the project begins to stabilise (because no more features are being added and developers are leaving the project) does it become possible for Seymour to make progress in addressing the many bugs hidden in the codebase.

Seymour is the last developer on the project: the hero tracking down and fixing the problems that others left unresolved. Eventually he succeeds: the system reaches an acceptable standard and work on the project is brought to a close.

Clearly the above tales are different views of the same failing project, and each of the tales describes someone who is doing their best to ensure that the project succeeds. There is no evil villain plotting to prevent the success of the project, nor anyone doing anything that is obviously stupid at an individual level. The problem lies in the interaction between individuals - our spirits do not consider the effect that their actions have on other project members or the project as a whole:

One reason that I've discussed these stories together is that these spirits travel together. Once a team member succumbs to one of the spirits they (unintentionally) encourage the other behaviours. This point is important when effecting an exorcism, so we will examine this mechanism more closely now.

By not enforcing an adequate quality check on the work done Mr Deadline creates an environment that encourages cutting corners. The spirit of Noah Shortcut will soon possess anyone who looks for the simplest way to complete a task. The opportunity to cut corners also affects Seymour Checks - although his selfdiscipline is sufficient to keep him from shortcuts it also tempts him to the opposite excess (introducing redundant tests). Mr Deadline also fails to ensure that rework is recognised as a continuation of the original task; this creates an environment that encourages Seymour to "just go ahead and fix it" and keeps from Noah an awareness of the cost of his carelessness.

Noah Shortcut's need for speed will lead him to skimp on any quality checks included in the project process (review meetings, tests or whatever) and to discourage any such "time wasting" procedures. Mr Deadline is eager to speed things up and will listen sympathetically to ideas that will "save time". At the same time Noah is leaving a trail of careless mistakes through the codebase - while each time Seymour is tripped up by one of them he becomes more determined to root them all out.

Seymour Checks is keeping key information to himself: the cost of the time spent fixing other people's mistakes. To him it is reasonable: by the time he's found the bug, it is quicker and easier to fix it than to explain it to someone else (who is probably in the middle of something important). But if Mr Deadline isn't aware that rework is happening (and that it is mainly in work produced by Noah) then he will assume that the code is of an adequate quality. Equally, if Noah isn't made aware that he is making mistakes he will not try to rectify them.

The first step is to be sure that what is going on really matches the tales I've told. Not every project manager is Mr Deadline, not every quick programmer is Noah Shortcut and not every slow programmer is Seymour Checks. But they are easy to recognise once you know their characteristic behaviours.

Now, it is pretty well known that if people are to be changed they must first want to change. And unless your colleagues (or you) realise that they are possessed by one of these three spirits then they won't take any steps to exorcise them. Accordingly the next step is to explain to them just what is going on. This isn't easy because, as the tales illustrate, each of them is already doing his best according to his understanding. The stories are a useful device to sidestep the tar pit of telling people they are doing something wrong. It is human nature to react defensively to such a confrontational approach (which allows the ghosts to entrench themselves). The stories are much less threatening - so feel free to use them to present the case for change.

Until I came up with these stories I struggled to get the necessary ideas across. Naturally the stories are not enough: you must tell them at the right time and have evidence to show that they apply to the current project. For example: you might find occasion to tell your project manager how substandard work products can impact downstream tasks and use the story of Noah Shortcut to illustrate that his fastest programmer could be a liability. He won't know if this is really what is happening on his project, so you also need evidence that Noah's shortcuts are costing other people on the project time. In one recent case I'd just told this tale to the project manager when the integration test team complained that it had been stalled for two hours trying to compile the system. A little investigation showed that Noah had checked in a change without first building the system properly. (This led to a tightening of procedures and a willingness to consider the other tales.)

There are no guarantees: the battle isn't over once there is agreement that there is a problem - habits do not change easily. The ghosts will still put up a fight! You will need to have an answer to the argument that "there isn't anything we can do about it". The different spirits have different ways of putting it:

You need answers that fit your organisation (like using pair programming, test harnesses or code reviews to ensure the standard of work). These are all techniques that focus on quality - but the truth is that is it quicker to develop things right than to develop them wrong and then fix them. This is often a hard cultural change for an organisation because the connection between quality and progress isn't easy to make explicit.

Whatever working practices you try to introduce you also must lead each of the affected individuals to realise that their habits are causing problems on the project. Unless the majority of these individuals are prepared to change at the same time the individuals concerned will return to the "comfort zone" of their habitual behaviour and the project will slip back into failure mode.