Editorial: Can We Change For The Better?

I've just read in the 27th August copy of New Scientist about some experiments with "cultural transmission" - one chimpanzee from each of two groups was taught a method of getting food from a complicated feeder. When the groups were allowed to watch, they followed the lead of their respective expert in spite of one method being more efficient. Even when some members of the less efficient group learnt the better way the majority reverted to the original inefficient strategy.

I once worked in an organisation where two teams adopted new development practices and both significantly outperformed the company norm for projects of the sort. One team embraced unit tests as part of the deliverable for each piece of work, the other adopted a strong independent verification culture. (Depending on the metric they were around 60-100% more efficient than would be expected from historical data.) Attempts to introduce practices from either team to the other were singularly unsuccessful - they'd be tried briefly and, even if they worked better, were soon abandoned in favour of the original strategy.

It may seem a long step from experiments on animals to software development, but if Pete Goodliffe can draw comparisons between programmers and monkeys, then surely I can be allowed one between developers and apes. I've seen many attempts to introduce more efficient ways of doing things fail in just this way. Depressing isn't it? (By the way, the New Scientist article is at http://www.newscientist.com/article.ns?id=dn7913 - it summarises a report in Nature and gives the reference as "DOI:10.1038/nature04047".)

I've given an example of the failure to adopt new habits from developing software, but there are others from other aspects of life (without stopping I can think of examples of driving, drinking and housework habits that illustrate the same theme). People, like the other apes described above, learn habits from their peers and then are very reluctant to change them. And groups that have a way of doing things will try to impose them on a newcomer with a better way rather than embracing change.

In the two teams described above there were two mindsets at work - one believed that developers could be trusted to have the discipline to produce working code, the other that developers need to be policed to impose the discipline to produce working code. The incompatibility of these mindsets was demonstrated when management shuffled the team members around to staff new projects. The new projects were far less successful because team members were being asked to work in ways that they knew were less effective than when they'd worked on one of the successful projects. The intended "spreading of new ideas" backfired as each camp demonstrated that the unfamiliar approach was unworkable. When the dust settled, both approaches had been abandoned and the community returned to the comfortable practices that pre-dated these two innovative projects.

Change is hard - I know that. I resist change all the time: I use C++ for some tasks that, for example, Python is more suited to. (The last time was "flattening" a directory structure while avoiding name clashes by incorporating the path elements in the target file names. And yes, there are probably tools that would be even better than Python.) Part of the reason is that I knew immediately how to do it in C++ but would need to look things up in Python (or any other tool) - this introduces unpredictability into the length of the task. But equally, I know that if I were to spend the time learning the tools to do these things that way I'd soon doing them faster.

Sometime in the last year a colleague was having trouble avoiding a lot of duplicate code and asked for suggestions. After a quick explanation it appeared that the problem was deducing an appropriate intermediate type for processing before conversion to a target type. I showed him a trick that introduced the needed "extra level of indirection" (what the cognoscenti would call a C++ template metafunction). He was initially very taken with this technique as it factored out the variability in just the way he was trying to achieve. But after a few days of working with this approach he went back to duplicating the code (with minor variations) for each of the types he was working with. Maybe a template metafunction was wrong in this case and there was a better solution (Adaptor pattern, perhaps), but the point is that he stuck with the comfortable old way of doing things. (And he did want to change - he had asked me for a way to avoid all this duplicate code in the first place.)

I've heard countless stories from developers that have successfully fought against the accepted way of running a project in order to introduce practices they then proceed to demonstrate are better. After which the organisation, for no very clear reason (and with a great sigh of relief), reverts to the "normal" way of doing things. And, in these case, the people involved will frequently have all agreed that the changes should be adopted.

The same report in New Scientist tells another story that indicates that there is hope. This story is of a killer whale that invented a new way of catching seagulls. (For those that are interested in catching seagulls the method is this: he regurgitates some fish onto the water's surface; then he waits below the surface for a seagull to appear and take the fish, whereupon the whale lunges catching the seagull in its mouth.) Whales do not patent such innovations and this idea spread rapidly through the local whale community. Clearly the reward of a tasty feathery snack was enough to overcome any natural reluctance to change. Or perhaps whales are just more intelligent than apes?

Maybe changes in behaviour are successful for reasons that have nothing to do with the effectiveness of the new behaviour? Certainly there is little evidence for many of the "best practices" adopted in our industry (and even less for many "standard practices"). There are plenty of practices in common use whose only merit appears to be that they are in common use and, therefore, one cannot be blamed for using them.

The role of cultural groupings in the adoption of new practices is also significant. An idea will only succeed if adopted by the majority of a group. This must make small groups more susceptible to change - an idea can circulate quickly around a small community and gain acceptance, whereas most of a large community may never become aware of it at all.

A lot of small communities all following their own whim means that a lot of ideas are being tried. My question, therefore, is whether there is any way to tell the good ones from the bad? If this is easy and the good ideas stay, then the division of the C++ community I wrote about in Overload 67 may be a blessing. But how to tell the good ideas from the bad?

It may sound easy - the ideas that work are good, the ones that don't are bad - but it isn't. Consider the two teams I described above. Both had ideas that worked, both had successful projects that appear to have proved them right. But the ideas didn't work together - indeed the two groups had difficulty finding enough common ground to establish a dialogue.

If you believe developers cannot be relied on to validate their own code then discussion of programmer-written tests is missing the point (and risks both the tests and code making the same error), while if you believe in testing your own work the idea of co-ordinating with an independent tester seems cumbersome (and risks disrupting the flow of work).

And it is rarely as simple as comparing two ideas. There are a lot of ideas in play simultaneously in any software development project. Projects vary in many ways: budgets, technology, etc. Teams also vary in size, experience and skills. In short, a direct comparison between two ideas in software development is rarely possible.

I started with an example of two teams in the same organisation, which eliminates a lot of variability. The teams were of similar size and experience and the projects had comparable time and staffing budgets and were based on the same technology. And even then there was no clear answer - except that by changing their development processes both teams improved on past results.

While these teams both succeeded in producing good results the benefit of this was short term. A larger group, the organisation, reacted in the same way as the apes in the New Scientist story - although a few individuals learnt a better way of developing software, the organisation as a whole reverted to the earlier, less efficient, methods. (And the individuals who wanted to practice what the organisation had enabled them to learn chose to leave.)

Knowing about a problem is not enough, nor is knowing the solution, nor even intending to apply the solution. Change requires a firm intent that doesn't come easily or cheaply, which is why habits are so persistent. Not changing also has a cost. The difference is that change is a one-off cost whereas not changing is a continual drain.

There is a very interesting attempt to change being made by the Commonwealth of Massachusetts. They appear to have realised that the choice of data formats for storing documents is significant. As they have an "open government" obligation to ensure that the information is accessible to members of the public both now and in the future, they are required to care about this.

In order to address this need they have been working with a range of parties in the software to identify a strategy that meets their goals of accessibility to these documents, and have adopted one based on open standards for their document formats. The point of this is that anyone can produce software that works with these formats now, or at any time in the future. And by working with suppliers on the choice of standard they have ensured that suitable software is already available - no doubt more will follow if this initiative survives.

Why do I say "if"? Well, they have a good technical solution to a problem, but that is far from all that is required to make a change. I'm sure that their current principle supplier of office applications would prefer not to have competition. (And Microsoft has considerable influence.)

After decades of suffering incompatibilities between office applications from different vendors, and even different versions from the same vendor, I think a change to a common standard could be a change for the better. The opportunity for change is here, and there has recently been a dramatic demonstration of the costs of a lack of interoperability in FEMA's handling of Hurricane Katrina.

Whales can learn new, improved ways to do things. Can we?