Title: Professionalism in Programming #20

"Change in all things is sweet." [Aristotle]

If only software grew like plants. You'd put the seed of an idea into some fertile programming soil, add a little water and keep the conditions just right. Maybe at first you'd have to do some work to tend it: put a little rod in to help the thing grow upright, and cover it to keep the birds off. In time a seedling would sprout and grow, and when the program plant was big enough you'd be able to release it to the world. If it needed a little extra functionality you'd just keep watering it, perhaps add some fertiliser, and it would continue to develop. The trunk would strengthen in order to support the new branches and the plant would keep in perfect balance. If it was growing in a direction you didn't like then a little pruning would soon set it straight.

Ah, if only it was an ideal world. Sadly it's not. Not by a long chalk.

The truth of the matter is that software is a live entity. OK, we're not quite to the point where it's sentient or organic, but it has a life of sorts. It goes from being conceived, develops steadily, then comes of age and is sent out into the big wide world to make its living, hopefully garnering respect, admiration, and ultimately fulfilment. It may continue to develop, perhaps to the point where it gains a bit of a middle age spread and no longer has the handsome looks of its youth. Over time it gets tired and old, and is eventually retired, put out to pasture in the digital knacker's yard where it can gracefully die.

That's an idealistic view of the lifetime of a piece of software, but it's reasonably accurate. We need to look at how we cultivate our programs, especially after the initial round of development is done and dusted. Unfortunately, programs require thoughtful tending and seldom receive the care and attention they really deserve. What can we do to prevent early death from a slowly spreading code cancer?

To answer this we'll work backwards. We'll take a look at the symptoms of bad code growth, explore how we grow our code, and using this determine some strategies to develop healthier software.

Bad things happen to good code. No matter how well you start off, no matter how good your intentions are, no matter how pure your design and how clean the first release's implementation, time will warp and twist your masterpiece. Never underestimate the ability of code to acquire warts and blemishes during its life.

The 'maintenance' phase^[2] of software development is always the longest, and overall where most of the effort goes - even if this effort is not scrunched into the compact focused ball that the initial design/development work is. We must explode the myth that software only develops during its initial stages of life. Boehm states as much as 40-80% of total development time is spent in maintenance [Boehm].

During the initial development stages you can keep a firm grip on the code and work it around as much as you like, within the available time constraints. After it has been released you're generally more restricted. These restrictions may be practical:

These restrictions may be psychological, the developers' (erroneous?) preconceptions:

simple lack of understanding - another programmer may not understand original author's mental model of the code, and this prevents them making the most appropriate modifications.

Despite all this, after a release code is never expected to stand still. No matter how well it was tested there will always be odd faults that crop up which will need fixing. Customers demand that new features are added. Requirements change under the development team's feet. Assumptions that were made during development prove to be incorrect in the Real World, and require adjustments.

The bottom line is that more code gets written after you think the project is completed. Where the fine line lies between 'maintaining an existing product' and working on new development of the 'next version' is a moot point. Whatever you call it, the original code base gets modified. Sometimes by the original author, often not.

And this is where the rot sets in. In fact, it's a damned-if-you-dodamned- if-you-don't scenario. If you never again touched the code, if you don't keep the program up to date with fixes and modifications, the program will degrade. In the worse case it will stop working as the platform it runs on changes, or the assumptions made become out of date. The 'Y2K' bug is a glorious example of this. Maybe the program will just putrefy as competing solutions develop more features, and gain more popularity. Untouched code slowly rots away.

However, if you do make extensions and fixes, the code doesn't only grow. It also rots. Fixing a fault usually sees the programmer introduce more faults as a side effect. Brooks found that as many a 40% of fixes introduced new faults [Brookes]. The Programmer's Drinking Song (sung to the tune of '100 Bottles of Beer') written by a minstrel unknown, sums this up neatly:

It's not only newly inserted faults that can rot our code. Even functionally working modifications can cause code blight. Quick and dirty fixes pile on top of one another, putting nail after nail into the original neat design's coffin. The more you maintain carelessly and degrade the structure of code, the harder future maintenance gets. The plant analogy above is a good one. Many, many quick modifications don't make the trunk grow any stronger. The more heavy branches you add to the top of the system, the less stable the entire code base is. Eventually it totters over.

Does all this sound unduly pessimistic? Surely code won't 'rot' if you're careful? Adequate care does not seem to be taken in today's software industry. It's a culture thing. Programs have a habit of hanging around much longer than they were ever intended to. So many quick hacks live on, well past their expected life.

We should be on the look out for rotten code. Beware of the telltale signs of bad code, whether you're writing it now or stumbling across someone else's mess. Rot sets in with any change that leads to a lack of clarity, or makes the system more complex. Unnecessary complexity comes in many guises. Here are some of the flashing red lights and klaxon calls:

Many of these forms of rot are particularly visible in the code, and can be seen with a quick inspection or even the use of certain tools. However, there is a class of more subtle 'invisible' degradations that usually occur at a higher level than the syntactic gunk. Modifications that fudge the original code architecture or subtly work around normal program conventions are much harder to spot until you're immersed deeply in the system.

Code rot often sets in more voraciously when the original author of a piece of code leaves the company or project. Although code ownership is not necessarily a good thing, and is seldom written in stone in a company culture, the creator often takes a pride in their work and does housekeeping on their source files, even when other people make modifications. Once they've gone, this maintenance role slips away, and their files begin to rot more quickly.

Why do we make such a big mess of code? The answer is simple: complexity. A program is a huge collection of information organised on several levels: the overall architecture, its design in a particular language, the interface mechanisms to the outside world, the actual implementation of each little bit of code, and so on. That's a lot to understand before you start working with a chunk of code. There is seldom enough time to work out how a few lines actually work, let alone how they fit into the overall picture. We haven't yet learnt to manage this vast complexity.

No code development really follows the classic model of: lock down all requirements, design completely, code completely, integrate, test, release. Unexpected modifications happen to an existing code base. New pieces get grafted in somehow. It's an incremental development cycle towards ever shifting goalposts.

In reality code growth happens by one of the following mechanisms, loosely ranked in order of disgust.

The problem with writing code is that doing it well takes a long time and a lot of effort. You may make false starts down wrong design alleyways, and encounter flaws in the original plan that need piloting around, whilst putting up with huge product redefinitions en route. There is never enough time to accommodate all this, so we try to shoehorn as much as we can into the limited time available. Something has to give, and it's usually the purity of the code.

So code is shaped by design, yes, but also by the organisation that built it and its life history. A case in point: We have some particularly baroque user interface code. It works (mostly), but is pretty much unfathomable unless you devote a significant portion of your life to meditation at the temple of complex code. It's just an intense lump of intertwining logic with no discernible architecture. And it's like that for a reason.

The code was initially created as a simple one-off television UI for a single customer, and was only ever specified to be a minimal closed system. It used the simplest communication protocol possible with the main system, and was as lightweight as it could be. However, it was then sold as a feature to a second customer, who wanted it to look different. A skin feature was hacked on. Then it was sold to another customer in a different country. Internationalisation was hacked on. Then it was sold to another customer, who wanted some new UI facilities, so these were hacked in. This story continued. For a long time. Today this simple UI component is unrecognisable from its former self. It's pretty much unmaintainable; each addition has been a quick hack since the whole thing has always needed rewriting.

If the initial design had incorporated all these features the code would still be lean and logical. However, it would have been too much work upfront and the company probably would never have started the project. Pity the poor programmers who work with this.

Perhaps the reason we see so much bad code and so many quick hacks is the mistaken belief that it takes longer to do the job properly. When you factor in the time spent debugging, and the ease of making more modifications later on this proves to be a wrong assumption. You may be able to close a single fault report quickly by hacking out a fix, but it's not a complete solution. As professional programmers, we need to be aware of this, and take responsibility for what we do to code.

In the corporate world, there is often a management expectation of quick fixes. It's reasonably easy to show a manager that a five tonne block of concrete stuck on top of a flimsily erected flagpole won't stay up for very long. It's harder to make them stand underneath the thing. And it's much harder to get the same message across when we're talking about software. They just don't get it. As far as most managers are concerned programmers are magicians. They practice dark mystical arts and have limitless powers. Managers just tell them what to do and when to do it by, and it will happen, however many all-nighters they need to pull.

And, being gifted and dedicated, sometimes we come through on this assumption. Doing so can actually make matters worse as management will now expect that this tactic will always work, and that it's our fault when it doesn't. Sadly, there comes a time when that kind of hacked up software just cannot be made to expand any more, when it really just wants to keel over and find its final resting place in a quiet dark corner somewhere. Management will not be happy^[4].

Code growth is easier if the culture of a company is to develop software in small iterative steps. This way evolution is almost built into the design strategy, and rewriting some code parts to accommodate change is implied. The alternative, when you have to attack a monolithic code edifice with a small pickaxe in twenty seconds flat, is nowhere near as reasonable.

Now that we've identified some of the problems of working with an evolving code base, how do we manage the mess? What strategies can we adopt to avoid some of this?

The first and most important thing is to have recognised the problem. Too many programmers hack away without thinking about what they're doing to the quality of their code. As long as they silence the users' screams in the shortest time possible they don't care what state they leave the code in. Someone else can deal with it next time.

Before we think about working with existing code, there are a few considerations for the creation of new code that will greatly aid later maintenance. Extension and malleability need to be designed in, but as we've seen, not at the expense of complexity. Modern component/object based paradigms promise greater reuse and extensibility. They do give us clear interface points between code modules. However, if the interfaces don't support later extensions then something has to budge. Think very carefully about your system interfaces as you create them. It's hard to design for change, so don't necessarily try to support kitchen-sink functionality.

Simple things like writing neat, clear code that can easily be understood and worked with, accompanied by good documentation and well-defined and clearly documented APIs are key. Consider using literate programming tools to document interfaces. This increases the chance that the documentation will be updated when the programmatic interface evolves.

Modularity and information hiding are the cornerstones of modern software engineering. Try to isolate any likely changes to a small part of the system, making your system more viscous and therefore stable under change.

Consider the interconnection of modules, and try not to make every module link to every other module. Inter-module dependencies can take several forms: making function calls, getting notifications, using header files, opening network connections, and so on. It's advisable to avoid having one central module that every other module depends on, since a single change there will affect every other module in the system.

KISS: Keep It Simple, Stupid. Don't over-complicate, don't overengineer. Optimise an algorithm only when you know that there are performance issues, not just because you think you know a good way to make it run faster. Simplicity is nearly always more desirable than performance, and it certainly makes later maintenance easier.

There is a difference between maintaining good code and maintaining bad code. With the former, you must carefully preserve the integrity of the design and ensure you don't introduce anything that is out of place with the system as a whole. With the latter you have to do your best not to make the mess any worse, and if at all possible try to improve things on your way through. If you can't get as far as a rewrite of the offending sections, a little refactoring can go a long way.

Before you even begin to touch any code, a couple of organisational issues should be considered:

As well as guarding against the warning signs we saw in a previous section, here are some practical suggestions to help you when working with existing code:

As professional programmers, we should naturally shy away from the pressure to do a quick bodge job. Sadly we don't work in ivory towers, and sometimes compromise is required; it's not always a commercial reality to complete a task in the theologically 'correct' way. It's unprofessional to flatly refuse to extend code in a distasteful manner. My experience with the TV UI is a good example of why. This explains why so much code is brittle, flaky and dangerous. But it also explains why there's any code out there at all. If there wasn't the commercial drive to get it shipped, programmers would spend forever tweaking code to get it just right, writing and rewriting, by which time the company would have totally collapsed around them.

I'm not sure whether I agree with Aristotle. Change can be a right pain in the rear end. We should manage code changes carefully. That way we can evolve our good programs into something greater, rather than degrade them into an unstable mess.

Perhaps not understanding how to maintain software well and expand it correctly is one of the reasons software development is not yet a true engineering discipline like mechanical or structural engineering.