Title: Professionalism in Programming #22

Nobody's perfect. Well, except for me that is. All day I have to sit down and work through tedious problems in other people's code. The test department discovers that our software falls over when they do such-and-such. So I trawl through the system to find what Programmer Fred did wrong three years ago, patch it up and send it back to test for them to break again.

Of course, you wouldn't find me making those sorts of elementary mistakes, not a chance. My code is watertight. Faultless. Low fat and cholesterol free. I don't write a line until I've gone over everything in my head, I don't complete a code statement without considering all the special cases that might occur, and I type so carefully that I've never once misplaced = for == in an if statement.

Totally fault free, me. Really.

Well, perhaps not quite.

I don't think anyone sits trainee programmers down and explains the facts of life to them. It's like this, son. There are the birds and the bees. Oh, and the bugs. Bugs are the inevitable dark side of constructing software, a simple fact of life. Sad, but true. Whole departments, and even industries, exist to manage them.

Everyone reading this will be only too aware of the proliferation of faults that exist in released software. How do bugs appear with such frightening regularity and in such great magnitude? It's all down to human nature. Programs are written by humans. Humans make mistakes. They make mistakes for a number of reasons (or excuses). They make mistakes because they don't understand the system they're working on well enough, because they don't correctly understand what they are implementing, but more often than not because they just don't pay enough attention to what they're doing. Most bugs are due to mindlessness. I once saw a wonderfully simple illustration of this, play along at home:

The yolk, right? Think about it. If you didn't fall for that one, then you were probably only paying attention because I'd just warned you. Hey, give yourself a brownie point anyway. But tell me who warns you every time that you're about to write a potentially flawed line of code? They'd deserve a lifetime supply of brownie points.

So as programmers we're all to blame for the bad state of software. We're all guilty. Do we learn to live with the guilt, or do we do something about it? There are two types of response. The first school is the it's not a fault, it's a feature school. A fault turns up and we respond in the words of the great philosopher Bart Simpson: I didn't do it. Nobody saw me do it. You can't prove anything [Simpsons]. We blame compiler quirks, OS flaws, random climate changes, or computers with a mind of their own. Or as I alluded to in the opening paragraphs, we blame other people. A Teflon raincoat can be a very handy programming tool.

However, we should really subscribe to the second school, the school that concedes that software errors are not entirely inevitable. Many of these kinds of mindless mistake can be picked up or even prevented, and as responsible programmers we should be taking steps to do so. In this article we'll find out about of this, and look at some good debugging techniques to employ when bugs do slip through the net.

Contrary to popular belief the term bug was in use before the advent of computers. In the 1870s Thomas Edison talked about bugs in electrical circuits. The story of the Harvard University Mark II Aiken Relay Calculator tells of the first recorded computer bug. In 1947, the early days of computers when they took up whole rooms, a moth flew in and managed to lodge itself in some circuits, causing a system failure. They taped it into the logbook and wrote below: First actual case of bug being found. For posterity's sake it has been preserved in the Smithsonian Institute.

Bugs are bad news. But what are they really? It's worth identifying the different varieties of bug we encounter, understand how they are born, survive and can be exterminated. It's also important to know what to call them; see the sidebar for more on this matter.

Software bugs fall into a few broad categories, and understanding these will help us to reason about them. Some bugs are naturally harder to find than others, and this usually turns out to be related to their category. Stepping right back and squinting at them from a distance, we see these three classes emerge:

A failure may manifest itself because of a defective single line of code, or may only show up when several interconnecting modules are finally glued together, their assumptions not quite matching up.

Moving in a bit, and looking more closely at runtime errors, a few more groupings of fault become clear. Here they are ranked in order of pain, from splinter to decapitation.

The best kind of runtime failures are the reliable ones. If they're reproducible, they are much easier to write tests for, and track down the cause of. The failures that don't always occur tend to be memory corruptions.

Now that we have things in neat little boxes, let's zoom right in and take a look at some of the specific types of runtime failure. These are some common semantic faults that we come across.

Different OSes, languages, and environments report these errors in different ways, with different wording. Some languages try to avoid types of error by not providing features you can shoot yourself in the foot with. Java, for example, has no pointers and checks every memory access you make automatically.

Like a hypochondriac, our code is constantly complaining about being ill. More often than not it genuinely is in need of some attention. We're the doctors. If our code is sick then we've got to perform the diagnosis, the surgery, and nurture it through its convalescence.

Weeding out bugs is hard. Not only do humans make mistakes when writing, they also make mistakes when reading. When I proof read these articles I have a tendency to read what I meant to write and not what I really wrote; it works the same for software. When we look at our faulty code we'll tend to see what we intended, not how the compiler actually interprets our instructions. In this respect the compiler is really quite pedantic, it can only produce exactly what we asked, not what we were hoping for.

Some programmers introduce far fewer faults into their code than their peers (as much as 60% less), can find and fix faults quicker (in as little as 35% of the time), and introduce fewer faults as they do so (figures from [Gould]). How do they do it? They are naturally able to pay more attention to the task, and can focus on the microscopic level of the code they're writing whilst keeping the broader picture in mind.

The professional programmer is always mindful of introducing faults, and will try to fix a detected problem sooner rather than later. Certainly, it's wrong to presume that we only check for problems when the software has been written. I've known many programmers who believe that the test department exists to detect their bugs for them. This is just plain wrong.

There is a clear difference between testing and debugging. Testing identifies the presence of a fault, e.g. the program output is incorrect, whereas debugging is the process of reproducing, locating, understanding, and fixing a fault.

Testing is QA, that is quality assurance; debugging is repairing a problem. You don't get quality by fixing bugs, you can't add it in at the end of software development, you must plan the quality into the architecture and implementation. Testing won't prove the absence of faults, it won't catch all errors. It's impossible to draft exhaustive test cases; software is just too complex. We will inevitably release software into the field containing faults that may still crop up. Yes, the quality of our software is in part down to the quality of our testing department, but also to our personal testing, and the quality of the fixes that we implement.

There is an art to debugging, and it's very much something to be learnt. It's a skill. Experience shows you how to become an effective debugger. And this is something that we will all get plenty of experience at. Now, different people's brains work in very different ways, and they have different ways of problem solving. What works for one programmer may not for another. However, there are some general principles that always apply.

The sidebar (next page) offers a whistle-stop tour of the tools available to aid our bug hunting. How we use these tools and where and when they are applicable will differ from situation to situation. However, one of the most potent weapons in our debugging arsenal is a distrust of anyone's code mixed with a healthy dose of cynicism. The cause of your errant behaviour could be absolutely anything, and in the act of diagnosis we should start by eliminating even the most unlikely of candidates.

How difficult it is to find a fault depends on how well you know the code it's lurking in. It's hard to jump into some random source and make any kind of judgement about it without knowing the structure and how it's intended to work. For this reason, if you have to debug some new code take time to learn it first, it really will pay off in the long run.

The ease of debugging is also dependent on the control you have over the execution environment, how much you can play around with the running program and inspect its state. In an embedded environment debugging can be much harder because the tool support is sparser. You're also probably running in an environment that is providing a lot less insulation from your own stupidity; little mistakes can have much bigger consequences.

There are two distinct facets to debugging: finding the fault and fixing the fault. The following sections describe a sensible approach to both.

The golden rule when debugging is this: Use Your Brain. Think. Consider what you're doing. Don't flail around thoughtlessly hacking at bits of code until something begins to look like it might be working. Now, sometimes a quick fiddle about will get you results, sometimes some hacky little exploratory tests will pinpoint the problem quickly. So is it a justifiable thing to do? Perhaps, but if you make the conscious decision to do some quick-and-dirty stabbing around, set yourself a hard time limit to do it in. It's all too easy to spend an entire morning with the 'just one more little go' approach. After the time limit is up, follow the more methodical approach laid out below.

If your quick stab turns up trumps and you do find the fault, reengage your thinking gear. Look at the How to fix faults section below, make the change carefully and thoughtfully. Just because the fault was easy to find, it doesn't necessarily mean that the fix is quite as obvious as it looks.

So how do we find bugs? If there was a simple three-step process we'd all have learnt it and our programs would be perfect by now. As it is, there isn't and they aren't. Let's try to distil the available bug hunting wisdom.

Compile time errors. We'll look at these first, since they are comparatively easy to deal with. When your compiler comes across something unpleasant it will not normally just complain the once, but take the opportunity to sound off about life in general, spitting out a ream of other subsequent error messages. It's been told to do this; upon encountering any error the compiler tries to pick itself back up and carry on parsing away. It's not always too good at it, but with code like yours who could blame it?

The upshot is that the later compiler messages can all be quite random and irrelevant. You should only need to look at the very first error reported, and sort out that problem. Have a glance further down the list by all means, there may be some other good things down there, but more often than not there isn't.

Even this first compiler error may be cryptic or misleading, depending on the quality of the compiler (if you're really stumped by what an error means try another compiler, perhaps). Hardcore C++ template code can produce inspired errors from some compilers. The reported fault usually is on the line that the compiler reports, but sometimes it may actually be on the preceding line - a syntax error there causes the following line to be nonsensical, and this is what thecompiler notices and moans about.

Linker errors, on the whole, are far less cryptic. The linker will tell you that it's missing a function or library and so you'd better go off and find it (or write it). Sometimes the linker may complain about arcane vtable related C++ problems, this is usually a symptom of missing a destructor's implementation or something like that.

Run time errors require a little more of a game plan. If your program contains a bug then it's likely that somewhere in the code a condition you believed to be true isn't. Finding the bug is a process of confirming what you think is correct until you find the place where the condition doesn't hold. You have to develop a model of how the code really works and compare this with how you'dintended it to. The only sensible way to do this is methodically.

Scientific method is the process scientists use to develop an accurate representation of the world. That sounds akin to what we are trying to do. There are four steps to scientific method: (i) observe a phenomenon, (ii) form a hypothesis to explain it, (iii) use thishypothesis to predict the results of further observations, and finally (iv) perform experiments to test these predictions. Now I'm not proposing that we use scientific method wholesale, for a start we're trying to get rid of the errant phenomenon rather than build a model of it. However, scientific method is a good backbone and you'll see it reflected in the steps below.

Sometimes you try all this but it just doesn't work, you're left wailing and gnashing your teeth, with a sore head from banging it against a brick wall for too long. When things get this bad I always find it helps to explain the whole problem to someone else. Somewhere in the description everything seems to slip into place and I see the one key piece of information I had been missing all along. Try it and see. Perhaps this is why pair programming is such a successful strategy.

You'll notice that this section is much smaller than the preceding one. Funny that. Usually the whole problem is finding the darned fault. Once you've worked out where it is, then the fix is obvious.

But don't let that lure you into a false sense of security. Don't stop thinking once you've diagnosed the source of your errant behaviour. It's very important not to break anything else as you make the fix - it's surprisingly easy to trample over something in the flower bed as you stroll over to pluck out a weed.

As you modify code always ask yourself 'what are the consequences of this change?' Be aware of whether the fix is isolated to a single statement, or affects other surrounding bits of code. Might the effect of your change ripple out to any code that calls this function, does it subtly alter the behaviour of the function?

Convince yourself that you have really found the cause of the problem, and not just another symptom. Then you can feel confident you've put a fix in the right place. Consider whether similar mistakes may have been made elsewhere in related modules, and go and fix them if necessary^[6].

Finally, try to learn from your mistake. We must learn or otherwise be doomed to repeat the same errors for all eternity. Is it a simple programming error you keep making, or something more fundamental, the incorrect application of an algorithm?

Anyone will tell you that prevention is better than a cure. The best way to manage bugs is to not introduce them. Sadly I don't think we'll ever completely reach this ideal, but careful programming can avoid so many problems. Good programming is about discipline and attention to detail.

This section could be enormous, but all prevention advice boils down to one simple statement: Use Your Brain. Enough said.

Like death and taxes, no matter how hard we try to avoid them, bugs happen. Sure, we should use every sort of anti-wrinkle cream available and manipulate our money in cunning ways to mitigate the effects. But if we don't know how to deal with faults when they stare us in the face then our code is doomed.

Debugging is a skill you develop. It doesn't rely on guesswork, but on methodical detection and thoughtful repair.