Title: Professionalism in Programming #32

This article series focused on software optimisation, explained the reasons not to optimise and also the reasons you should. We looked at general classes of optimisation, and at some quite specific code-level optimisations.

This article looked at the thorny topic of testing the software that we write. We looked at when, how and why we test software.

There are many perfectly valid reasons for not writing 'optimised' code on the first attempt:

An obvious mistake is to think that the taken to run something is proportional to the amount of effort spent writing it^[2]. You might have written some file parsing code in hours; it will always takes ages to execute, because disks are slow. The complex code you spent half a week getting right may only consume a few hundred processor cycles. The efficiency of a piece of code, or the amount of time you need to spend optimising it, bears no relation to the amount of time you spent writing it.

The performance requirements should not be arbitrarily chosen. They should be justified; not just a time limit pulled out of thin air. Every performance requirement is important - there are no specifications that don't matter. How much concern a particular requirement generates depends on how hard it is to meet. Whether it's hard or not, you still have to come up with a design that satisfies it.

Only the most dramatic performance improvements can be detected without a profiler, or some other good timing tests. Human perception is easily fooled - when you've just slaved to speed up the program, it will always appear faster to you.

Test performance improvements carefully, and discard those that are not worthwhile. It's better to have clear code than a miniscule speed increase and unmaintainable logic.

Without a clear specification, no one can really complain that your program isn't fast enough!

There are a large number of tests you should run, even though there are very few 'invalid' input combinations. Thinking of invalid inputs first: test for zero. It may or may not be an invalid value (we've seen no spec, so we can't tell), but you'd expect the code to cope reasonably with it.

Next, write tests considering combinations of 'usual' inputs (say of 1, 10, and 100 in all orders). Then try numbers with no common multiple like 733 and 449. Test for some very large numbers, and also for some negative numbers.

How do you write these test cases? Use a good test framework, or put them all in a single test function. For each test, don't programmatically calculate what the correct output value should be^[3], just check against a known constant value. Keep your test code as simple as possible:

assert(greatest_common_divisor(10,  100) == 10);
assert(greatest_common_divisor(100, 10)  == 10);
assert(greatest_common_divisor(733, 449) == 0);
... more tests ... 

There are a surprisingly large number of tests for this simple function. You could argue that for such a small piece of code it's easier to inspect, review, and prove correctness rather than laboriously create a set of tests. This seems like a valid argument. But what if later on someone makes modifications? Without the tests you'd have to carefully re-inspect and re-validate the code, an easy task to overlook.

Did you find the mistake in greatest_common_divisor? There's a clue coming up. If you don't want the puzzle spoiled then look away now. Try feeding it a negative argument. This is a more robust (and more efficient) version written in C++:

int greatest_common_divisor(int a, int b)
{
  a = std::abs(a);
  b = std::abs(b);
  for (int div = std::min(a,b); div  0; -div)
    {
      if ((a % div == 0)  (b % div == 0))
          return div;
  }
  return 0;
}

If you think about this for long enough it will give you a headache. You can't keep testing test code how can you be sure the test code for your test code's test code is correct? The trick is to keep tests as simple as possible. This way, the most likely testing errors will be missing important test cases, not problems with the actual lines of test code.

Testers are more concerned with the black box style of testing, and usually only perform product testing. It's rare to have testers working at the code level, because most products are executable software; there are comparatively few code library vendors.Programmers are more concerned with white-box tests, making sure their masterful creations work as they planned.

The secret aim of any programmer writing tests is to prove that their code works, not to find cases where it doesn't! I can easily write a load of tests to show how perfect my code is by deliberately avoiding all the bits I know are dodgy. This is a good argument for getting someone other than the original programmer to create test harnesses.

Don't feel obliged to a write test harness for every scrap of code. Use your head. Inspections are fine for small functions. But you must be sure that you are performing the adequate and appropriate testing for which you are responsible, not just skipping an unpleasant task.

Ask yourself this: how many errors have bitten you recently which could have been prevented with a good set of tests? Make sure you do something about it.

There are several possible reactions:

Some of those are so plainly wrong that I'll assume you can rise above them.

Does it seem a little daft to suggest you might be happy to find a fault? Surely that's the reasonable reaction for a quality-conscious engineer, as it's far better to find faults during development than for a user to find them in the field.

Your level of excitement will depend on where in the development lifecycle the fault is found. Discovering a show-stopping bug the day before release won't make anyone smile.