Title: Editorial

It's now three years since I joined the Overload editorial panel. In those three years I've spent my time working on articles - getting people to write them, shepherding authors through the process of writing them. Recently, I've turned my attention more to writing, but this is my first attempt at writing the editorial. So, what to write about? I turned to accu-general for inspiration.

Despite not posting very much on this mailing list, I do lurk on it a lot - one of the things I like about it is the culture that allows one topic of discussion to lead to another. From time to time something is said that prompts an original short post to lead to a large thread covering a much broader area of subject matter, or even ending up on a totally different topic to the one that started it.

The thread entitled "A little story..." caught my attention. As I write this, my email client contains 122 posts on this thread! It started with a short tale about a chance encounter with a programmer who had given up reading books because he believed he knew how to do his job, and had no need for furthering his knowledge. It led to (among other things), a discussion of the different interests of "techies" and managers. During this thread, the "shiny new hammer" expression - that refers to the desire to use the latest feature or technique learned, regardless of whether or not it is appropriate - came up, as it has in previous threads. As was the case in the "A little story..." thread, often mentioned within close proximity of the "shiny new hammer" expression, is C++ template meta-programming. Fair enough, I don't recall the latter being mentioned directly as an example of the former on the "A little story..." thread, but it certainly has in accu-general threads in the past.

C++ template meta-programming - which I'm going to call TMP for short - is indeed greeted with enthusiasm by some, but there are those who regard it as something not suitable for practical use. A couple of conferences ago, having admitted to using TMP in production code, I was involved in a debate with someone who argued the case of it being unsuitable for production use on the grounds that most developers would not understand the technique, and therefore it would prejudice the chances of the software being maintainable in the long term. My counter-argument was that the TMP approach was actually the best chance for maintainability. I chose to use a TMP approach believing it to offer the best of the available sets of tradeoffs - a decision I stand by to this day!

Now, this editorial is not about C++ TMP per se - that is the subject of technical articles within Overload, not a subject for the editorial. However, I think it is reasonable to assert that it is an example of an advanced approach - and here, advanced is the critical word. It is the use and/or abuse of advanced approaches and techniques in general that is often the substance of (sometimes heated) debate, and here I want to use TMP as a topical example of an advanced approach.

What I want to talk about centres around the interaction between the commercial and managerial forces acting on development, and the use of advanced techniques. I hope to be able to argue, using TMP as an example, that the use of advanced techniques is not in tension with the commercial interests, and therefore is not in tension with the interests of managers.

Solutions to some problems have naturally occurring complexity. That is, a certain amount of complexity is inherent in any solution anyone might come up with - and it is the failure to recognise this that perpetuates its own set of problems and leads to trouble in software development in general. Complexity exists and it can't be ignored or got rid of, so the only choice is to live with it and do what we can to tame it. How can we go about doing this when designing software? Three options spring immediately to mind: ignore the complexity, produce lots of code to handle it, or use advanced techniques (associated with advanced features of the implementation language) to tame it. The first of the three options has, unfortunately, a large following. I think it can be safely put in the rubbish bin without further discussion. The second option (write lots of code) is quite common, and may be necessary, depending on the nature of the complexity. In passing, this reminds me of an accugeneral thread from a few years ago (I think it was 1999, but that's without checking). It was posted by someone telling the tale of how his management had banned the use of C++ templates - management believed that if developers weren't allowed to use advanced language features, they wouldn't write complicated code. Management's ears remained closed to any discussion despite the developers being quite happy with templates. The code base became more and more complex because of the bloat caused by having to repeat code.

So far I have talked about complexity without defining what I actually mean. This is important - is a piece of software complex because it uses complicated algorithms, because it contains lots of classes, because it requires the integration of components developed independently by different parties, for some other reason, or because of some combination of these? However, there is a type of complexity that arises where there is more than one class having between them commonality and variability that are well defined. The variability can involve pretty much any part of a class's implementation - constants, types, data structures, operations and algorithms (actually, the variability can even extend to the interfaces, but I'm in serious danger of digressing).

Having got that part straight, let's get back to approaches to addressing complexity. Writing lots of code may be necessary for some types of complexity, but for the one I've just described it leads to repetition of code. Optically the code may look different because its implementation contains different types and different operations, but on closer scrutiny it can be seen that the structure is the same. Therefore there is a large area of commonality that requires maintenance and testing - or putting it another way, the same thing is being maintained and tested more than once!

By contrast, if the same code had been generated from a metaprogram, the meta-code would express a clear separation of concerns between the commonality and variability. Therefore, the two could be maintained independently, and at least some of the testing done separately. This sounds good, doesn't it, but is there a downside? Well, there may be, but it's a matter of perspective. The downside that may be perceived is that TMP uses templates, regarded by some as an advanced language feature, and one to be used by experts rather than ordinary programmers. Further, TMP uses advanced techniques on top of an advanced language feature, and is therefore highly complicated.

So far I have focused on the technical topics of discussion, but now it's time to put our project management hats on, and I think this is a good point at which to insert a short digression - I want to briefly make a general comment on something else that came up in the "A little story..." thread, namely the interests of project managers. Some people reading this will know that my interests lie mainly in designing and implementing software - i.e. I'm a developer (and I have in the past been called a "techie", but "techie" is a characterisation I'm not at all keen on). However, on a recent project, I found my role somewhat extended by force. I think the following description using metaphors is fair: if project management can be likened to a swimming pool, then I had to dip my foot in the shallow end. Now, previously I thought I was aware of project management issues and considerations through having read about it and talked/listened to other people. I was in for a surprise - the first hard lesson I learned was that project management looks very different when you're actually doing the job! Having done just a small amount of project management was a serious education, and has given me a broader perspective on software development. OK, digression over, let's get back on topic...

Project managers have to balance several forces, but two of the principal ones are the minimisation of risk and keeping costs under control. Note that the latter is not a case of actually minimising cost - that's different, but the point is, when the costs are added up, the total must be within budget. Risks however, must always be minimised. Risk minimisation and cost control are not independent forces - there is interplay between them. Now before bringing project management into the discussion, I was talking about handling a certain type of complexity either by writing and replicating lots of code, or by using C++ language features to absorb it by using a TMP approach. It's time for these two separate topics to meet up.

If the approach of writing - and replicating - lots of code has been used, then each piece of (what is effectively the same) code must be maintained. This increases the risk, because there is a risk that code will get missed in maintenance, and this risk must be addressed. One way to address it is ensure that maintenance work is properly reviewed, but this takes people's time and pushes up the cost. Further, each piece of code must be tested, and testing more than one piece of code costs more than testing just one. This is an ongoing cycle for as long as the software is being maintained. If instead the TMP approach is used, only one piece of code needs to be tested and maintained. It may seem that other risks creep in, namely that there is a burden placed on the knowledge of the developers that write and maintain the code. Whether or not this is really a problem is debatable, but one thing is certainly true - lack of good developers always increases the risk dramatically! The TMP approach actually has a big plus point that is far from obvious, but reduces both risk and cost - the commonality and variability is documented without a word ever being written!

Software developers have a responsibility to apply the use of advanced techniques for the benefit of projects, and not for the sake of using them - if we expect managers to take us seriously, we must conduct ourselves professionally, and wielding a "shiny new hammer" has no place in professional software development. Managers have a responsibility to make sure that the software developers working on the projects they manage are competent professionals, and to understand that in the hands of competent professionals, advanced techniques are of benefit to their projects. Unfortunately, while the industry has few software developers who can be described as competent professionals, it is sad that it also has few managers who fit that description.