Title: The Philosophy of Extensible Software

A change in one module is just that: a change in one module of the system. The principles of abstraction and data hiding tell us that we should be able to make changes to hidden code which have no side-effects elsewhere. Often though we find there is a ripple effect.

If we drop a small stone into a pond we see ripples spread out across the pond. The water surface is perfect for propagating the effect. Software is, if anything, better than water at propagating the effect. Changing the interface of a module means we must make corresponding changes to the use of the interface elsewhere.

Speaking of the interface means more than just the class definition in some header file. This is the most clearly stated part of the interface but it is like an iceberg, there is much we can't see. The compiler can check the function signatures but can it check the comments?

Consider some code:

// SerialPort.hpp
// Read up to bufferSize characters from
// serial port
// store in buffer and return ptr to
// buffer
char* ReadSerialPort(char *buffer,
                     int bufferSize);
...

// SerialPort.cpp
// Read at least bufferSize chars from
// serial port
// store in buffer and return ptr to end
// of buffer
char* ReadSerialPort(char *buffer,
                     int bufferSize) {
  assert(buffer != 0)
  assert(bufferSize > 16)
  ...
}

The interface the compiler can check only forms part of the interface. The comments form another vital part of the interface and in this case they differ. Which set is correct? The developer must break the abstraction and look under the hood to see what is happening.

This is typically how a ripple starts, we've found something which isn't quite right, not so wrong it breaks the program but not good either.

Both sets of comments fail to tell us that the buffer supplied must be pre-allocated and at least 16 characters long. Sure, it may seem logical to allocate the buffer before calling the function but the C function time never worried too much about this.

At one level this is implementation detail, at another level it is interface, we can fix the comments, we can even change the function signature to reduce the problems with this function:

// SerialPort.hpp
// Read up to bufferSize characters from
// the buffer
// store in buffer and return number of
// bytes read
// bufferSize must be at least 16 bytes
// return value < bufferSize

int ReadSerialPort(char *buffer,
                   int bufferSize);

Now we have created a ripple effect, not only this module but several others will need recompiling. Wherever this function is used we must now change the code, our change has slipped out of our chunk. While it would be a pretty relaxed compiler that still allowed you to write:

char* buf = ReadSerialPort(buffer, 32);

A developer in search of a quick fix may be tempted to write:

char* buf = (char*)
ReadSerialPort(buffer, 32);

While static_cast will refuse this reinterpret_cast has no such qualms, and as demonstrated the old-style casts are still in the language and available.

The general rule of ripple effect is that he who made the ripple has to stop it, you find yourself running around all over the code, fixing ripples where they appear. This is all but impossible if you don't have a reliable build process - if you can't integrate your code easily then you can't tackle these issues. A good source code control system is essential in case things go wrong, or time runs out, and you need to back out your changes.

Nor do we have perfect foresight, we may grep the code for every instance of ReadSerialPort before we make our change, but we can't expect to find every case. Just searching the code may be a bigger job than actually performing the change. A logical directory structure is important here, if our code is scattered over several dozen disparate directories on different hard discs then what chance have we got of finding it?

Suppose we now find:

// pointer to general read function
typedef char*
           (*ReadPortFunc)(char*, int);

// read function
char* ReadData(ReadPortFunc reader) {
  int bufferSize = 256;
  char* buffer = new char[bufferSize];
  memset(buffer, 0, bufferSize);
  return reader(buffer, bufferSize);
}

Instead of diminishing, our ripple has grown. We can fix this, but suppose we find:

char* ReadUsbPort(char *buffer,
                  int bufferSize);
char* ReadKeyboardPort(char *buffer,
                       int bufferSize);

Do we fix these functions too? Or fudge it? The ripple is not just a simple compile time fix now, it has uncovered a bigger problem, and while we may have the code in a compilable state, we (should) feel a certain moral commitment to fixing this problem. The ripple has grown.

Ripples like this, and fear of ripples, is one of the main ways code resists change.

The ripple effect demonstrates the friction that can occur when changes are made. If these changes are within the same module then the friction is less because the changes are not visible elsewhere. The bigger the change, the more modules involved, the greater the friction. When a system is changing rapidly, dividing it up can be counter productive because there is a constant friction as changes ripple out of one module and into others.

But friction between modules comes in other forms too. Where there are several developers on a project there is always the opportunity for conflicting changes to happen. While exclusive locking through source code control can help, it is not a complete answer. At best it forces one developer to wait while another completes a change, the second developer then has the task of integrating the change with their requirements. Nonexclusive locking systems can hide this problem until the second developer checks their code in but the same problem arises.

Either way, friction is generated because two developers must co-ordinate their actions. If the developers are located in different teams, or even different countries, the friction is much greater still.

When a change introduces a new dependency into a module, say a new file must be #include'd, the initial friction may be small, a slightly increased build time. But when this changes the overall dependencies of the module, and in particular if this introduces a circular dependency the potential friction is greater still.

Observant readers may have noted the potential contradiction is talking of "ripple effect" and "friction" - after all ripples occur in frictionless water. Ripples are waves, and waves can only occur when two modules share a common boundary. Such a boundary propagates the wave - think of the way an earthquake wave carries.

Sound waves cannot travel in a vacuum, likewise software ripples cannot pass from one module to another if they are well spaced. Since our modules don't exist in isolation we can't place them in a vacuum, what we can do is try to minimise the friction at the boundary and thus minimise wave propagation by allowing each module to change without creating a wave beyond its own boundaries.

Software takes on many of the attributes of the organisation and process that creates it.

This idea is summarised as Conway's law - although the exact wording of the law differs. Jim Coplien's process pattern of the same name has the solution "Make sure the organisation is compatible with the product architecture."

Where an organisation is conservative and resists change their software will too. This may manifest itself in many ways: a business which resists change may create code which resists change, or, it may mean managers refuse to allocate time for modifications.

This can be a frustrating position for a software developer, they may know of a bug, they may know how to fix it but they may be refused permission to deal with it. Or, to fix it may require raising a bug report, having the work prioritised, authorised, scheduled, changed, tested, signed-off and released. Sure we need a process, but we must not put the process before the product. In processcentric organisations we find managers who know the price of everything but the cost of nothing.

Some organisations refuse to recognise refactoring as an exercise. "If it processes data, it can't be broken, can it?" "Reworking something means you made a mistake, right?"

Developers have refactored code since the beginnings of time, but only with the publication of Martin Fowler's book (2000) has it been a respectable activity. Unfortunately, it is still not an acceptable activity in many organisations. Failure to refactor code makes it more rigid, as we put change upon change it becomes inflexible and set in its way. Unfortunately it still processes data.

This is like not servicing a car, it continues working, there is no apparent problem, but the further you get beyond an oil change the more damage is being done to the internals. Like a car, over time software changes and without active attempts to improve the quality it invariably deteriorates.

As we develop software we learn, we learn more about the problem domain that the software addresses and we learn more about our solution domain - the tools we have used to address it. Naturally, this leads to new insights into both.

We also have time to dwell on problems and issues. We may take a week to draw up an class hierarchy, but we have the rest of our lives to rethink it and consider how we could have done it better. This can make life hard for us if we come to believe we made a mistake, or no longer agree with our original designs - or just see a better way. Maybe what once seemed a brilliant design now seems top heavy, or inefficient, or simplistic. Don't be too hard on yourself, admit you made mistakes if necessary. If we don't do this we will not move forward, it is now us who are resisting change.