Title: File Format Conversion Using Templates and Type Collections

A recent project involved upgrading some files from an old format to a new one (and possibly back again), as a result of changes to the data types being stored. Several possible implementations were considered. The final solution made use of template methods, and type-collection classes, and supported forward and backward file format conversion with no code duplication and minimal overhead.

Many years ago, one of our projects was converted from a 16-bit application running on Windows 3.11 to a 32-bit one running on Win32. Most of the code was ported at the time, but some changes were not made because they would have required a change to the file formats. 16-bit identifier values were being stored in a file. Changing the file format was seen as too much of an upheaval (especially at a time when so many other changes were being made). And besides, 16-bits should be enough for anyone...

Time passed. Suddenly, 16-bits were no longer enough everyone. The file format needed to be upgraded. Discussions were had, and the following requirements emerged:

Support for the new format in the software, and for both formats in the upgrade utility, required old and new versions of the persistence code for the data types involved, as well as some form of user-interface for the upgrade utility, and logic for converting the files as a whole. Suggestions were put forward for tackling the serialisation issues:

The first suggestion set warning bells ringing left right and centre. Every time I have ever copied code around it has come back to haunt me. When the same, or similar, code is in two places you have twice as much code to manage. Changes need to be made in two places instead of one, which is highly error-prone. Furthermore, people inevitably forget about one or other of the copies, and so it gets out of date, it doesn't get built properly, documentation stagnates, and it causes endless confusion to new team members when they stumble across it. Re-use good; copy-and-paste bad!

However, criticisms were levelled at the second suggestion too. The application would need to cart around both old and new serialisation code, despite only ever using the new code. Small classes would find the majority of their source code comprising multiple persistence methods. Changes and fixes would still need to be made to both versions. Even if they sit right next to each other in the source file it is easy to miss one when editing the code through a tiny keyhole source code window^[1].

Finally, the third suggestion leads to spaghetti serialisation code, with huge conditional blocks based on ever-more complicated version dependencies. In later versions you have a mess of if blocks checking for file formats that have not been supported for years [Blundell00]. As with the previous suggestion lean classes become fat with persistence methods.

In our project we were making no changes other than the types and sizes of various data values. Instead of a version flag, why not parameterise the persistence methods on the relevant types? This way we can support a whole raft of file formats using different types all with the same code. Simple wrapper methods can then be written to forward to the parameterised method with the correct types.

As a rather trivial example, consider the code for a class that stores an array of id values (see [Blundell99]).

  // id_array.h
  class id_array {
    ...
    short m_size; // should be plenty...
    short *m_ids; // should be wide enough
    };

  // id_array.cpp
  void id_array::extract(out_file &f) const
  {
    f << m_size; // raw write, 16-bits
    for (short i = 0; i != m_size; ++i)
    f << m_ids[i];
  }

  void id_array::build(in_file &f) {
    short size;
    f >> size; // raw read of 16-bits
    resize(size);
    for (short i = 0; i != size; ++i)
    f >> m_ids[i];
  }

As you can see, there is very little change to the code. The two methods are prefixed with a template declaration containing the type required. This type is then used inside the methods. One point worth noting here is that the type must be used in any overloaded function calls rather than the data members from the class itself. Writing f << m_size; will output m_size as the type defined in the class itself, rather than the required type T. Hence you must write T size = m_size; f << size; instead. Easy to overlook, that one (he says from experience :-)^[2].

It soon becomes clear that, strictly, we should have parameterised the class both on the capacity and the contained type, because these are not necessarily the same. Thus, our class is now parameterised on two types:

  template <typename Count, typename T>
  void id_array::extractT(out_file &f) const{
    Count size = m_size;
    f << size;
    for (Count i = 0; i != m_size; ++i) {
      T value = m_ids[i];
      f << value;
    }
  }
  template <typename Count, typename T>
  void id_array::buildT(in_file &f) {
    Count size;
    f >> size;
    resize(size);
    for (Count i = 0; i != size; ++i) {
      T value;
      f >> value;
      m_ids[i] = value;
    }
  }

More complicated data structures may have even more types, and when you have many such low-level data types you can end up with a huge number of types and a huge number of different parameters to each method. It gets nasty very quickly.

What we really want is to be able to say, "My old file format used types t1, t2, …, tn, whereas in my new format I use types T1, T2, ..., Tn." It would be nice to be able to group these relevant types together so you can just say "new format" or "old format" rather than "short, unsigned short, int and short" to one method and something else to another. Enter the class as a method of naming things as a group:

  // format_types.h
  class old_types {
  public:
    typedef short count_t;
    typedef short my_id_t;
    ... // lots more follow, if nec.
  };
  class new_types {
  public:
    typedef size_t count_t;
    typedef int my_id_t;
    ... // lots more...
  };

Now, rather than passing in as many parameters as each class requires, persistence methods can be parameterised solely on a single format type. These methods then pull out whatever named types they require from the file format 'types class':

  template <typename Format>
  void id_array::extractT(out_file &f) const{
    Format::count_t size = m_size;
    f << size;
    for (Format::count_t i = 0;
         i != size; ++i) {
      Format::my_id_t value = m_ids[i];
      f << value;
    }
  }

  template <typename Format>
  void id_array::buildT(in_file &f) {
    Format::count_t size;
    f >> size;
    resize(size);
    for (Format::count_t i = 0;
         i != size; ++i) {
      Format::my_id_t value;
      f >> value;
      m_ids[i] = value;
    }
  }

We did not want to alter the interfaces of the data classes more than necessary. In particular, we wanted persistence from our main application to work exactly as before. To achieve this we created one more typedef for the types currently in use:

  // format_types.h
  // current_types points to new_types
  // now (not old_types)
  typedef new_types current_types;
  ...

and wrote forwarding functions to call the buildT() and extractT() template methods with the correct types:

  // id_array.h
  class id_array {
  public:
    // these are the original method names
    void extract(out_file &f) const; 
    void build(in_file &f);
    // these are new forwarding methods
    void extract_old(out_file &f) const;
    void extract_new(out_file &f) const;
    void build_old(in_file &f);
    void build_new(in_file &f);

  private:
    // These are the implementations
    template<typename Format>
    void extractT(out_file &f) const;
    template<typename Format>
    void buildT(in_file &f);
  };

We then implemented these forwarding methods:

  void extract(out_file &f) const {
    extractT<current_types>(s);
  }

  void build(in_file &f) const {
    buildT<current_types>(s);
  }

  void extract_old(out_file &f) const {
    extractT<old_types>(s);
  }
  ... // etc.

These are all just one-liners, making it trivial to implement and maintain.

If a new format is required in the future (64-bits, etc.) supporting it is simple:

If you want you can omit step 3 and expose public templated serialisation methods. That way, clients can use any file format they choose by calling the method with the correct types class. We did not do this, (a) to control access to the different formats more closely, and (b) because our compiler, Visual C++ 7 (the latest .NET version) requires template methods to be implemented inline, which we did not want to do. Some of our persistence methods were quite involved. Implementing them in the header files could have introduced additional compilation dependencies from extra #include directives being required.

Our workaround involved declaring a private friend helper class at the top of each data class:

  // id_array.h
  class id_array {
    class persister;
    friend class id_array::persister;
  public:
  ...
  };

Class persister then simply had two methods: the two template persistence methods moved from the main class:

  // id_array.cpp
  class id_array::persister {
  public:
    template<typename Format>
    static void extractT(const id_array &a,
                         out_file &f) {
      ... // inline because of VC++7
    }
    template<typename Format>
    static void buildT(id_array &a,
                       out_file &f) {
      ... // inline because of VC++7
    }
  };

The use of this private helper class allowed us to move the inline implementations of these template methods out of the header file. Making it a nested class avoided name clashes because we were not polluting the scope of our data classes with additional names (and therefore each class could use the same nested class name, persister). The forwarding methods within each data class could now simply forward to the static methods of class persister, passing in a reference to themselves:

  // id_array.cpp
  void id_array::extract(out_file &f) const {
    persister::extractT<current_types>(*this, f);
  }
  ... // etc.

Alas we were not quite in the clear yet. Another weakness of VC++7 is that it does not support explicit specification of template parameters for template methods/functions. We had to work around this one as well by passing in a dummy object to each method and letting the compiler sort out which function to call:

  // id_array.cpp
  class id_array::persister {
  public:
    template<typename Format>
    static void extractT(const Format &,
                         const id_array &a,
                         out_file &f) {
      ...
    }
    ...
  };
  ...

  void id_array::extract(out_file &f) const {
    id_array::persister::extractT(current_types(), *this, f);
  }
  ...

Classes were used as a scope to package up the whole set of types, used when serialising to a given file format, into a 'types class'. A typedef was provided to allow current_types always to refer to the primary types class, and hence the current file format. Template serialisation methods were used to localise a single serialisation algorithm for each class in a single place to aid implementation and maintenance. One-line (non-template) forwarding methods were used to provide an easy interface to the current, old, and new file formats. And finally the use of a private nested friend class and dummy template function parameters allowed us to work around various weaknesses in the Microsoft C++ compiler and to move our templated persistence methods out of the header files.

None of these choices were rocket science, but the end result was a seamless implementation of multi-format persistence with very little overhead, either overall (just the format classes were needed) or in each of the persisted classes.