A framework for object serialization in C++

The first thing we will consider is how to save the Widget class hierarchy. After all, we cannot load anything without having first saved it.

The framework implementation will be easier to understand if we first see how to use it. The user, say an application called WidgetEditor, will create a WidgetSerializer, and use it to save a Widget to file in some function saveWidget thus:

static const std::string MY_NAME 
                      = "WidgetEditor";
void saveWidget(const std::string &filename, Widget *widget){
  WidgetSerializer serializer(MY_NAME);
  std::ofstream out(filename.c_str());
  if (!out) ... // error
  serializer.save(out, widget);
}

In this function, and until the very end of the document I am going to conveniently ignore error handling. At this point you are free to imagine how the system will go bang.

WidgetSerializer is pretty easy to implement. Its responsibility is to save the file prologue and epilogue, and serialize the appropriate object in between. What exactly that means is best shown by implementation:

static const std::string FILE_IDENTIFIER_STRING = "WidgetFile";
WidgetSerializer::WidgetSerialiser(
            const std::string &appname)
    : header(appname){ }
WidgetSerializer::save(std::ostream &out,
             Widget *widget){
  out<< FILE_INDENTITIFIER_STRING << endl
    << "Header\n";   // (*) see below
  header.save(out, 0);   
// the 0 is explained below
  out << "Widget\n";   // (*) see below
  widget->save(out, 0);  
// the 0 is explained below
// Perhaps save some application choices
// in an epilogue here
}

So all the WidgetSerializer's save method does is ensure that the file identifier line and the header are saved, and then pass responsibility to the Widget class to serialize itself.

The key to using Serializable::save is shown at (*) above. The Serializable::save method causes an object to serialize itself as a file chunk on the given ostream. The information that it saves will consist of the opening brace line, data and the closing brace line. Specifically, the chunk identifier will not be saved. Why not?

It is the caller's responsibility to place an identifying tag line on the ostream before calling this method so that it can uniquely identify the file chunk when it has to read it back. For example, if we are saving two objects of the same class and want then to have different chunk names, then we have to write the identifier ourselves.

The save methodof the base Serializable class is defined to save an empty chunk. This is a useful default if you have not yet decided what information to save, or if that particular object has no saveable state at this point, but may have in the future.

void Serializable::save(std::ostream &out,
           int indentLevel) const {
  out << indent(indentLevel) << "{\n"
      << indent(indentLevel) << "}\n";
}

This simple method illustrates two more details. First, Serializable::save's second parameter, indentLevel defines what level of indentation the chunk is being saved at. Remember, at the top level we passed 0. Second, the curious pair of protected indent methods in the Serializable class are used to perform the indentation. They are a simple means of inserting a whitespace indent when we save data into a chunk using C++ stream manipulation. It is outside the bounds of this article to describe fully how it works, however. Bjarne Stroustrup decribes the mechanism in [Stroustrup]

The WidgetSerializer private Header class' save mechanism is implemented as follows:

// These are defined in some suitable
// way, somewhere
static const int VERSION_MAJOR = 100;
static const int VERSION_MINOR = 0;
WidgetSerializer::Header::Header(
         const std::string &originator):
            originator(originator) { }
WidgetSerializer::Header::~Header() { }
void WidgetSerializer::Header::save(
       std::ostream &out, int i) const {
  out << indent(i) << "{\n"
     << indent(i+1) << "Version-Major:" 
     << VERSION_MAJOR << \n"
     << indent(i+1) << "Version-Minor:"
     << VERSION_MINOR << \n"
     << indent(i+1) << "Originator:"
     << originator  << \n"
     << indent(i)   << "}\n";
}

This explains how we get the Header chunk into the file.

The final part of saving is to make the Widget save itself, and all the objects it contains. It overrides the Serializable::save method with its own version, just as Header has done above, and puts the relevant saving magic in it. To save sub-chunks for contained objects you simply write a chunk identifier line, and then call save(out_stream, i+1) on the sub-object.

For example, Widget::save may look like this:

Widget::save(std::ostream &out, int i) const {
  out << indent(i) << "{\n"
    << indent(i+1)<< "Title:"<<title << "\n"
    << indent(i+1)<< "Date:" <<date << "\n";
  for (int n = 0; n < no_sprockets; n++)  {
    out << "Sprocket\n";
    sproket[n]->save(out, i+1);
  }
  for (int n = 0; n < no_cogs; n++) {
    out << "Cog\n";
    cog[n]->save(out, i+1);
  }
  out << indent(i) << "}\n";
}

The data you place after a 'Tag:' can be as adventurous as you want, not just ints and strings. Of course, you have to be willing to write some code to parse it back in. Which brings us neatly on to...

To make you feel better, saving was the easy bit.

We have created a file containing the serialized version of our Widget. WidgetEditor has been shut down, at some future point it is fired back up and we want to be able to edit our widget again. We now need to write a parser to read the data and reconstruct the Widget class hierarchy. That is what all those (as yet) unexplained load methods are for, naturally.

First, we will investigate the SerializableLoadInfo structure that is passed into the load methods. It looks like this:

struct SerializableLoadInfo {
  int   major;
  int   minor;
  Widget *widget;
// Any other data that may be read from 
// the header
};

This structure will collect information from the input file's header and allow it to be passed to each Serializable object as it parses a data chunk. The information in this structure may alter how some data in the file is interpereted.

The WidgetEditor uses the loading API in some loadFile function like this:

Widget *loadFile(std::string &filename) {
  std::ifstream in(filename.c_str());
  if (!in) ... // error
  WidgetSerializer serializer(MY_NAME);
  Widget *w = serializer.load(in);
  return w;
}

The way WidgetSerializer::load is designed means that it hands back a newly allocated Widget object based upon the contents of the given file. It is WidgetEditor's responsibility to delete the Widget when it is finished.

Again, the top-level API is nice and easy to use. So how do we implement it?

WidgetSerializer::load(std::istream, *in) {
  Widget *widget = new Widget();
  SerializableLoadInfo  info;
  info.widget = widget;
// Check first line matches file identifier
  std::string id;
  std::getline(in, id);
  if (id != FILE_IDENTIFIER_STRING) ...
// error
// Now scan each chunk left in the file
  std::tring line;
  while (std::getline(in, line)) {
    if (line == "Header") {
      header.load(in, info);
    }
    else if (line == "Widget") {
      widget->load(in, info);
    }
    else if (line != "") {
      FileBlockParser parser;
      parser.parse(in, info);
    }
  }
  return widget;
}

What's that FileBlockParser thing doing there? For the moment you can happily accept that it skips an unrecognised chunk. Just how it does that and why we will see in a bit.

The parsing code ensures that the file type is correct, then reads the first chunk identifier, "Header", and directs the rest of the chunk to the parser in header's load method. So what does that look like?

void WidgetSerializer::Header::load(
          std::istream &in, 
          SerializableLoadInfo &info){
  std::string open;
  std::getline(std::ws(in), open);
  if (open != "{") ... // error
  std::string line;
  bool more = true;
  while (more &&
       std::getline(std::ws(in), line)){
    std::getline(std::ws(in), line);
    if (line.substr(0,14).compare( 
              "Version-Major:") == 0){
      std::istrstream si(line.c_str()+14);
      si >> info.major;
    }
    else if (line.substr(0,14).compare(
               "Version-Minor:") == 0){
      std::istrstream si(line.c_str()+14);
      si >> info.minor;
    }
    else if (line ==  "}") more = false;
  }
}

There are a couple of useful C++ standard library facilities being used here. First, std::ws(in): it is a stream manipulator that strips leading whitespace from an input stream. Secondly, istrstream is an 'input string stream', which allows you to treat strings as streams to be read/written to. It is a type-safe way of doing sscanf.

This is the only load method in the whole Serializable framework that ever writes to the SerializableLoadInfo structure, it gets the data from the file's Header chunk.

So far we have hand written two stream parsing loops. The chunk parsing code for each Serializable object could look very similar to this. It works fine. In each Serializable class, the load method loops around, pulling out chunk identifiers and data lines and acts on them accordingly.

It is not exactly elegant, for every Serializable class we have to write another parsing loop. That is a lot of duplicated effort, which tells us that there is something wrong with the design. Add to this the problem of manual string length counting in the compares above - it is a pretty error prone way of writing code: if you get a comparison wrong the compiler will not tell you.

The solution to this is the FileBlockParser class that we breezed past earlier. The intention of FileBlockParser is to do that loop for us, to save writing it thousands of times, and to safely recognise the tags and chunk identifiers for us.

This is what it looks like:

class FileItemParser;
// a class that interprets data lines
class FileBlockParser {
public:
  FileBlockParser();
  void add(const std::string &name,
             Serializable *block);
  void add(const std::string &name,
             FileItemParser *item);
  void add(FileItemParser *item);
  void parse(std::istream &in,
           SerializableLoadInfo &info);
private:
  void skipChunk(std::istream &i);
  std::map<std::string,
               FileItemParser*> items;
  std::map<std::string,
               Serializable*> blocks;
    FileItemParser *catchAll;
};

The FileBlockParser's parse method sends it scampering across the input stream a line at a time. It will expect to find an opening brace line first (just as our hand written load methods would do).

Then if it finds a chunk identifier line it looks in an internal map associating chunk names to Serializable objects. This is set up before calling parse using the appropriate FileBlockParser::add method. If the chunk identifier is not recognised, the FileBlockParser will skip the chunk (using its private skipChunk method).

If the FileBlockParser finds a data line then it will call the appropriate FileItemParser to interpret that line. If it does not recognise the tag name in its map then it will call a catch-all FileItemParser if you have added one. This may be useful in some chunks. If there is no catch-all object then the data line will be ignored.

Finally, when the FileBlockParser finds the closing brace it will stop parsing and return to its caller.

This class has taken responsibility for writing that ubiquitous loop for us. It saves us time and effort, and minimises the risk of bugs in our file parsing.

FileItemParser is a base class for objects that handle data lines. The base class looks like this:

class FileItemParser {
public:
  FileItemParser() {}
  virtual ~FileItemParser() = 0;
  virtual void parse(
          const std::string &data) = 0;
private:
  FileItemParser &operator=(
              const FileItemParser &);
  FileItemParser(const FileItemParser &);
}

With this in mind we can implement the FileBlockParser as follows:

FileBlockParser::FileBlockParser()
                : catchAll(0) { }

void FileBlockParser::add(
          const std::string &name,
           Serializable *block) {
  blocks[name] = block;
}
void FileBlockParser::add(
          const std::string &name,
           FileItemParser *item){
  items[name] = item;
}
void FileBlockParser::add(
            FileItemParser *item){
  catchAll = item;
}
void FileBlockParser::parse(
          std::istream &in, 
          SerializableLoadInfo &info){
  std::string line;
  std::getline(std::ws(in), line);
  if (line != "{") ... // error
  bool more = true;
  while (more && 
      std::getline(std::ws(in), line)) {
    if (line == "}") { more = false; }
    else if (line.find(":") 
      == std::string::npos) {
// 'line' is a chunk identifier
      std::map<string, 
          Serializable *>::iterator I
              = blocks.begin();
      bool done = false;
      while (!done && i !=blocks.end()){
        if (i->first == line){
          i->second->load(in, info);
          done = true;
        }
        i++;
      }
      if (!done) skipChunk(n);
    }
    else {
// 'line' is a 'tag:data' pair
      const std::string name =
         line.substr(0, line.find(":"));
      const std::string data = 
         line.substr(line.find(":")+1);
      std::map<string, 
          FileItemParser *>::iterator I
                    = items.begin();
      bool done = false;
      while (!done && i != items.end()){
        if (i->first == name) {
          i->second->parse(data);
          done = true;
        }
        i++;
      }
      if (!done && catchAll) [
        catchAll->parse(line);
      }
    }
  }
}
void FileBlockParser::skipChunk(
                std::istream &in){
  std::string open;
  std::getline(std::ws(in), open);
  if (open != "{") ... // bad format
  int depth = 1;
  std::string line;
  do {
    std::getline(std::ws(in), line);
    if (line == "{") depth++;
    else if (line == "}") depth--;
  } while (!in.eof() && depth);
}

We can easily now define the base Serializable::load method. It will ignore everything in a chunk by simply creating a FileBlockParser with no FileItemParsers. This will scan the chunk, ignore every data line and sub-chunk in it, and stop parsing after the appropriate closing brace.

This is the logical behaviour for Serializable::load since we defined Serializable::save to save an empty block. Perfectly complimentary.

Serializable::load(std::istream &in,
           SerializableLoadInfo &info){
  FileBlockParser parser;
  parser.parse(in, info);
}

The only thing that remains to be done is define a number of FileItemParsers that implement 'Tag:Data' parsing functionality. I have implemented a number of these, some for simple integer values, some for string values, others for more elaborate data lines with several fields. The following example shows the outline of their structure. It is a FileItemParser that reads a boolean value which has been written to the output as a string (either On/Off or Yes/No).

FileItemParser_OnOff is a template class. The template type you supply is the class with the data item to modify. Making it a template saves us from having to write a version of FileItemParser_OnOff tailored to each specific accessor method of each class that needs it.

As an example class that we will be accessing, here Is the Cog class:

class Cog : public Serializable {
public:
  Cog() : data(0) {}
  virtual ~Cog() {}
  bool data() { return _data; }
  void setData(bool d) { _data = d; }
// implement Serializable interface here
private:
  bool _data;
};

The member function to set a boolean value must have the signature void setXXX(bool), like setData in Cog above. FileItemParser_OnOff contains a typedef for this member function signature, fn_t.

template <class T>
class FileItemParser_OnOff 
          : public FileItemParser {
public:
  typedef void (T::*fn_t)(bool);
  FileItemParser_OnOff(T *obj, fn_t mfun)
  : obj(obj), mfun(mfun) {}
  void parse(const std::string &data) {
    (obj->*mfun)(data == "On" 
                || data == "Yes");
  }
private:
  T  *obj;
  fn_t  mfun;
};

When you create a FileItemParser_OnOff you specify the object whose value is to be set, and the member function that performs the setting.

Cog cog;
FileItemParser_OnOff<Cog>
        cog_parser(&cog, Cog::setData);

We then pass the FileItemParser_OnOff to a FileBlockParser to read the block:

FileBlockParser fbp;
fbp.add(&cog_parser);
fbp.parse(some_istream);

Now we have a complete mechanism to load the files that have been saved. Objects can be serialised to a byte stream (probably a file) and reconstructed from the same byte stream at a later time.

If an earlier version of the parser sees a file saved by a later system there will be some data that it does not recognise and just skips over.

If you wanted to inform the user that this has happened but not halt the file parsing process you can add new boolean flag to the SerializableLoadInfo structure. It is initially false, but can be set to true by the FileBlockParser if it meets unexpected data. When the load is complete the parser can return the flag to the caller to handle appropriately.

Up to this point, error handling has been conveniently ignored. You can add the error handling mechanism of your choice to the system. For it to be robust there must be some form of error handling.

I would favour raising exceptions, but have not incorporated that into the framework here for clarity, and to avoid complicating matters with exception safety concerns.

The classes to be serialized do not actually have to inherit from Serializable themselves. It is possible to create an adaptor class that acts as a serializable for a particular type of class and save its state - providing the data class' public API allows you to sufficiently capture and restore its state.

I have used both approaches with success.

It is not difficult to see how to convert this system from an ASCII text based approach to a binary file format. In doing so you would remove the need for text searches, comparisons, and whitespace stripping.

The advantages of doing so include reducing the file size, taking less time to write, and more importantly less time to parse. For some applications these may be big issues. Certainly, the complexity of the parsing operations increases greatly as the file size increases. This may be of particular importance when serializing an object over a network connection - the less data that needs to be sent the better.

The binary version is no longer human editable - this may in fact also be an advantage for some applications.