Achieving FitNesse in C++

Sometimes a very simple idea can make a very big difference. FitNesse (and "Fit" on which it builds) are very simple ideas - and when I first encountered them my reaction was "so what". It was only after talking to people using them that I found time to investigate them more seriously. What then, do they offer?

What they offer is feedback on a system under development meeting functional requirements. They encode tests in a format that can both be executed and explained (or even edited) by a customer and can also be executed directly against the system to demonstrate the results. As a perennial problem with software development is a failure of communication, anything that helps narrow the gap between requirements and deliverables is worthy of investigation.

Fit is a Java component that takes some HTML input, interprets it into tests, exercises the tests against the system being tested, and outputs an updated version of the HTML that incorporates the results. The developer needs to write some lightweight classes that define the mapping between fields in the requirements document and the parameters of the methods to be invoked in the system. These classes are called "Fixtures" and, in the Java implementation must conform to a "fixture" convention that allows Fit to use the reflections API to set values and call methods. (Conventions like this are common in the Java world: Junit, EJBs, JavaBeans and NakedObjects are all technologies that adopt this approach.)

FitCpp is a translation of the Fit functionality into C++. Clearly, C++ lacks a direct analogue of "reflections", but by using a combination of macros and templates it allows the developer to achieve the same effect.

What both of these tools do is to scan the input HTML for tables; these are used to specify the fixture to use, together with the input data and results expected. The input is shown in Table 1.

The reporting is very simple: the results of tests that succeed turn green (light grey in illustration) and the results (both expected and actual are shown) of tests that fail turn red (dark grey). The output is shown in Table 2.

I have to admit that my initial reaction to this was "so what?" - but what I had missed is that tables of test data and results are very easy to incorporate into requirements documents and are understood by users. Of course, editing the raw HTML that lies behind the above display would put off the typical user: that is where FitNesse comes in...

FitNesse is a Java based Wiki server that includes the facility to passing the HTML on a page to Fit (or to FitCpp) and display the results. It also allows the test pages to be organised into test suites and provides summary reporting on the test results for each page in the suite. (As with the individual tests these are colour coded.)

The idea is that the functional requirements can be captured as a combination of text and tables on the Wiki site. Obviously, when these are first added all the tests will fail, but as the system is developed more and more tests will pass - and as the tests change colour they automatically provide visible feedback on progress for both developers and customers.

All of this is easier to explain and confirm with the customer than a test written in Java, C++, or a test scripting language. (And the text that underlies the Wiki is easier to follow than HTML.)

If you've read this far then you'll want to know what you need to do to set this all up. There are basically four things to do:

Getting FitNesse is pretty trivial (just go to the website given in the references below and follow the links). There are even helpful .bat and .sh scripts for starting it - basically it runs straight out of the archive.

Getting fitcpp isn't quite as simple - the published code I downloaded had a few bugs and is reliant on a number of Microsoft C++ "features" that allow non-standard code to compile. I've updated it with some fixes and to get it to compile with gcc - I've made this updated version available on my website (and also passed the changes back to the author).

Capturing the requirements as a Wiki page is pretty easy. The above example looks like:

!define COMMAND_PATTERN {
../c++/fitcpp/overload/FitOverload.exe }

The system must be able to calculate the
number of vowels and/or consonants in a
sentence.

|!-sentence_analyser-!|
|sentence|vowels()|consonants()|
|Hello world!|3|7|
|This sentence has four vowels and thirty three consonants.|4|33|
|"0123456789" Does it handle numbers 0123456789?|7|12|
|*(@$£)! &^%$£?@# <>/|0|0|

The first line overrides the default Java "fit" handler for the page and directs FitNesse to invoke an alternative hander - we'll describe creating and using this file below. The paragraph that follows is transcribed without change, and the table is marked out using the vertical bars.

The ".exe" file mentioned in the preceding section needs to come from somewhere: obviously it contains any fixtures used for the tests and some code from the FitCpp library. (Clearly, in any real world usage it will also link against the application code.)

Writing a fixture takes a little more explaining, because one needs to understand the role of a fixture and the conventions involved. While there are many types of fixture possible (they simply provide an interface for processing tables) the most useful in writing tests is the "column" fixture used in the illustration above (I didn't choose the name). This fixture is defined as follows:

#include "ColumnFixture.h"
#include "MainFixtureMaker.h"

class sentence_analyser
         : public ColumnFixture {
public:
  sentence_analyser();
  string sentence;
  int vowels();
  int consonants();
};

The name of the fixture corresponds to the first line of the table - you would typically have multiple fixtures and multiple requirements tables for an application. Looking at the body of the fixture you will see that it exposes public member data^[1] and parameter-less member functions that return values. These correspond to the columns of the table. Actually, in C++ (unlike Java) you don't have to make the names correspond to the columns but it is easier to go with this convention.

In Java the reflections API is used to detect these corresponding data and methods, but in C++ we have to write a few more lines:

sentence_analyser::sentence_analyser() {
  PUBLISH(sentence_analyser,
          string, sentence);
  PUBLISH(sentence_analyser,
          int, vowels);
  PUBLISH(sentence_analyser,
          int, consonants);
}

These macros register the corresponding members with the fitcpp processing engine.

In a real test harness the vowel() and consonant() member functions would invoke functionality in the application being tested. However, for the purposes of compiling and running the above demonstration the vowel() method is implemented as follows:

int sentence_analyser::vowels() {
  return std::count_if(
    sentence.begin(),
    sentence.end(),
    is_vowel());
}

Where is_vowel is a functor with the obvious functionality.

In its original distribution FitCpp required some additional "boiler plate" code that provides a "Maker" class to build the Fixtures that are identified by the tables and a driver "main" method that pushes the HTML through the FitCpp Parse engine. I have factored all this out into a MainFixtureMaker class and a generic main method and placed these into the FitCpp core library. The result of this is that only a single line is needed to set up the creation of a Fixture:

REGISTER_FIXTURE_MAKER(sentence_analyser)

This is based upon the assumption of statically linking the Fixtures with the library and that statically linked objects of global scope are initialised before entering main(). (This behaviour isn't strictly required by the ISO standard but, as far as I can tell, it is common across all C++ implementations.)

If we compile and link this code we will have an executable that can be invoked from the FitNesse test page described above. What happens when we invoke "test" on this page? And how do we combine tests together?

Pressing "test" causes FitNesse to invoke the program specified in the COMMAND_PATTERN directive. (This should be the program containing the fixtures). The program then reads the HTML supplied by Fitnesse from standard input and writes to standard output.

The FitCpp framework scans the input looking for HTML tables. When it encounters one it examines the first cell and constructs a corresponding fixture (the contents of the first cell specifies the type of the fixture). The next row of the table is used to match up the column names with the members of the corresponding Fixture "published" during construction. For column fixtures (like the example we've been following) the remaining rows are processed one cell at a time setting data or calling functions as appropriate. (The example shows how this allows the sentence parameter required by count_vowels() to be supplied prior to invoking it). The results of any function calls are compared with the values embedded in the HTML to select the appropriate output (green if it matches, red if not).

There can be (and usually are) several tables within a functional requirements page (which is a single invocation of the program) the plugins corresponding to these tables will be invoked sequentially as part of the same process. Neither FitNesse nor FitCpp provides a mechanism for passing information between these tests but, as program state is maintained, the various fixtures can communicate using global data/objects.

There are several issues of scale that need to be addressed: the principal ones are managing a collection of functional requirements, and allowing developers to apply the tests to their local version of the system (instead of to the integration build on a server somewhere). FitNesse addresses both of these concerns quite neatly by having pages with special properties.

The web pages with FitNesse are organised into a hierarchy like a file system (with the same convention seen in Java package names - using a "." character as a path separator). For example, a page might be called "SystemRequirements" and beneath this would come other pages such as "SystemRequirements.ChapterOne" etc. By setting the "suite" or "virtual" property on a page then the processing of the sub-pages can be tailored.

If the top-level page is given the "suite" property then it acquires a "suite" option that can be used to run all the tests in pages that lie beneath it. This results in a summary of the results of these tests with links to the results of individual tests so that the user can "drill down" if desired. Setting up a "suite" page that covers all the functional requirements allows progress to be demonstrated in an immediate and visible manner (as more functionality is delivered more of the tests show "green").

The "virtual" attribute is used in a local instance of the FitNesse Wiki server running on a developer's machine. This causes FitNesse to fetch the content of all the subpages from another URL (the remote Wiki server - in the configuration I have set up this is the integration "build" machine). The local Wiki server will run tests (and suites) locally against the developer's version of the system but the content of the functional tests is maintained on the central server. To avoid confusion the pages retrieved from the remote server are given a blue background before being shown by the local server.

As with any test framework it is normally appropriate to consider beginning the test with a "StartUp" fixture and ending with a "TearDown" fixture. (There is more on patterns of test design in the FitNesse documentation.)

In the discussion above I've only discussed Column Fixtures, there is a second type of Fixture supported by FitNesse/FitCpp - RowFixtures. These are used for describing collections of rows that may be returned by the application (think "result set") and allow missing and/or extra rows to be reported. The way they are implemented is directly analogous to the ColumnFixtures discussed above, and their use is adequately covered in the FitNesse documentation.