Title: doctest – the Lightest C++ Unit Testing Framewor

Author: Martin Moene

Date: 03 February 2017 15:57:47 +00:00 or Fri, 03 February 2017 15:57:47 +00:00

Summary: C++ has many unit testing frameworks. Viktor Kirilov introduces doctest.

Body: 

doctest is a fully open source light and feature-rich C++98 / C++11 single-header testing framework for unit tests and TDD. A complete example with a self-registering test that compiles to an executable looks like Listing 1.

#define DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN
#include "doctest.h"

int fact(int n) {
  return n <= 1 ? n : fact(n - 1) * n; 
}

TEST_CASE("testing the factorial function") {
    CHECK(fact(0) == 1); // will fail
    CHECK(fact(1) == 1);
    CHECK(fact(2) == 2);
    CHECK(fact(10) == 3628800);
}
			

And the output from that program is in Listing 2.

[doctest] doctest version is "1.1.3"
[doctest] run with "--help" for options
=================================================
main.cpp(6)
testing the factorial function

main.cpp(7) FAILED!
  CHECK( fact(0) == 1 )
with expansion:
  CHECK( 0 == 1 )

=================================================
[doctest] test cases:  1 |  0 passed |  1 failed |
[doctest] assertions:  4 |  3 passed |  1 failed |
			

Note how a standard C++ operator for equality comparison is used – doctest has one core assertion macro (it also has macros for less than, equals, greater than...) – yet the full expression is decomposed and the left and right values are logged. This is done with expression templates and C++ trickery. Also the test case is automatically registered – you don’t need to manually insert it to a list.

Doctest is modeled after Catch [Catch], which is currently the most popular alternative for testing in C++ (along with googletest [GoogleTest]) – check out the differences in the FAQ [Doctest-1]. Currently a few things which Catch has are missing but doctest aims to eventually become a superset of Catch.

Motivation behind the framework: how it is different

doctest is inspired by the unittest {} functionality of the D programming language and Python’s docstrings – tests can be considered a form of documentation and should be able to reside near the production code which they test (for example in the same source file a class is implemented).

A few reasons you might want to do that:

• Testing internals that are not exposed through the public API and headers of a module becomes easier.
• Lower barrier for writing tests. You don’t have to:
  • make a separate source file
  • include a bunch of stuff in it
  • add it to the build system
  • add it to source control

You can just write the tests for a class or a piece of functionality at the bottom of its source file – or even header file!

• Faster iteration times – TDD becomes a lot easier.
• Tests in the production code stay in sync and can be thought of as active documentation or up-to-date comments, showing how an API is used.

The framework can still be used like any other even if the idea of writing tests in the production code doesn’t appeal to you – but this is the biggest power of the framework, and nothing else comes close to being so practical in achieving this.

This isn’t possible (or at least practical) with any other testing framework for C++: Catch [Catch], Boost.Test [Boost], UnitTest++ [UnitTest], cpputest [CppUTest], googletest [GoogleTest] and many others [Wikipedia]. Further details are provided below.

There are many other features [Doctest-2] and a lot more are planned in the roadmap [Doctest-3].

What makes doctest different is that it is ultra light on compile times (by orders of magnitude – further details are in the ‘Compile time benchmarks’ section) and is unobtrusive.

The key differences between it and the others are:

• Ultra light – below 10ms of compile time overhead for including the header in a source file (compared to ~430ms for Catch); see the ‘Compile time benchmarks’ section
• The fastest possible assertion macros – 50 000 asserts can compile for under 30 seconds (even under 10 sec)
• Offers a way to remove everything testing-related from the binary with the DOCTEST_CONFIG_DISABLE identifier
• Doesn’t pollute the global namespace (everything is in the doctest namespace) and doesn’t drag any headers with it
• Doesn’t produce any warnings even on the most aggressive warning levels for MSVC / GCC / Clang
  • -Weverything for Clang
  • /W4 for MSVC
  • -Wall -Wextra -pedantic and over 35 other flags not included in these!
• Very portable and well tested C++98 – per commit tested on CI with over 220 different builds with different compilers and configurations (gcc 4.4-6.1/clang 3.4-3.9/MSVC 2008-2015, debug/release, x86/x64, linux/windows/osx, valgrind, sanitizers...)
• Just one header and no external dependencies apart from the C / C++ standard library (which are used only in the test runner)

So if doctest is included in 1000 source files (globally in a big project) the overall build slowdown will be only ~10 seconds. If Catch is used – this would mean over 350 seconds just for including the header everywhere.

If you have 50 000 asserts spread across your project (which is quite a lot) you should expect to see roughly 60–100 seconds of increased build time if using the normal expression-decomposing asserts or 10–40 seconds if you have used the fast form [Doctest-5] of the asserts.

These numbers pale in comparison to the build times of a 1000 source file project. Further details are in the ‘Compile time benchmarks’ section.

You also won’t see any warnings or unnecessarily imported symbols from doctest, nor will you see a valgrind or a sanitizer error caused by the framework. It is truly transparent.

The main() entry point

As we saw in the example above, a main() entry point for the program can be provided by the framework. If, however, you are writing the tests in your production code you probably already have a main() function. Listing 3 shows how doctest is used from a user main().

#define DOCTEST_CONFIG_IMPLEMENT
#include "doctest.h"

int main(int argc, char** argv) {
  doctest::Context ctx;
  // default - stop after 5 failed asserts
  ctx.setOption("abort-after", 5);
  // apply command line - argc / argv
  ctx.applyCommandLine(argc, argv);
  // override - don't break in the debugger
  ctx.setOption("no-breaks", true); 
  // run test cases unless with --no-run
  int res = ctx.run();
  // query flags (and --exit) rely on this
  if(ctx.shouldExit())
    // propagate the result of the tests
    return res;
  // your code goes here
  return res; // + your_program_res
}
			

With this setup the following 3 scenarios are possible:

• running only the tests (with the --exit option)
• running only the user code (with the --no-run option)
• running both the tests and the user code

This must be possible if you are going to write the tests directly in the production code.

Also this example shows how defaults and overrides can be set for command line options.

Note that the DOCTEST_CONFIG_IMPLEMENT or DOCTEST_CONFIG_IMPLEMENT_WITH_MAIN identifiers should be defined before including the framework header – but only in one source file – where the test runner will get implemented. Everywhere else just include the header and write some tests. This is a common practice for single-header libraries that need a part of them to be compiled in one source file (in this case the test runner).

Removing everything testing-related from the binary

You might want to remove the tests from your production code when building the release build that will be shipped to customers. The way this is done using doctest is by defining the DOCTEST_CONFIG_DISABLE preprocessor identifier in your whole project.

The effect that identifier has on the TEST_CASE macro for example is the following – it gets turned into an anonymous template that never gets instantiated:

  #define TEST_CASE(name)                       \
    template <typename T>                       \
    static inline void ANONYMOUS(ANON_FUNC_)()

This means that all test cases are trimmed out of the resulting binary – even in Debug mode! The linker doesn’t ever see the anonymous test case functions because they are never instantiated.

The ANONYMOUS() macro is used to get unique identifiers each time it’s called – it uses the __COUNTER__ preprocessor macro which returns an integer with 1 greater than the last time each time it gets used. For example:

  int ANONYMOUS(ANON_VAR_); // int ANON_VAR_5;
  int ANONYMOUS(ANON_VAR_); // int ANON_VAR_6;

Subcases: the easiest way to share setup / teardown code between test cases

Suppose you want to open a file in a few test cases and read from it. If you don’t want to copy/paste the same setup code a few times you might use the Subcases mechanism of doctest (see Listing 4).

TEST_CASE("testing file stuff") {
  printf("opening the file\n");
  std::ifstream is ("test.txt",
                     std::ifstream::binary);

  SUBCASE("seeking in file") {
    printf("seeking\n");
    // is.seekg()
  }
  SUBCASE("reading from file") {
    printf("reading\n");
    // is.read()
  }
  printf("closing... (by the destructor)\n");
}
			

The following text will be printed:

  opening the file
  seeking
  closing... (by the destructor)
  opening the file
  reading
  closing... (by the destructor)

As you can see the test case was entered twice – and each time a different subcase was entered. Subcases can also be infinitely nested. The execution model resembles a DFS traversal – each time starting from the start of the test case and traversing the ‘tree’ until a leaf node is reached (one that hasn’t been traversed yet) – then the test case is exited by popping the stack of entered nested subcases.

Examples of how to embed tests in production code

If shipping libraries with tests, it is a good idea to add a tag in your test case names (like this: TEST_CASE("[the_lib] testing foo")) so the user can easily filter them out with

  --test-case-exclude=*[the_lib]*

if he wishes to.

• If you are shipping a header-only library there are mainly 2 options:
  1. You could surround your tests with an ifdef to check if doctest is included before your headers like Listing 5.
  2. You could use a preprocessor identifier (like FACT_WITH_TESTS) to conditionally use the tests like Listing 6.
  In both of these cases the user of the header-only library will have to implement the test runner of the framework somewhere in his executable/shared object.
• If you are developing an end product and not a library for developers, then you can just mix code and tests and implement the test runner like described in the section ‘The main() entry point’. You could define the DOCTEST_CONFIG_DISABLE preprocessor identifier in the Release config so no tests are shipped to the customer.
• If you are developing a library which is not header-only, you could again write tests in your headers like shown above, and you could also make use of the DOCTEST_CONFIG_DISABLE identifier to optionally remove the tests from the source files when shipping it – or figure out a custom scheme like the use of a preprocessor identifier to optionally ship the tests - MY_LIB_WITH_TESTS.

// fact.h
#pragma once

inline int fact(int n) {
  return n <= 1 ? n : fact(n - 1) * n; 
}

#ifdef DOCTEST_LIBRARY_INCLUDED
TEST_CASE("[fact] testing the factorial function")
{
  CHECK(fact(0) == 1); // will fail
  CHECK(fact(1) == 1);
  CHECK(fact(2) == 2);
  CHECK(fact(10) == 3628800);
}
#endif // DOCTEST_LIBRARY_INCLUDED
			

// fact.h
#pragma once

inline int fact(int n) {
  return n <= 1 ? n : fact(n - 1) * n; 
}

#ifdef FACT_WITH_TESTS

#include "doctest.h"

TEST_CASE("[fact] testing the factorial function") {
  CHECK(fact(0) == 1); // will fail
  CHECK(fact(1) == 1);
  CHECK(fact(2) == 2);
  CHECK(fact(10) == 3628800);
}
#endif // FACT_WITH_TESTS
			

Compile time benchmarks

So there are 3 types of compile time benchmarks that are relevant for doctest:

• cost of including the header
• cost of assertion macros
• how much the build times drop when all tests are removed with the DOCTEST_CONFIG_DISABLE identifier

In summary:

• Including the doctest header costs around 10ms compared to 250–460ms of Catch – so doctest is 25–50 times lighter
• 50 000 asserts compile for roughly 60 seconds, which is around 25% faster than Catch
• 50 000 asserts can compile for as low as 30 seconds (or even 10) if alternative assert macros [Doctest-5] are used (for power users)
• 50 000 asserts spread in 500 test cases just vanish when disabled with DOCTEST_CONFIG_DISABLE – all of it takes less than 2 seconds!

The lightness of the header was achieved by forward declaring everything and not including anything in the main part of the header. There are includes in the test runner implementation part of the header but that resides in only one translation unit – where the library gets implemented (by defining the DOCTEST_CONFIG_IMPLEMENT preprocessor identifier before including it).

Regarding the cost of asserts – note that this is for trivial asserts comparing 2 integers – if you need to construct more complex objects and have more setup code for your test cases then there will be an additional amount of time spent compiling. This depends very much on what is being tested. A user of doctest provides a real world example of this in his article [Wicht].

In the benchmarks page [Doctest-4] of the project documentation you can see the setup and more details for the benchmarks.

Conclusion

The doctest framework is really easy to get started with and is fully transparent and unintrusive. Including it and writing tests will be unnoticeable both in terms of compile times and integration (warnings, build system, etc). Using it will speed up your development process as much as possible – no other framework is so easy to use!

Note that Catch 2 is on its way (not public yet), and when it is released there will be a new set of benchmarks.

The development of doctest is supported with donations.

References

[Boost] http://www.boost.org/doc/libs/1_60_0/libs/test/doc/html/index.html

[Catch] https://github.com/philsquared/Catch

[CppUTest] https://github.com/cpputest/cpputest

[Doctest-1] https://github.com/onqtam/doctest/blob/master/doc/markdown/faq.md#how-is-doctest-different-from-catch

[Doctest-2] https://github.com/onqtam/doctest/blob/master/doc/markdown/features.md

[Doctest-3] https://github.com/onqtam/doctest/blob/master/doc/markdown/roadmap.md

[Doctest-4] https://github.com/onqtam/doctest/blob/master/doc/markdown/benchmarks.md

[Doctest-5] https://github.com/onqtam/doctest/blob/master/doc/markdown/assertions.md#fast-asserts

[GoogleTest] https://github.com/google/googletest

[UnitTest] https://github.com/unittest-cpp/unittest-cpp

[Wikipedia]https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C.2B.2B

[Wicht] http://baptiste-wicht.com/posts/2016/09/blazing-fast-unit-test-compilation-with-doctest-11.html

Notes: 

More fields may be available via dynamicdata ..