Title: Testing Times (Part 1)

Author: Bob Schmidt

Date: 05 January 2018 17:35:11 +00:00 or Fri, 05 January 2018 17:35:11 +00:00

Summary: Pete Goodliffe explores how to test code to ensure it works as expected.

Body: 

Quality is free, but only to those who are willing to pay heavily for it.
~ Tom DeMarco and Timothy Lister,
Peopleware: Productive Projects and Teams

Test-driven development (TDD): to some it’s a religion. To some, it’s the only sane way to develop code. To some, it’s a nice idea that they can’t quite make work. And to others, it’s a pure waste of effort.

What is it, really?

TDD is an important technique for building better software, although there is still confusion over what it means to be test driven, and over what a unit test really is. Let’s break through this and discover a healthy approach to developer testing, so we can write better code.

Why test?

It’s a no-brainer: we have to test our code.

Of course you run your new program to see whether it works. Few programmers are confident enough, or arrogant enough, to write code and release it without trying it out somehow. When you do see corners cut, the code rarely works the first time: problems are found, either by QA, or – worse – when a customer uses it.

Shortening the feedback loop

To develop great software, and develop it well, programmers need feedback. We need to receive feedback as frequently and as quickly as possible. Good testing strategies shorten the feedback loop, so we can work most effectively:

• We know that our code works when it’s used in the field and returns accurate results to users. If it doesn’t, they complain. If that was our only feedback loop, software development would be very slow and very expensive. We can do better.
• To ensure correctness before we ship, the QA team tests candidate releases. This pulls in the feedback loop; the answers come back more quickly, and we avoid making expensive (and embarrassing) mistakes in the field. But we can still do better.
• We want to check that our new subsystems work before integrating them into the project. Typically, a developer will spin up the application and execute their new code as best they can. Some code can be rather inconvenient to test like this, so it’s possible to create a small separate test harness application that exercises the code. These development tests again reduce the feedback loop; now we find out whether our code is functioning correctly as we work on it, not later on. But we can still do better.
• The subsystems are comprised of smaller units: classes and functions. If we can easily get feedback on correctness and quality of code at this level, then we reduce the feedback loop again. Tests at the smallest level give the fastest feedback.

The shorter the feedback loop, the faster we can iterate over design changes, and the more confident we can feel about our code. The sooner we learn that there’s a problem, the easier and less expensive the fix is, because our brain is still engaged with the problem and we recall the shape of the code.

To improve our software development we need rapid feedback, to learn of problems as soon as they appear. Good testing strategies provide short feedback loops.

Manual tests (either performed by a QA team, or by the programmers inspecting their own handiwork) are laborious and slow. To be at all comprehensive, it requires many individual steps that need repeating each time you make a minor adjustment to the code.

But hang on, isn’t repeated laborious work something that computers are good at? Surely we can use the computer to run the tests for us automatically. That speeds up the running of the tests, and helps to close the feedback loop further.

Automated tests with a short feedback loop don’t just help you to develop the code. Once you have a selection of tests, you needn’t throw them away. Stash them in a test pool, and keep running them. In this way your test code works like a canary in a mine – signalling any problem before it becomes fatal. If in the future someone (even you on a bad day) modifies the code to introduce errant behaviour (a functional regression), the test will point this out immediately.

Code that tests code

So the ideal is to automate our development testing as much as possible: work smarter, not harder. Your IDE can highlight syntax errors as you type – wouldn’t it be great if it could show you test breakages at the same speed?

Computers can run tests rapidly and repeatedly, reducing the feedback loop. Although you can automate desktop applications with UI testing tools, or use browser-based technology, most often development tests see the coder writing a programmatic test scaffold that invokes their production code (the SUT: System Under Test), prodding it in particular ways to check that it responds as expected.

We write code to test code. Very meta.

Yes, writing these tests takes up the programmer’s precious time. And yes, your confidence in the code is only as good as the quality of the tests that you write. But it’s not hard to adopt a test strategy that improves the quality of your code and makes it safer to write. This helps reduce the time it takes you to develop code: more haste, less speed. Studies have shown that a sound testing strategy substantially reduces the incidence of defects. [1].

It is true that a test suite can slow you down if you write brittle, hard to understand tests, and if your code is so rigid that a change in one method forces a million tests to be re-written. That is an argument against bad test suites, not against testing in general (in the same way that bad code is not an argument against programming in general).

Who writes the tests?

In the past some have argued for the role of a dedicated ‘unit-test engineer’ who specialises in verifying the code of an upstream programmer. But the most effective approach is for the programmers themselves to write their own development tests.

After all, you’d be testing your code as you write it, anyway.

We need tests at all levels of the software stack and development process. However, programmers particularly require tests at the smallest scope possible, to reduce the feedback loop and help develop high-quality software as quickly and easily as possible.

Types of tests

There are many kinds of tests, and often when you hear someone talk about a ‘unit test’ they may very likely mean some other kind of code test. We employ:

Unit tests

Unit tests specifically exercise the smallest ‘units’ of functionality in isolation, to ensure that they each function correctly. If it’s not driving a single unit of code (which could be one class or one function) in isolation (i.e., without involving any other ‘units’ from the production code), then it’s not a unit test.

This isolation specifically means that a unit test will not involve any external access: no database, network, or filesystem operations will be run.

Unit-test code is usually written using an off-the-shelf ‘xUnit’ style framework. Every language and environment has a selection of these, and some have a de facto standard. There’s nothing magical about a testing framework, and you can get a long way writing unit tests with just the humble assert. We’ll look at frameworks later.

Integration tests

These tests inspect how individual units integrate into larger cohesive sets of cooperating functionality. We check that the integrated components glue together and interoperate correctly.

Integration tests are often written in the same unit test frameworks; the difference is simply the scope of the system under test. Many people’s ‘unit tests’ are really integration-level tests, dealing with more than one object in the SUT. In truth, what we call this test is nowhere near as important as the fact that the test exists!

System tests

Otherwise known as end-to-end tests, these can be seen as a specification of the required functionality of the entire system. They run against the fully integrated software stack, and can be used as acceptance criteria for the project.

System tests can be implemented as code that exercises the public APIs and entry points to the system, or they may drive the system from outside using a tool like Selenium, a web browser automator. It can be hard to realistically test all of an application’s functionality through its UI layer, in which case we employ subcutaneous tests that drive the code from the layer just below the interface logic.

Because of the larger scope of system tests, the full suite of tests can take considerable time to execute. There may be much network traffic involved or slow database access to account for. The set-up and tear-down costs can be huge to get the SUT ready to run each system test.

Each level of developer tests establishes a number of facts about the SUT, and constructs a series of test cases that prove that these facts hold.

There are different styles of test-driven development. A project can be driven by a unit-test mentality: where you would expect to see more unit tests than integration tests, and more integration tests than system tests. Or it may be driven by a system-test mentality: the reverse, with far fewer unit tests. Each kind of test is important in its own right, and all should be present in a mature software project.

When to write tests

The term TDD (that is, test-driven development) is conflated with test-first development, although there really are two separate themes here. You can ‘drive’ your design from the feedback given by tests without religiously writing those tests first.

However, the longer you leave it to write your tests, the less effective those tests will be: you’ll forget how the code is supposed to work, fail to handle edge cases, or perhaps even forget to write tests at all. The longer you leave it to write your tests, the slower and less effective your feedback loop will be.

The test-first ‘TDD’ approach is commonly seen in XP circles. The mantra is: don’t write any production code unless you have a failing test. The test-first TDD cycle is:

• Determine the next piece of functionality you need. Write a test for your new functionality. Of course, it will fail.
• Only then implement that functionality, in the simplest way possible. You know that your functionally is in place when the test passes. As you code, you may run the test suite many times. Because each step adds a small new part of functionality, and therefore a small test, these tests should run rapidly.
• This is the important part that’s often overlooked: now tidy up the code. Refactor unpleasant commonality. Restructure the SUT to have a better internal structure. You can do all this with full confidence that you won’t break anything, as you have a suite of tests to validate against.
• Go back to step 1 and repeat until you have written passing test cases for all of the required functionality.

This is a great example of a powerful, and gloriously short, feedback loop. It’s often referred to as the red-green-refactor cycle in honour of unit-test tools that show failing tests as a red progress bar, and passing tests as a green bar.

Even if you don’t honour the test-first mantra, keep your feedback loop short and write unit tests during, or very shortly after, a section of code. Unit tests really do help ‘drive’ our design: not only does it ensure that everything is functionally correct and prevent regressions, it’s also a great way to explore how a class API will be used in production – how easy and neat it is. This is invaluable feedback. The tests also stand as useful documentation of how to use a class once it’s complete.

Write tests as you write the code under test. Do not postpone test writing, or your tests will not be as effective.

This test-early, test-often approach can be applied at the unit, integration, and system level. Even if your project has no infrastructure for automated system tests, you can still take responsibility and verify the lines of code you write with unit tests. It’s cheap and, given good code structure, it’s easy. (Without good code structure, an attempt to write a test will help drive you towards better code structure.)

Another essential time to write a test is when you have to fix a bug in the production code. Rather than rush out a code fix, first write a failing unit test that illustrates the cause of the bug. Sometimes the act of writing this test serves to show other related flaws in the code. Then apply your bugfix, and make the test pass. The test enters your test pool, and will serve to ensure that the bug doesn’t reappear in the future.

When to run tests

You can see a lot by just looking.
~ Yogi Berra

Clearly, if you develop using TDD, you will be running your tests as you develop each feature to prove that your implementation is correct and sufficient.

But that is not the only life of your test code.

Add both the production code and its tests to version control. Your test is not thrown away, but joins the suite of existent tests. It lives on to ensure that your software continues to work as you expect. If someone later modifies the code badly, they’ll be alerted to the fact before they get very far.

All tests should run on your build server as part of a continuous integration toolchain. Unit tests should be run by developers frequently on their development machines. Some development environments provide shortcuts to launch the unit tests easily; some systems scan your filesystem and run the unit tests when files change. However, I prefer to bake tests right into the build/compile/run process. If my unit-test suite fails, the code compilation is considered to have failed and the software cannot be run. This way, the tests are not ignorable. They run every time the code is built. When invoked manually, developers can forget to run tests, or will ‘avoid the inconvenience’ whilst working.

Injecting the tests directly into the build process also encourages tests to be kept small, and to run fast.

Encourage tests to be run early and often. Bake them into your build process.

Integration and system tests may take too long to run on a developer’s machine every compilation. In this case, they may justifiably run only on the CI build server.

Remember that code-level, automated testing doesn’t remove the need for a human QA review before your software release. Exploratory testing by real testing experts is invaluable, no matter how many unit, integration, and system tests you have in place. An automated suite of tests avoids introducing those easily fixable, easily preventable mistakes that would waste QA’s time. It means that the things the QA guys do find will be really nasty bugs, not just simple ones. Hurrah!

Good development tests do not replace thorough QA testing.

Next time

In the next instalment, we’ll look at what should be tested, what a (good) test looks like, and how we structure tests.

See you next time.

Questions

• How many styles of testing have you been exposed to?
• Which is the best development test technique: test-first, or test (very shortly) after coding? Why? How has your experience shaped this answer?
• Is it a good idea to employ a specialist unit-test writing engineer to help craft a high-quality test suite?
• Why do QA departments traditionally not write much test code, and generally focus on running through test scripts and performing exploratory testing?

Reference

[1] David Janzen and Hossein Saiedian, ‘Test-Driven Development Concepts, Taxonomy, and Future Direction,’ Computer 38:9 (2005).

Notes: 

More fields may be available via dynamicdata ..