Title: Effective Software Testing

Author: Bob Schmidt

Date: 10 January 2019 16:28:36 +00:00 or Thu, 10 January 2019 16:28:36 +00:00

Summary: Pete Goodliffe describes a healthy software test regimen.

Body: 

Fights would not last if only one side was wrong.
~ François de la Rochefoucauld

The mid-twentieth-century philosophers and purveyors of jaunty, tuneful hair, The Beatles, told us all you need is love. They emphasised the point: love is all you need. Love; that’s it. Literally. Nothing else.

It’s incredible how long a career they had given that they didn’t need to eat or drink.

In our working relationships with other inhabitants of the software factory, we would definitely benefit from more of that sentiment. A little more love might lead to a lot better code! Programming in the real world is an interpersonal endeavour, and so is inevitably bound up in relationship issues, politics, and friction from our development processes.

We work closely with many people. Sometimes in stressful scenarios.

It is not healthy for our working relationships, nor for the consequent quality of our software, if our teams are not working smoothly together. But many teams suffer these kinds of problem.

As a tribe of developers, one of our rockier relationships is with the QA enclave; largely because we interact with them very closely, often at the most stressful points in the development process. In the rush to ship software before a deadline, we try to kick the software soccer ball past the testing goalkeepers.

So let’s look at that relationship now. We’ll see why it’s fraught, and why it must not be.

What is QA good for?

To some it’s obvious what they do. To others it’s a mystery. The ‘QA’ department (that is, Quality Assurance) exists to ensure that your project ships a software product of sufficient quality. They are a necessary and vital part of the construction process.

What does this entail? The most obvious and practical answer is that they have to test the living daylights out of whatever the developers create in order to ensure:

• That it matches the specification and requirements – that every feature that should be implemented has been implemented.
• That the software works correctly on all platforms – that is, it works on all OSes, on all versions of those OSes, on all hardware platforms, and on all supported configurations (e.g., meeting minimum memory requirements, minimum processor speeds, network bandwidth, etc.).
• That no faults have been introduced in the latest build – the new features don’t break any other behaviour, and no regressions (the reintroduction of previous bad behaviour) have been introduced.

Their name is ‘QA,’ not just ‘the testing department,’ and for a reason. Their role is not just pushing buttons like robots; it’s baking quality into the product.

To do this, QA must be deeply involved throughout, not just a final adjunct to the development process.

• They have a hand in the specification of the software, to understand – and shape – what will be built.
• They contribute to design and construction, to ensure that what’s built will be testable.
• They are involved heavily in the testing phase, naturally.
• And also in the final physical release: they ensure that what was tested is what is actually released and deployed.

Software development: shovelling manure

In unenlightened workplaces, the development process is modelled as a huge pipe: conveying raw materials pumped in the top, through various processes, until perfectly formed software gushes (well, perhaps dribbles) out the end. The process goes something like this:

• Someone (perhaps a business analyst or product manager) pours some requirements into the mouth of the pipe.
• They flow through architects and designers, where they turn into specifications and pretty diagrams (or good intentions, and smoke and mirrors).
• This flows through the programmers (where the real work gets done, naturally), and turns into executable code.
• Then it flows into QA. Where it hits a blockage as the ‘perfectly formed’ software magically turns into a non-functioning disaster. These people break the code!
• Eventually the developers push hard enough down the pipe to break this blockage, and the software finally flows out of the far end of the pipe.

In the fouler development environments, this pipe more closely resembles a sewer. QA feels like the developers are pumping raw sewage down to them, rather than handing them a thoughtfully gift-wrapped present. They feel they are being dumped on, rather than worked with.

Is software development really this linear? Do our processes really work like this simple pipeline (regardless of how pure the contents)?

No. They don’t.

The pipe is an interesting first approximation (after all, you can’t test code that hasn’t been written yet), but far too simplistic a model for real development. The linear pipeline view is a logical corollary of our industry’s long fascination with the flawed waterfall development methodology. (This model is often attributed to Winston Royce. Although he wrote about it in the 1970s, it was as an illustration of a flawed development process, not a laudable one. [1])

It is wrong to view software development as a linear process.

However, this view of the development process does explain why the development team’s interaction with the QA team isn’t as smooth as it should be. Our processes and models of interaction are too often shaped by the flawed sewage-based development metaphor. We should be in constant communication, rather that just throwing them software towards the end of the development effort.

A false dichotomy

Our inter-team interactions are hindered because they are just that: interactions between separate teams. The QA people are considered a different tribe, distinct from the ‘important’ developers. This bogus, partitioned vision of the development organisation inevitably leads to problems.

When QA and development are seen as separate steps, as separate activities, and therefore as very separate teams, an artificial rivalry and disconnect can too easily grow. This is reinforced physically by our building artificial silos between testers and developers. For example:

• The two teams have different managers and different reporting lines of responsibility.
• The teams are not colocated, and have very different desk locations (I’ve seen QA in separate desk clusters, on different floors, in different buildings, and even – in an extremely silly case – on another continent).
• There are different team structures, recruiting policies, and expected turnover of staff. Developers are valued resources, whereas testers are seen as replaceable cheap mercenaries.
• And most pernicious: the teams have very different incentives to complete tasks. For example, the developers are working with the promise of bonus pay if they complete a job quickly, but the testers are not. In this case, the developers rush to write the code (probably badly, because they’re hurrying). They then get very cross when the QA guys won’t sanction it for a timely release.

We reinforce this chasm with stereotypes: developers create, testers break.

There is an element of truth there. There are different activities and different skills required in both camps. But they are not logically separate, silo-ed activities. Testers don’t find software faults to just break things and cause merry hell for developers. They do it to improve the final product.

They are there to bake in quality. That’s the Q in QA. And they can only do this effectively in tandem with developers.

Beware of fostering an artificial separation between development and testing efforts.

By separating these activities, we breed rivalry and discord. Often, development processes pit QA as the bad guy against the developer hero. Testers are pictured standing in the doorway, blocking a plucky software release getting out. It’s as if they are unreasonably finding faults in our software. They are nit-picking over minutiae.

It’s almost as if they’re running into the forests, catching wild bugs, and then injecting them into our otherwise perfect code.

Does that sound silly?

Of course it does when you read it here, but it’s easy to start thinking that way: ‘The code is fine; those guys just don’t know how to use it.’ Or: ‘They’ve been working far too long to find such a basic bug; they really don’t know what they’re doing.’ Software development is not a battle. (Well, it shouldn’t be.) We’re all on the same side.

Fix the team to fix the code

Conway’s famous law [2] describes how an organisation’s structure – and specifically the lines of communication between its teams – dictates software structure. It is popularly paraphrased as: “If you have four teams writing a compiler, it’ll be a four-pass compiler.” Experience shows this to be pretty accurate. In the same way that team structure affects the code, so does the health of the interactions within the software.

Unhealthy team interactions result in unhealthy code.

We can improve the quality of our software, and the likelihood of producing a great release, by addressing these health issues: by improving the relationship between developers and QA. Working together rather than waging war. Love, remember, is all you need.

This is a general principle and applies far more broadly than just between developers and QA. Cross-functional teams in all respects are very helpful.

This comes down to the way we interact and work with QA. We should not treat them as puppets whose strings we pull, or to whom we throw ropey software to test. Instead, we treat them as coworkers. Developers must have good rapport with QA: a friendship and camaraderie.

Let’s look at the practical ways we can work better with these inhabitants of the QA kingdom. We’ll do this by looking at the major places that developers interact with QA.

But we have unit tests!

We are conscientious coders. We want to make rock-solid software. We want to craft great lines of code with a coherent design, that contribute to an awesome product.

That’s what we do.

So we employ development practices that ensure our code is as good as possible. We review, we pair, we inspect. And we test. We write automated unit tests. 

We have tests! And they pass! The software must be good. Mustn’t it?

Even with unit tests coming out of our ears, we still can’t guarantee that our software is perfect. The code might operate as the developers intended, with green test lights all the way. But that may not reflect what the software is supposed to do.

The tests may show that all input the developers envisioned was handled correctly. But that may not be what the user will actually do. Not all of the use cases (and abuse cases) of the software have been considered up front. It is hard to consider all such cases – software is a mightily complex thing. Thinking about this is exactly what the QA people are great at.

Because of this, a rigorous testing and QA process is still a vital part of a software development process, even if we have comprehensive unit tests in place. Unit tests act as our responsible actions to prove that the code is good enough before we hand it over to the testers to work on.

Releasing a build to QA

We know that the development process isn’t a straight line; it’s not a simple pipeline. We develop iteratively and release incremental improvements: either a new feature that needs validation or a fixed bug that should be validated. It’s a cycle that we go around many times. Over the course of the construction process, we will create numerous builds that will be sent to QA.

So we need a smooth build and hand-off process.

This is vital: the hand-off of our code must be flawless – the code must be responsibly created and thoughtfully distributed. Anything less is an insult to our QA colleagues.

We must build with the right attitude: giving something to QA is not the act of throwing some dog-eared code, built on any old machine, over the fence for them. It’s not a slapdash or slipshod act.

Also, remember that this is not a battle: we don’t aim to slip a release past QA, deftly avoiding their defence. Our work must be high quality, and our fixes correct. Don’t cover over the symptoms of the more obvious bugs and hope they’ll not have time to notice the underlying evils in the software.

Rather, we must do everything we can to ensure that we provide QA with something worthy of their time and effort. We must avoid any silly errors or frustrating sidetracks. Not to do so shows a lack of respect to them.

Not creating a QA build thoughtfully and carefully shows a lack of respect to the testers.

This means that you should follow the guidelines covered in the following sections.

Test your work first

Prior to creating a release build, the developers should have done as good a job as possible to prove that it is correct. They should have tested the work they’ve done beforehand. Naturally, this is best achieved with a comprehensive suite of regularly run unit tests. This helps catch any behavioural regressions (reoccurrences of previous errors). Automated tests can weed out silly mistakes and embarrassing errors that would waste the tester’s time and prevent them from finding more important issues.

With or without unit tests, the developers must have actually tried the new functionality, and satisfied themselves that it works as well as is required. This sounds obvious, but all too often changes or fixes that should ‘just work’ get released, and cause embarrassing problems. Or a developer sees the code working in a simple case, considers it adequate for release, and doesn’t even think about the myriad ways it could fail or be mis-used.

Of course, running a suite of unit tests is only as effective as the quality of those tests. Developers take full responsibility for this. The test set should be thorough and representative. Whenever a fault is reported from QA, demonstrative unit tests should be added to ensure that those faults don’t reappear after repair.

Release with intent

When a new version is being released to QA, the developer must make it clear exactly how it is expected to work. Don’t produce a build and just say, “See how this one works.”

Describe clearly exactly what new functionality is and isn’t implemented: exactly what is known to work and what is not. Without this information you cannot direct what testing is required. You will waste the testers’ time. You communicate this in release notes.

It’s important to draw up a set of good, clear release notes. Bundle them with the build in an unambiguous way (e.g., in the deployment file, or with a filename that matches the installer). The build must be given a (unique) version number (perhaps with an incrementing build number for each release). The release notes should be versioned with this same number.

For each release, the release notes should clearly state what has changed and what areas require greater testing effort.

More haste, less speed

Never rush out a build, no matter how compelling it seems. The pressure to do this is greatest as a deadline looms, but it’s also tempting to sneak a build out before leaving the office for the evening. Rushing work like this encourages you to cut corners, not check everything thoroughly, or pay careful attention to what you’re doing. It’s just too easy. And it’s not the right way to give a release to QA. Don’t do it.

If you feel like a school kid desperately trying to rush your homework and get ‘something’ in on time, in the full knowledge that the teacher will be annoyed and make you do it again, then something is wrong! Stop. And think.

Never rush the creation of a build. You will make mistakes.

Some products have more complex testing requirements than others. Only kick off an expensive test run across platforms or OSes if you think it’s worthwhile, when an agreed number of features and fixes have been implemented.

Automate

Automation of manual steps always removes the potential for human error. So automate your build or release process as much as possible. If you can create a single script that automatically checks out the code, builds it, runs all unit tests, creates installers or deploys on a testing server, and uploads the build with its release notes, then you remove the potential for human error for a number of steps. Avoiding human error with automation helps to create releases that install properly each time and do not contain any regressions. The QA guys will love you for that.

Respect

The delivery of code into QA is the act of producing something stable and worthy of potential release, not the act of chucking the latest untested build at QA. Don’t throw a code grenade over the fence, or pump raw software sewage at them.

On getting a fault report

We give the test guys a build. It’s our best effort yet, and we’re proud of it. They play with it. Then they find faults. Don’t act surprised. You knew it was going to happen.

Testing will only reveal problems that software developers added to the system (by omission or commission). If they find a fault, it was your fault!

On finding a bug, they lodge a fault report: a trackable report of the problem. This report can be prioritised, managed, and once fixed, checked for later regression.

It is their responsibility to provide accurate, reliable fault reports, and to send them through in a structured and orderly way – using a good bug tracking system, for example. But faults can be maintained in a spreadsheet, or even by placing stories in a work backlog. (I’ve seen all these work.) As long as there’s a clear system in place that records and announces changes to the state of a fault report.

So how do we respond to a fault report?

First, remember that QA isn’t there to prove that you’re an idiot and make you look bad. The fault report isn’t a personal slight. So don’t take it personally.

Don’t take fault reports personally. They are not a personal insult!

Our ‘professional’ response is, “Thanks, I’ll look into it.” Just be glad it was QA who they found it, and not a customer. You are allowed to feel disappointed that a bug slipped through your net.

You should be worried if you are swamped by so many fault reports that you don’t know where to start – this is a sign that something very fundamental is wrong and needs addressing. If you’re in this kind of situation, it’s easy to resent each new report that comes in.

Of course, we don’t leap onto every single fault as soon as it is reported. Unless it is a trivial problem with a super-simple fix, there are almost certainly more important problems to address first. We must work in collaboration with all the project stakeholders (managers, product specialists, customers, etc.) to agree which are the most pressing issues to spend our time on.

Perhaps the fault report is ambiguous, unclear, or needs more information. If this is the case, work with the reporter to clarify the issues so you can both understand the problem fully, can reproduce it reliably, and know when it has been closed.

QA can only uncover bugs from development, even if it’s not a fault that you were the direct author of. Perhaps it stems from a design decision that you had no control over. Or perhaps it lurks in a section of code that you didn’t write. But it is a healthy and professional attitude to take responsibility for the whole product, not just your little part of the codebase.

Our differences make us stronger

Whenever you’re in conflict with someone, there is one factor that can make the difference between damaging your relationship and deepening it. That factor is attitude.
~ William James (philosopher and psychologist)

Effective working relationships stem from the right developer attitudes.

When we’re working with QA engineers, we must understand and exploit our differences:

• Testers are very different from developers. Developers often lack the correct mind-set to test effectively. It requires a particular way of looking at software, particular skills and peccadilloes to do well. We must respect the QA team for these skills – skills that are essential if we want to produce high-quality software.
• Testers are inclined to think more like a user than a computer; they can give valuable feedback on perceived product quality, not just on correctness. Listen to their opinions and value them.
• When a developer works on a feature, the natural instinct is to focus on the happy path – on how the code works when everything goes well (when all input is valid, when the system is working fully with maximum CPU, no memory or disk space issues, and every system call works perfectly).

  It’s easy to overlook the many ways that software can be used incorrectly, or to overlook whole classes of invalid input. We are wired to consider our code through these natural cognitive biases. Testers tend not to be straightjacketed by such biases.

• Never succumb to the fallacy that QA testers are just ‘failed devs’. There is a common misconception that they are somehow less intelligent, or less able. This is a damaging point of view and must be avoided.

Cultivate a healthy respect for the QA team. Enjoy working with them to craft excellent software.

Pieces of the puzzle

We need to see testing not as the ‘last activity’ in a classic waterfall model; development just doesn’t work like that. Once you get 90% of the way through a waterfall development process into testing, you will likely discover that another 90% of effort is required to complete the project. You cannot predict how long testing will take, especially when you start it far too late in the process.

Just as code benefits from a test-first approach, so does the entire development process. Work with the QA department and get their input early on to help make your specifications verifiable, ensure their expertise feeds into product design, and that they will agree that the software is maximally testable before you even write a line of code.

The QA team is not the sole owner of, nor the gatekeeper of ‘quality.’ It is everyone’s responsibility.

To build quality into our software and to ensure that we work well together, all developers should understand the QA process and appreciate its intricate details.

Remember that QA is part of the same team; they are not a rival faction. We need to foster a holistic approach, and maintain healthy relationships to grow healthy software. All we need is love.

Questions

• How close do you think your working relationship with your QA colleagues is? Should it be better? If so, what steps can you take to improve it?
• What is the biggest impediment to software quality in your development organisation? What is required to fix this?
• Ask the QA team what would help them most.
• Who is responsible for the ‘quality’ of your software? Who gets the ‘blame’ when things go wrong? How healthy is this?
• How good do you think your testing skills are? How methodically do you test a piece of code you’re working on before you check in or hand off?

References

[1] Winston Royce (1970) ‘Managing the Development of Large Software Systems’, Proceedings of IEEE WESCON 26:1–9.

[2] Melvin E. Conway (1968) ‘How Do Committees Invent?’ Datamation 14:5 : 28–31.

Pete Goodliffe Pete is a programmer who never stays at the same place in the software food chain. He has a passion for curry and doesn’t wear shoes.

Notes: 

More fields may be available via dynamicdata ..