Software Testing: Deciding When Enough Is Enough

.

Software Testing / Chapter 01 01 / 06

Deciding
When Enough
Is Enough.

Prioritisation · Completion criteria

How much testing is enough, and when do we stop?

◉ ISTQB FOUNDATION · SECTION 1.5

The question · finite and limited 02 / 06

How much testing is enough?

We cannot test everything.

We have so far decided that we cannot test everything, even if we would wish to. We also know that every system is subject to risk of one kind or another and that there is a level of quality that is acceptable for a given system.

These are the factors we will use to decide how much testing to do.

◉Cannot test everythingCONSTRAINT

◉Every system has riskFACTOR

◉Acceptable quality levelFACTOR

The most important aspect of achieving an acceptable result from a finite and limited amount of testing is prioritisation.

The strategy · prioritisation 03 / 06

Do the most important tests first

At any time, the tests that have been done are more important than the ones still to be done.

Even if the testing activity is cut in half it will still be true that the most important testing has been done. The most important tests will be those that test the most important aspects of the system.

Priority A

F

Most important
functions.

Functions as defined by the users or sponsors of the system — the core reasons the software exists.

Priority B

NF

Most important
behaviour.

The most important non-functional behaviour — performance, security, usability — that enables the system to deliver.

Priority C

R

Most significant
risks.

Address the areas where failure would cause the greatest harm — the vulnerabilities that matter most.

The boundary · completion criteria 04 / 06

When is it safe to stop?

Objective criteria so that time and pressure do not confuse the outcome.

The next most important aspect is setting criteria that will give you an objective test of whether it is safe to stop testing.

These criteria, usually known as completion criteria, set the standards for the testing activity.

01How much of the software is to be testedCOVERAGE

02What levels of defects can be toleratedTOLERANCE

Completion criteria provide a basis for planning — but the triangle of resources still applies.

The reality · resource compromise 05 / 06

The triangle of resources

The desired level of quality and risk may have to be compromised.

Priorities and completion criteria provide a basis for planning, but time, money, and quality still consume one another. In the end, the desired level of quality and risk may have to be compromised.

But our approach ensures that we can still determine how much testing is required to achieve the agreed levels — and we can still be certain that any reduction in the time or effort available for testing will not affect the balance.

Conclusion · the guarantee 06 / 06

Whenever we stop

The most important tests are those already done.

— PRINCIPLE / 01

Prioritise

— PRINCIPLE / 02

Set Criteria

— PRINCIPLE / 03

Accept Compromise

Even if testing is cut short, the approach ensures that the most important tests will still be those that have already been done — and that the balance of risk is understood.

.