Software Testing: General Testing Principles

By admin123
.
Software Testing / Chapter 01 01 / 09

General
Testing
Principles.

Seven principles that guide testers
Developed over the years from a variety of sources.
◉ ISTQB FOUNDATION · SECTION 1.7
The framework · seven principles 02 / 09
Principles that prevent the types of problems described previously

Testing is a very complex activity, and it can be difficult to do well.

P/01
Presence
of bugs
Discovery
P/02
Exhaustive
testing
Impossible
P/03
Early
testing
Timeliness
P/04
Defect
clustering
Distribution
P/05
Pesticide
paradox
Evolution
P/06
Context
dependent
Adaptation
P/07
Absence
of errors
Fallacy
Principle 01 & 02 · discovery and limits 03 / 09
Principle 01
+

Testing shows
the presence
of bugs.

Running a test can only show that one or more defects exist. Testing cannot show that the software is error free. Consider the top 10 wanted criminals website — no functional defects, yet the website failed.

Principle 02
∞

Exhaustive
testing is impossible.

For large complex systems, exhaustive testing is not possible. Even small pieces of software have too many data combinations. Risk and priorities are used to concentrate on the most important aspects.

3-character password permutations (standard keyboard)
16,777,216COMBINATIONS
Principle 03 · early testing 04 / 09
The earlier, the better

The earlier a problem is found, the less it costs to fix.

As a deployment date approaches, time pressure increases dramatically. The earlier the testing activity is started, the longer the elapsed time available. Testers do not have to wait until software is available to test.

Many problems can be traced back to missing or incorrect requirements. Creating tests early highlights these issues — are requirements testable? Are they ambiguous or missing?

Cost escalation model
STAGE / 01
Requirements
Error introduced and detected here. Cost to fix: minimal.
↓ ×10
STAGE / 02
Design
Error migrates. Rework required in design documents.
↓ ×100
STAGE / 03
Code
Error replicated across multiple components. Rework multiplies.
↓ ×1000
STAGE / 04
Production
Defect found at acceptance. Multiple work-products reworked. Cost rises steeply.
Principle 04 · defect clustering 05 / 09
The Pareto principle in software

Approximately 80% of the problems are found in about 20% of the modules.

In a large application, it is often a small number of modules that exhibit the majority of the problems. Testing should target these areas — but must still search diligently elsewhere.

C/01
System complexity
C/02
Volatile code
C/03
Change upon change
C/04
Staff inexperience
Principle 05 · the pesticide paradox 06 / 09
Diminishing returns

Running the same set of tests continually will not continue to find new defects.

Developers will soon know that the test team always tests the boundaries of conditions, so they will test these before delivery. Continuing to use the same test set results in decreasing effectiveness.

Using other techniques will find different defects. Any regression test set needs to change to reflect business needs and current risks.

→Same tests repeatedDECLINING
→Developers adaptANTICIPATED
→New techniques neededEVOLVE
→Regression sets must changeADAPT
The software may fail in production because the regression tests are no longer relevant to the requirements of the system.
Principle 06 · context dependent 07 / 09
Different testing is necessary in different circumstances
Low Risk

Mortgage
calculator.

An application for calculating the length of a mortgage. Important, but limited scope. Lower investment in testing required.

versus
High Risk

Air traffic
control.

We need to test with far more rigour. The higher the possibility of losses, the more we need to invest in testing before implementation.

For an e-commerce site, we should concentrate on security aspects. Can payment be made with an invalid credit card? Security testing is a specialist area, not appropriate for all applications.

Principle 07 · absence of errors fallacy 08 / 09
The most dangerous assumption

Software with no known errors is not necessarily ready to be shipped.

Does the application under test match up to the users' expectations of it? The fact that no defects are outstanding is not a good reason to ship the software.

Before dynamic testing has begun, there are no defects reported against the code delivered. Does this mean that software that has not been tested — but has no outstanding defects against it — can be shipped? We think not!

Defects outstanding before testing begins
0DOES NOT MEAN READY
Conclusion · the seven principles 09 / 09
Principles that guide testers

Not all obvious, but essential.

— P / 01
Presence of bugs
— P / 02
Exhaustive impossible
— P / 03
Early testing
— P / 04
Defect clustering
— P / 05
Pesticide paradox
— P / 06
Context dependent
— P / 07
Absence of errors
Testers use these principles with the test techniques to prevent the types of problems that cause software to fail — and to ensure that testing effort is directed where it matters most.
.