Introduction To Software Testing - Keeping Software Under Control

Software Engineering / Lecture 02 01 / 08

Keeping
Software
Under Control.

Risk · Quality · Resources

Deciding when enough is enough.

◉ EST. 2024 · DEPT. OF CS

The core problem · themes 02 / 08

Identifying the common threads

Either insufficient testing, or the wrong type.

With all of the examples we have seen so far, what common themes can we identify? There may be several, but one theme is clear: either insufficient testing or the wrong type of testing was done.

More and better software testing seems a reasonable aim, but that aim is not quite as simple to achieve as we might expect.

Exhaustive testing of complex systems is simply not possible.

If every possible test had been run on the smartphone mapping app, problems would have been detected and rectified prior to launch.

However, if every test had been run, the testing would still be running now, and the product launch would never have taken place.

Testing and risk · proportionality 03 / 08

Risk is inherent in all software development

Greater risk implies more and better testing.

High Risk System

Automatic
Flight Control.

Intuitively, we would test an automatic flight control system extensively. The probability of failure in a complex system is greater, and the impact of that failure is catastrophic.

vs.

Lower Risk System

Video Game
System.

While still requiring quality, the implications of failure are significantly lower. What we test, and how much we test it, must be directly related to the level of risk.

Testing and quality · requirements 04 / 08

Quality is notoriously hard to define

If a system meets users' requirements, that is a good starting point.

Functional Failure

UK Online Tax Returns

An obvious functional weakness allowed one user to view another user's financial details. It is hard to imagine any user finding this situation anything other than unacceptable.

Non-Functional Failure

Top 10 Criminals Website

There was no failure of functionality here. The system was simply swamped by requests for access after radio and TV coverage. It could not deliver services under peak load.

Case study · Airbus A380 05 / 08

The Airbus A380 Story

Different versions. Fatal mismatch.

Although completed subparts could not be brought together to build an entire aircraft, each of the subparts was ‘correctly manufactured’.

The large subparts were made in two different countries, with different versions of the CATIA design software in each manufacturing base. When bringing together two halves of the aircraft, the different software meant that the wiring on one did not match the wiring on the other.

Testing may have taken place, but it did not test something as straightforward as software version compatibility.

530km

Cables. The total length of wiring inside the A380 aircraft.

100k

Wires. Individual connections required to make the systems function.

40.3k

Connectors. Physical joining points that failed to align due to software mismatch.

The resource triangle · balance 06 / 08

Competing demands for resources

Time. Money. Quality.

The software development process must balance competing demands. If we need to deliver a system faster, it will usually cost more. These three items affect one another, and influence the features included in delivered software.

Testing cannot directly remove defects, nor can it directly enhance quality. By reporting defects, it makes their removal possible and contributes to enhanced quality.

Quality

Time

Money

Deciding when 'enough is enough' 07 / 08

How much testing is enough, and how do we stop?

Prioritisation is the key to an acceptable result.

01

Do the most important tests first.

At any time, you can be certain that the tests done are more important than the ones still to do. Even if testing is cut in half, the most important testing has been done.

02

Set objective completion criteria.

Set criteria that give an objective test of whether it is safe to stop. These define how much of the software is to be tested and what levels of defects can be tolerated.

03

Maintain the balance.

The resource triangle still applies. Any reduction in time or effort available will not affect the balance—the most important tests will always be those that have already been done.

Conclusion · keeping control 08 / 08

We cannot test everything, but we can test smart.

Every system is subject to risk. Manage it.

By prioritising tests based on risk, setting objective completion criteria, and respecting the resource triangle, we ensure that any reduction in time or effort does not compromise the systems that matter most.