Software Testing Is About Risk Assessment

I have an unconventional view of software testing. Let me explain.

Most people believe that the primary purpose of software testing is to identify defects. The underlying development methodology is largely irrelevant — waterfall, Scrum, XP, Kanban, etc. They all rely on software testing to verify that the software is well-behaved and to validate that the software does what it was specified to do.

My view of software testing is much broader and more strategic. The primary purpose of software testing is to evaluate risk. Identifying defects through verification and validation is simply a means of assessing risk.

The traditional testing approach

If problems or discrepancies are found, a defect report is generated and handed off to the developers for further analysis and repair. These defect reports are tracked and the software cannot be shipped out until some definition of acceptable quality is reached.

It sounds simple enough in principal though in practice many pitfalls appear. Defects are usually prioritized across three to five levels. The software must meet specified criteria such as no priority one defects, minimal priority two defects, management review of all other defects, etc.

There is a lot of subjectivity in this type of analysis. The ultimate evaluation of any defect lies with the end user. Something that appears minor to testers may actually be a blocking issue. Conversely, major defects in obscure parts of the application may not be critically important.

The risk evaluation approach

Evaluating defects one-by-one is helpful but understanding risk is more complex. Defects need to be grouped by software function. In turn, software functions need to be prioritized. Critical functions should have minimal defects. Obscure functions can be allowed greater leeway.

Ultimately, the key questions are something like, “What will our user base think of this software? Will they be pleased with it? Or will they be disappointed?” Disappointed users are a huge risk to the survival of a business.

I’ll elaborate more on this in future posts. In the meantime, Markus Gärtner wrote an interesting post called “Questions to ask during Debriefs“. In it, he explores open-ended questions to ask during debriefing of the test engineers. Take a look and let me know what you think.

Updated: April 10, 2011 — 10:35 pm