Software Outcomes Matter, Test Results Don’t

How do you determine the success (or not) of a software project? How do you measure the final outcome? How do you know if the project delivered what was expected?

You might be inclined to refer to the number of remaining known defects, the number of test cases, the improvements made in the code, the number of new features added, the feedback from the Product Owner, etc. Sadly, those are all the wrong metrics and don’t measure business outcomes.

What matters is end user perceptions and business value, not internal software metrics.

Real people, conducting real business with the software is the only true measure of outcomes. Here are a few real world examples where software teams thought they did a good job but the users disagreed. (I’ve avoided specifics to protect the guilty parties.)

  • The software application functioned perfectly but was too slow to be useful by the large, target group. (Performance testing might have caught this.)
  • The software worked properly but required too much manual data entry. (Useability testing might have caught this.)
  • The software ran into a conflict with another desktop application meaning some people could not use it. (Additional analysis and testing of the target environments might have caught this.)
  • The software worked fine but the new workflow actually took longer to get things done than did the old workflow. (Again, useability testing might have caught this.)

I’ve used the phrase “might have caught this” above because there is no guarantee that any laboratory testing will catch any particular complex failure scenario. Testing will catch the major problems and the expected ones. The subtle environmental issues are another matter.

This leads me to a more central question. If the ultimate measure of successful outcomes is the users’ perception, why aren’t teams doing more user testing? It’s not a simple thing to do. I get that. But, clearly we have to try harder. Here are a few suggestions.

  • If you’re doing straight beginning to end waterfall, stop! You need to introduce interim deliverables. Break up the waterfall flow into a series of mini-waterfalls that each deliver useful functionality. Or, even better, dump waterfall entirely and switch to Scrum, Kanban, Lean or XP.
  • If you’re already doing waterfall iterations, get the software into the hands of the end users sooner and more often. Draw them into the process. Engage them. Setup a small number of systems where users can stop by anytime and try out the latest build. Have lunch and learn sessions. Get creative. You need real feedback from real people.
  • If you’re already using an agile development approach, focus more effort on the needs and concerns of the end users. It’s not about technology. It’s about them. Empathize more and they will respond in kind.

Recognize that software is a tool not an end game. Real people use software to accomplish something that adds value to their job functions or enhances their personal lives. Laboratory test environments can’t replicate every real work condition. There are too many variables.

Updated: May 13, 2012 — 9:32 pm

1 Comment

Comments are closed.