Testing Survival Means Knowing What to Test

Does every feature and function in the software need to be 100% tested and approved? I don’t think so.

Firstly, let’s split testing into two major domains:

1. Verification: Does the software do what the specifications and the developers intended it to do?
2. Validation: Does the software do what the business wants and needs it to do?

While these concepts are intimately related, they are also independent. In the case of #1, the software may reliably do exactly what the developers intended but they may have misunderstood what the business wants — fail.

Conversely for #2, the software may do exactly what the business wants but that may not be enough to form a complete solution. This is a little harder to visualize so here’s an example. The business wants to be able to search for customers that meet defined criteria and see a list. Fair enough.

What happens if the search delivers 1,000,000 results? Will the user’s computer be unavailable for an extended period of time while the request is processed? What will happen if the user cancels the request or her computer freezes? You get the idea.

The business is responsible for defining the results they expect. The developers are responsible for explaining the many possible outcomes and working with the business to determine how to handle them.

Okay, back to the original question — test 100% of everything?

Ideally, the development team would have the people, equipment and time to thoroughly and completely test everything. I don’t know about your situation, but no project I’ve ever worked on had such luxury. There never seems to be enough time even if we’re lucky enough to have people and equipment available.

This problem is often worse on waterfall projects where the bulk of the testing takes place after the code is written and bench tested. QA tends to get squeezed between late software delivery and a fixed deployment date.

Agile projects such as those using Scrum, XP or Kanban are somewhat less susceptible to this problem because testing takes place during every sprint (iteration). However, even agile projects often squeeze the testing time as the code is rushed to completion and the sprint deadline looms.

The solution lies in prioritizing features and functions — ideally via stories. Following this, prioritize failure scenarios. Test the high-priority (i.e. high-visibility) stories well. Test the most likely failure scenarios well. Then, as time allows work through the remaining stories and failures testing broadly but not always deeply.

Is this a ‘best practice’?

Of course, not. It is a survival practice. Do the best you can with the time, people and tools available — and be proud of it. Make sure you openly communicate to senior management how the testing was conducted and what the risks are in shipping the product as is.

Testing software, just like writing software, is not a perfect science. You do your best, evaluate the risks, and make a business decision as to what to do next. Do you have anything to add?

Make It Simple and Fast Then Refactor for Better Software

Refactoring is a controversial topic within agile software development. There is no clear and simple definition of “refactor”. Martin Fowler wrote an entire book of over 400 pages on the subject called “Refactoring: Improving the Design of Existing Code”. You can find it on Amazon.

The generally accepted definition of refactoring goes something like this: changing the internal structure (or composition) of the software without changing the externally observed behavior (or interface).

That’s a pretty broad definition leaving much open to interpretation. Let’s examine some areas that are valid refactoring activities and some that are not.

Here are some areas that are covered by refactoring:

• Cleaning up software source code to make it simpler or more easily maintained.
• Adding test conditions to the software to help with automated regression tests.
• Breaking up the code into modules.
• Replacing an algorithm with one that is more suitable.
• Changing data structures to make them more cohesive.

Conversely, here are some areas that are not considered refactoring:

• Fixing defects.
• Implementing new error handling logic.
• Adding code for logging and tracking purposes.
• Adding a feature that was inadvertently left out of a sprint.
• Changing the code because the business has requested something new.

Refactoring is an important activity for agile teams. Just because code needs to be refactored doesn’t mean it was poorly written. Software developers routinely deal with incomplete information and tight deadlines — also known as uncertainty.

When faced with uncertainty, the most agile approach is almost always the simplest and quickest one available. Why spend a lot of extra time when you don’t know all the facts or you expect changes to be made? Do it simple and fast then refactor as needed.

The best agile teams understand the importance of refactoring regularly. Just don’t get carried away. Refactoring the same area of the software over and over is a signal that something is wrong with the stories, the design or the developer. What are your thoughts?

Product Owner By Proxy

Sounds like a legal loophole but it’s a valid agile approach.

The business stakeholders won’t assign a Product Owner to your agile development project. Now what? Can you implement Scrum, XP or Kanban without a Product Owner (PO)?

This is a common occurrence and there are insidious variations. At times, a PO is assigned but he is powerless, clueless or useless. It’s hard to be agile when the PO isn’t.

Product Owner by Proxy

The alternative is for the business stakeholders to appoint an agent (or proxy) to act on their behalf. The agent can be anyone in the organization as long as a few conditions are met. The agent must:

1. Have a good understanding of what the business needs from the software.
2. Be familiar with the underlying business processes that will use the software.
3. Understand who the users are and how they will interact with the software.
4. Be respected by the business stakeholders.
5. Be empowered to make decisions on behalf of the stakeholders and users.
6. Have the respect and trust of the software development team.
7. Be able to balance features, costs, time and quality to achieve an optimal outcome.

PO by proxy isn’t an approach I’d strongly recommend. It should only be used when the business stakeholders balk at assigning a dedicated representative.

The agent is often a member of the IT or engineering organization — usually a business analyst. This creates an obvious conflict of interest. It’s been said that you cannot serve two masters. The agent is paid by IT/engineering but serves the business. It can work but there’s a very fine line to straddle.

Give it a try if you must. And one more thing — the agent has to be a good diplomat. There will be lots of negotiating, convincing and compromising.

Good Team Communication Takes a Lot of Work

No software development approach is perfect. Consider the use of daily stand-up meetings. These are popular when using Scrum, XP, Kanban, or almost any agile approach.  A quick daily meeting can improve team communication and keep everyone in sync. But…

The team members need to communicate throughout the day. There is often a tendency to wait until the next daily meeting to share information. There may even be a feeling that most or all team communication can be handled during the daily stand-ups. Why can’t the rest of the day be interrupt-free?

You don’t need more than one team meeting per day but you need to make it easy for team members to talk or exchange information at any time. They can use any or all of the following:

• Face to face conversation
• Telephone
• Instant messaging
• Private social networking software
• Electronic team message board

It’s good to have a few options available so that people can use whatever form of interaction makes them comfortable. The simpler and faster the method, the more likely people will be to use it. The more comfortable they are using it, the more information they will provide.

Don’t under-estimate the time and effort required for teams to communicate well. Even a small team of 7-9 people will have to spend a lot of time sharing information. Make it as easy as you can.

That Stockpile of Things To Do Will Come In Handy

To do Scrum well, does everyone on the software development team have to 100% busy for every sprint? Think about it. A ‘yes’ answer might mean everyone on the team has to have assigned tasks that keep them fully occupied for the duration of each sprint. That’s a lot harder to do than it looks.

I think the Scrum concept of fixed-length sprints scares many a manager. The thought of constantly planning, allocating and verifying in order to keep everyone busy can be daunting — but is it necessary?

Firstly, we have the related concept of self-organizing teams. Part of being self-organizing is empowering the team to manage their workload — as a team. If one member is overloaded, another member who is more lightly loaded should be able to jump in and help out.

Self-organizing is self-balancing.

Also consider that every project has an unofficial stockpile of things that should be done as time allows. They are not critical items. They don’t have an impact on the project’s goals. But, it would be better to get them done than not. What are they?

• Code reviews
• Refactoring
• Infrastructure items like software upgrades, server maintenance, file system cleaning, etc.
• Education and training
• Servicing technical debt
• Deep dives into the environment to better understand it
• Documentation

You get the idea. There is always plenty of work to do that may not be critical in the short-term but will be invaluable in the long-term.

So don’t agonize over precisely aligning everyone’s workload with the sprint timeline. Rely on the Scrum team to self-balance. When imbalances appear that don’t lend themselves to re-balancing, take something from the stockpile and get it done.

Agile Techniques Work in System Maintenance

I know a group of database administrators that pretty regularly messes things up. They perform system upgrades, new software installs, server migrations, etc. and often get them wrong. For example:

• Work takes much longer than anticipated.
• Software applications are impacted that they never expected.
• Things go wrong that they never considered.

To make matters worse, they typically don’t bother to notify anyone that maintenance work will be performed and some disruption is possible. Is this kind of stuff normal where you work?

The key question for this post is, “Can agile techniques improve this situation?”.

Let’s explore how this group might employ agile techniques. What would they do and how would it help?

1. Team Assignments. The team responsible for the system changes is assembled and roles defined. Everyone knows their responsibilities.
2. Initial Planning. The team meets and crafts stories around the system changes to be made. The stories do not define every task to be done but, rather, lay out the major goals and associated activities.
3. Detailed Planning. The team meets daily leading up the system changes. During the daily sessions, details of the work to be done are defined and assigned to the responsible DBA.
4. Communications Planning. In parallel, the team identifies all of the groups that use the system(s) to be changed. A communication plan covering expected and disaster scenarios is crafted.
5. Deployment Tasks. Finally, a deployment task list is generated. It defines all of the work to be done, who is responsible, how long it will take, what inter-dependencies exist, and how it will be verified.

The only part left is actually making the changes.

The steps outlined above may seem obvious (and they are!). The key point in all of this is assembling a team. By getting a small group together and facilitating group discussion, all of the tasks and risks are rooted out. Everyone on the team learns from everyone else.

This process works far better than relying on the one or two people who are normally assigned to the target systems to ‘just take care of it’. Don’t you think?

Software Testing Is About Risk Assessment

I have an unconventional view of software testing. Let me explain.

Most people believe that the primary purpose of software testing is to identify defects. The underlying development methodology is largely irrelevant — waterfall, Scrum, XP, Kanban, etc. They all rely on software testing to verify that the software is well-behaved and to validate that the software does what it was specified to do.

My view of software testing is much broader and more strategic. The primary purpose of software testing is to evaluate risk. Identifying defects through verification and validation is simply a means of assessing risk.

The traditional testing approach

If problems or discrepancies are found, a defect report is generated and handed off to the developers for further analysis and repair. These defect reports are tracked and the software cannot be shipped out until some definition of acceptable quality is reached.

It sounds simple enough in principal though in practice many pitfalls appear. Defects are usually prioritized across three to five levels. The software must meet specified criteria such as no priority one defects, minimal priority two defects, management review of all other defects, etc.

There is a lot of subjectivity in this type of analysis. The ultimate evaluation of any defect lies with the end user. Something that appears minor to testers may actually be a blocking issue. Conversely, major defects in obscure parts of the application may not be critically important.

The risk evaluation approach

Evaluating defects one-by-one is helpful but understanding risk is more complex. Defects need to be grouped by software function. In turn, software functions need to be prioritized. Critical functions should have minimal defects. Obscure functions can be allowed greater leeway.

Ultimately, the key questions are something like, “What will our user base think of this software? Will they be pleased with it? Or will they be disappointed?” Disappointed users are a huge risk to the survival of a business.

I’ll elaborate more on this in future posts. In the meantime, Markus Gärtner wrote an interesting post called “Questions to ask during Debriefs“. In it, he explores open-ended questions to ask during debriefing of the test engineers. Take a look and let me know what you think.

Focus on Core Agile Principles, Not Little Stuff

Here’s a suggestion.

One of the best practices for Agile development teams is to conduct regular retrospectives. The goal is continuous improvement by drawing attention to team activities that could be done better.

What if you did the opposite? Focus on the core strengths of Agile development instead. Here is an excerpt from the “What is Scrum” page on the Scrum Alliance website…

“We value

• Individuals and interactions over processes and tools
• Completed functionality over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

That is, while there is value in the items on the right, the items on the left matter more.”

Try focusing on these four points in your next retrospective. Ask yourself and your team the following questions:

• How well does the team communicate internally? Do they share information? Do they help one another? If teamwork is lacking, what’s wrong? How can the team work better together?
• Does the team deliver working software during each sprint? If not, what’s distracting them? If they are not writing software, what are they spending time on?
• Is the customer or end user actively engaged in the development effort? Is there collaboration among the technical and business teams? If not, what can you do to improve it?
• Does the team readily accept changes (not within a sprint, perhaps, but between sprints)? If, not what’s stopping them?

To get the full benefits of agile and Scrum development, you need to take full advantage of their core strengths. Too many teams spend too much time trying to improve small details like updating the Scrum board. Yet, if you can’t master the core principles, the little stuff won’t make any difference.

It’s generally a good idea to focus on your strengths, not your weaknesses. Employ that principle and try focusing on agile development’s strengths. You can always work on the little stuff later.

Research Is Not the Same as Software Design

Does agile development suffer from lack of architecture and design? Are agile solutions inferior to those developed using non-agile approaches?

Twenty years ago the answer would have been ‘yes’. Twenty years ago the software profession was less mature than it is today. Software engineering lacked standardization back then.

Patterns Rule

Today, we have patterns — architecture patters, design patterns, solution patterns, even workflow patterns. Whatever problem your team is trying to solve, it’s likely that someone has already solved a similar problem and published a pattern for the solution.

Thus, software professionals can get more done in less time by adopting a pattern and customizing it to their needs.

Solutions at the speed of change

Some will argue that every problem is unique requiring analysis and design time to craft the optimal solution. I would counter that in today’s fast-paced business world there just isn’t time to craft the ideal solution. Furthermore, business models and technology implementations are changing so fast that solutions are outdated shortly after they are delivered.

The lack-of-design argument against agile just doesn’t hold up any longer — with one exception. If your team is blazing a new trail, attempting to solve a problem that has never been solved before, you will need significant analysis and design time before development can begin.

In this situation, I’d argue that you are conducting research, not building software. Once the research is complete, why not proceed using an agile approach?

If you want a good source of software patterns, check out the Hillside Group Patterns Library.

Do you have anything to add?

Enterprise Solutions Don’t Have to Take Forever

Have you ever worked on an enterprise-scale project in a big company? By ‘big’, I mean one with an IT organization of several hundred people — or more. If you have, you’ve likely had the frustrating experience of sitting in a room with 10-20 people (or more), some of them senior managers from several IT departments, trying to agree on an approach to solving a problem.

By definition, those managers will have differing agendas. Their priorities reflect the functional areas that they oversee, not the needs of the project.

Oftentimes, the group reluctantly agrees to meet periodically as they talk through the issues and try to arrive at a consensus. This is guaranteed to take forever (well, it will seem that way) and is anti-agile from any perspective.

Here’s a more agile approach.

1. Start the meeting by offering an overview of the project with its key goals and priorities.
2. Quickly focus the meeting on the problem at hand — do not get pulled into tangential issues. (If pressed, agree to follow up on those tangential issues later.)
3. Give everyone a brief opportunity to ask questions and/or express an opinion.
4. Ask for help in creating an acceptable solution. (Asking for help is important in winning support for the effort.)
5. Suggest forming a core team of experts to fully assess the problem and recommend a solution. (The core team will likely have to include at least one member from each IT department represented at the meeting.)
6. Request to have the core team meet two or three times a week. This will emphasize the importance of the issue and send the message that being on the core team is a commitment.
7. Let the whole group know how you will keep them informed and when you will meet again.

This approach will result in a better solution in less time as long as you have a strong leader on the core team. Be agile, give it a try, and send me some feedback.