Scrum Success Can Be Difficult But Scrum Failure Is Preventable

Scrum teams can fail for many reasons just like any other team, agile or not. Describing how to succeed is tough because success is often situational. Describing how to avoid failure is simpler. Let’s explore some of most preventable ways to fail.

1. Wrong people on the team especially SM or PO

Every project team needs the right people with the right attitude. If you’ve got the wrong people on the team or people with the right skills but a poor fit, the project will suffer. This is particularly true of anyone in a leadership role such as the ScrumMaster or the Product Owner. Find good people. [If you end up with a bad apple, eliminate that person from the team at the earliest opportunity. You, the team and person being eliminated will be better off.]

2. Team over-commits and loses credibility

Many of us have a tendency to over-commit. We’re generally optimists and it often hurts us. Teams need to find an equilibrium between commitment and delivery as quickly as possible. Credibility is hard to gain and easy to lose.

3. Poor definition of done or poor control over done

Much has been written about the subject of being “done”. I won’t belabor it here. Get the team to agree on what actions must occur before a team member can move from one objective to another. Then, enforce it.

4. Velocity is not tracked or not used properly

Measuring velocity is critical to understanding how much the team can get done in a unit of time. You have to measure it and you have to refer to it when planning future sprints. No excuses.

5. Skipping retrospectives or not following through

Retrospectives are a critical learning tool. How can the team improve without assessing it’s strengths and weaknesses regularly? Those assessments must be translated into actionable improvement steps at every opportunity.

6. Technical debt piles up

Every Scrum team will incur technical debt. That’s not such a bad thing as long as the team recognizes the debt and takes action to reduce it in the current or subsequent sprints. If technical dept is allowed to pile up, the project will topple. Don’t let that happen.

7. Disorganized backlog(s) and not adding defects to them

Product, release and sprint backlogs have to be managed regularly. New defects that cannot be repaired during the current sprint must be added to the product or release backlog. Ignoring these actions will result in increasing chaos over time.

8. Version control and branching mismanaged

Good configuration management practices are important to any moderately sized project regardless of methodology. CM is even more important to Scrum and other agile projects because of the frequent builds and releases. Find a good open source or commercial CMS and use it.

9. Sprints and product release cycles are too long

Sprints of more than four weeks are too long. Product releases composed of more than three to four sprints are too long. Sure, there are exceptions but these general guidelines are worth remembering.

10. Inadequate testing (too manual)

The Achilles heel of waterfall is testing — it often gets shortchanged because it’s jammed at the end of the project. The same thing can happen to Scrum teams at the end of sprints or release cycles. Don’t let it. Automate as much testing as you reasonably can and test, test, test.

10 Steps to Successfully Replacing Old Departmental Software

I see a common mistake made over and over by business and technical teams, and I think agile development can help.

In most of these situations, a business operations group is using a minimal software solution and wants to upgrade to something better. The software is often a Microsoft Access application that was quickly developed to address a need. I’ve also seen simple business applications developed in Lotus Notes and other scripting environments. In every case, someone stepped up and developed a quick and simple solution to a business problem.

After  a while, the group decides they have outgrown the simple solution. They hold a series of meetings to determine all of the features and functions needed by the organization. The attitude being … ‘while we’re in the process of upgrading, we may as well go for all the bells and whistles we might need’.

Existing software applications used by other groups in the company are reviewed. Vendors are contacted. More meetings are scheduled. And, after much discussion and analysis, a decision is made.

The new software is purchased and customized to meet the organization’s needs or the software is designed and built as a fully custom application. Either approach is fine as they both entail a lot of work.

Finally, the new software is rolled out to the user community.

They are immediately overwhelmed by all the new features. Everything is different, even the terminology. There are more menus, more fill-in fields, more picklists, more required data entries, more security — more of everything. What took less than 5 minutes to do in the old application takes 15 minutes in the new one. This is progress?

While the new software may have far more capability than the original, the users are so overwhelmed by the complexity that they have no appreciation for the enhancements.

Why does this happen?

Be honest. We all have a tendency to over-buy. Our electronic gadgetry often has far more capability than we can use. We spend the extra money and put up with the complexity because we might need one of those obscure features some day.

That’s fine but there is more at play in the business world. IT groups have a reputation for being slow to deliver. Perhaps even worse, we have a reputation for being hard to deal with after delivery.

The end result is that people often feel that once they get IT’s attention, they need to get everything they can because it will be a long time before IT is available again.

The attitude toward IT is a side effect of the waterfall development approach.

Projects often take too long and the end results are often unexpected. The next time you are faced with an upgrade situation, try the following agile approach:

1. Seek to understand the current situation. What works well? What doesn’t work at all?
2. What additional functions are desperately (yes, desperately!) needed?
3. Stop. That’s good enough to get started.
4. Go through your discussion and analysis. Select a solution path.
5. Implement or configure the bare minimum feature set required to replace the old application.
6. Deploy it and move everyone over as rapidly as you can.
7. Meanwhile, build out and configure the remaining desperately needed functions.
8. Re-deploy.
9. Finally, add any other features that have been requested and can reasonably be included.
10. Roll out the extra features over one or more additional deployments.

There will almost always be time and money constraints on the above. That’s fine. Timebox and moneybox the solution. Keep the feature set flexible as described and everyone will be much happier.

If the users can justify more time and money, you’ll get additional funding and the effort can continue. If not, the old system is replaced. The major problems are addressed. The future is brighter. That’s agile.

Themes Offer More Value Than You Think

I’m sure you’ve heard that when doing agile development, your team should start with themes (high-level objectives) and develop epics (high-level stories) that support the themes. The epics go into the product backlog. Later on, the epics will be split into small stories and then tasks suitable for implementation.

Just as epics can be broken down into stories, themes can be broken down into sub-themes. Those sub-themes can provide a goal or objective for a release or a sprint.

Simple enough (at least in theory). I’d like to focus some attention on themes because I think that themes often get overlooked or undervalued. That’s unfortunate because they can serve two powerful purposes when used effectively.

Themes help focus the development team.

Grouping a set of stories around a common theme helps the development team to coalesce around a small number of top-level objectives for the sprint. Rather than working on a discrete set of unrelated features, the team can share common objectives and be better positioned to self-organize and help each other.

I know that experienced agile teams will “figure it out” on their own. The themes will be obvious to them. Less experienced teams may not see the synergies among stories and may feel that the team is disjointed.

Themes help ‘sell’ the software.

The second area where themes add value is with the end users. It is much easier for the user community to understand what has changed in the software if you can group the changes around themes. They will be more inclined to investigate the new or upgraded features and try them out.

If the software is for sale, the marketing department will have an easier time selling the advantages of the upgrade if there are themes as opposed to random sets of changes and improvements. Themes will make a bigger marketing splash.

Everyone wins!

The next time you plan a release or a sprint, take a little extra time to group your stories into themes and see if it helps the team, the user community, and the company. Do you have any personal experience with themes that you can share?

Top 200 Agile Blogs at AgileScout.com

If you follow activity in the agile software development community, you’ve likely heard of AgileScout.com. The website is owned and operated by Peter Saddington. Peter is an independent agile coach and AgileScout is a news site for all things related to agile software development.

I’m telling you this because Peter has published a list of the Top 200 Agile Blogs. He followed that up with a list of the Top 200 Agile Blogs RSS + Twitter Lists. These are fabulous resources for anyone interested in learning more about agile development.

I’m sure it took a lot of work to compile these lists using independent ranking sources such as Alexa, Compete, Google, Klout and Yahoo.

This is not the first time that AgileScout has published useful and informative posts. Here are a few of my favorites:

• Best Agile Tools
• Essential ScrumMaster Interview Questions
• The Perfect ScrumMaster Job Description
• Top 10 Essential Product Owner Characteristics
• [Product Owner] – Top 10 Backlog Tips

By the way, BrainsLink made the Top 200 at position 163. Not bad when you consider that this site has existed for less than a year and remains a work in progress.

Thanks, Peter, and good luck with AgileScout!

[Note: I do not know Peter. We have never met. We may not always agree but the exchange of ideas helps all of us. This commentary is unsolicited and sincere. - Vin]

A Tale of a Rigid PMO and How We Prevailed

Never under-estimate the ability of people to find ways around the waterfall project, command-and-control structure. Here’s a real, personal story.

We’re working on a project to make several upgrades to a large database. It’s about a two-month effort. The PMO (Project Management Office) forces us to follow a rigid waterfall process despite my complaints and suggested alternatives. (I know from prior experiences with PMOs that arguing with them is often futile but I had to try.)

The PMO has published guidelines that allow a modified waterfall approach using overlapped phases. They even have guidelines for Scrum projects though they have corrupted Scrum so badly that it’s barely recognizable. Regardless, they refuse to compromise.

We quickly realize that the sequence of events demanded by the PMO simply cannot be done on this project. Specifically, they require that UAT (User Acceptance Testing) be done prior to production deployment.

In order to conduct a non-production UAT on a large and complex system, you need what I would call a staging environment. That is, you need a system environnment, physical or virtual, that mirrors production so that the end users can login and run some tests.

We had no such environment or anything even close to it. Creating one would have easily doubled the length and cost of the project. That wasn’t going to fly.

What could we do?

We met with the engineering and business managers. Everyone agreed that we would simply release the software into production when SQA was complete and UAT was scheduled to begin.

Of course, we didn’t tell the PMO that. Luckily, the PMO was not sophisticated enough to catch this type of “work-around”. They were happy. The users were happy. The engineers were happy. Everyone won.

Except …

This “work-around” was clearly not the right thing to do. However, imposing a rigid set of project management rules on everyone was not the right thing to do either. Two wrongs.

You have to recognize that resourceful people (and engineers are very resourceful) will find a way to get things done no matter the obstacles — technical, logistical or administrative.

Go ahead and lock down your software development process. If you do, here’s my advice. Implement plenty of checks, gates and audit trails all along the process timeline to catch those who would circumvent the process.

Resourceful people will get their jobs done even if they have to “work-around” obstacles. On the other hand…you could simply be more flexible (dare I say, agile?) and accommodate special requests.

Don’t Just Try Something New, Embrace it

I’ve helped many clients, prospects and coworkers solve difficult problems over the years. Often the contact person articulates a problem. I ask open-ended questions to draw out more information. After some back and forth, I offer a concept or two that might help. Sometimes, the response is along the lines of “We tried that and it didn’t work.”

So if I had just mentioned Scrum, XP or Kanban, I might have a tendency to abandon those as possible solutions. After all, if the business has already tried something, why would they try it again?

Yes, but…

It’s important to determine what really happened and the context within which it happened. The following questions help:

• What was the situation in the group/company at the time?
• Is the situation different now?
• What changes were implemented?
• What were the results?
• What was measured and what was observed?
• Who did the analysis?
• What specifically leads you to conclude that it didn’t work?

This line of questioning draws out what was really done and what really happened. One of two results is likely.

1) I discover that whatever was tried was not what I just proposed.

I often find that they only partially tried a new process. They cut corners, took shortcuts, didn’t get everyone to buy-in, and otherwise only half-heartedly tried to change. Of course, “…it didn’t work”.

2) The actual result was different from the perceived result.

In this case, some areas improved and some areas did not. The non-improvements may have overshadowed the positive ones. The metrics used to perform a meaningful before and after comparison may have been flawed.

This line of reasoning applies to any solution space. It could be a software development process, a technical architecture, a manufacturing process, a laboratory procedure, or even complex data analysis.

Back to agile development — don’t take the bait when someone says “We tried agile and it didn’t work.” (It may have been Scrum, Kanban, XP, Lean, etc.). Many agile approaches can work in the right context. Almost any agile approach will fail in the wrong context.

Bottom line: If you decide to try something new or make a significant change, embrace it. Make an honest attempt. Consider getting professional assistance. If you and your organization don’t make a serious effort, don’t claim you “tried that and it didn’t work”.

There Is No Agile-In-A-Bottle Approach to Software Development

I encounter it all the time — the one-size-fits-all approach to developing software. The underlying methodology can be anything from waterfall to Scrum, Lean, Kanban, XP, etc. People often fall into the “that’s the way we have to do it” trap.

These one-size-fits-all (OSFA) approaches may arise when a team is successful and wants to repeat. They try to ‘bottle’ what worked and use it over and over.

OSFA can happen in the reverse too. A team fails and wants to avoid a repeat so they bottle-up controls, checks and audits, add them to their development approach, and hope to catch problems before they spill out.

It can be seen in any size company though is most common in larger firms. It happens in regulated and unregulated industries. You see it in government and non-government environments.

Many of us are comfortable with the OSFA approach. It gives us a sense of security in that we can approach every problem in the same manner and expect similar results. The project may take longer but at least we’ll get the desired result — we hope. (In practice the end result tends to deviate from the expected simply because the problem domain is different enough to cause new and unexpected problems.)

It’s not the least bit agile.

Even if an agile approach such as Scrum, XP or Kanban is being used, making every project fit a pre-defined set of detailed guidelines is not agile. Remember that stories are a basis for ‘conversation’ — exploration, discussion, collaboration.

Stories are not detailed problem statements complete with positive and negative outcomes. The engineers and scientists among us (myself included) find this discomforting. We want neatly defined problems that we can go off and solve.

The good news is that stories are a great way to draw out details and define work effort (yes, use cases work also). Some of your story estimates will be too high and others will be too low. If the team is generating good story estimates on average, the expected results should be close to the observed results. If not, use retrospectives to self-adjust.

I find that non-technical people are more comfortable with stories and their inherent lack of rigor. The business folks generally want the technical team to handle all of the error handling, scalability, performance, etc. details. (There will be exceptions to these observations, I’m sure.)

If you truly want to be more agile, that is, responsive, adaptive and accommodating:

• Ask more questions.
• Listen more and talk less.
• Be educated and then educate.
• Make small, short-term promises.
• Deliver, deliver, deliver.

Oh, and forget OSFA. Sure it works but it takes far too long and it’s just not agile. Keep your approach simple, high-level and adaptable. Monitor a small number of key metrics and seek continuous improvement.

Without Context, You Can’t Make an Informed Decision

What’s more important, project management or system engineering?

Let’s see. Can a group of project managers build a software system without any engineers? No.

Can a group of engineers build a software system without any project managers? Yes.

That settles it. Engineers are more important than project managers.

But wait!

I hope you know that it’s a trick question. Before anyone can answer it intelligently, they would need to know a little more about the software system? For example:

• What are the major functions of the system?
• Does it replace an existing system or does it perform entirely new functions?
• Will it be built on proven components or will it be cutting edge?
• How many people will use the system?
• How much data will the system process?
• What is the rough budget estimate?
• What is the expected delivery timeline for the building the system?

I could go on but you get the idea. If the system is small and simple, any reasonably competent team of engineers can build it with little management overhead. If the system is massive in scope and scale, a large team with strong management coordination is needed.

I point this out because I think that the online debates that take place about software development are often an apples-to-oranges comparison.

For example, person #1 says that agile development works while person #2 says it doesn’t. Both are correct but they are talking about completely different situations. Person #1 is talking about a small project in a small company where everyone received agile training. Person #2 is talking about a large project in a large company where no one was trained and not everyone accepted agile principles.

The same thing happens when people debate whether agile projects need project managers or whether a sprint zero is useful. There’s something missing — context.

In the end, agile doesn’t succeed or fail. Neither does waterfall. People do.

Enterprise Agile Deliverables Are More Complex and Varied Than Just Code

I work on a variety of projects. They vary from traditional waterfall to modified (incremental) waterfall to various agile approaches including Scrum. As you can imagine, it can be difficult to keep all the rules straight and determine what artifacts, deliverables or work products are due during any given week.

[To add a little more complexity, I perform business analysis, project management, user interface design, and in my spare time, I write software and respond to user problems. I get tired just thinking about it all.]

The subject of deliverables for agile projects seems to be controversial. Some folks appear to believe that the source code and system builds are the only things that really matter. Everything else is a work product, that is, something that can be discarded at the end of the project.

For a small agile project having a short lifespan in a small company, I agree. Focus on the source code. Get the system built and move on.

For a large agile project expected to have a long lifespan in a major enterprise, a few more deliverables will be needed. The following list is food for thought.

I’ve produced most of the following artifacts over the years though I have never seen ALL of them produced for a single project. Don’t even think about doing that!

Comprehensive List of Development Artifacts

Decide which of the above are important for your project. Think about artifacts that will be short-lived (work products) and don’t spend a lot of time on them. Work products are intended to convey information needed during the development process. You will likely archive them but not refer to them after the project ends.

The deliverables are artifacts that will live on and be updated after the development team finishes its work. Examples include final source code, user docs, support docs, some technical docs, etc. The development team may not create all of these but will need to be involved in supplying information and reviewing drafts.

If you want to do enterprise agile, I believe you need to accept a broader definition of deliverable.

What have I missed? Which artifacts does your team use? Let me know.

The Scrum of Scrums Approach Lacks Definition

The concept of using Scrum-of-Scrums meetings for large projects following the Scrum approach presents major problems. Mike Cohn has written on the subject. So has Tim Bowler. I think it is more complicated than they suggest.

Let’s say you have a project with 100 people assigned. I know 100 people seems like a lot but it’s not unusual in a big company. Imagine we have an average of 5 people on a Scrum team giving us 20 teams. The teams will have to be grouped in some fashion. Imagine there are 4 groups of 5 teams.

We now have 20 daily Scrum meetings, 4 daily Scrum-group meetings and 1 daily Scrum-project meeting. Here are the problems:

1. A lot of daily meetings. Granted, worst case is that someone at the top of the hierarchy must attend all three types of meetings. To lesson the burden, it may be better to reduce the frequency of the Scrum-of-Scrum meetings. Two or three times per week may be sufficient.
2. The big picture may be lost. Each Scrum team has a ScrumMaster and a Product Owner who are experts in their areas. Who is the expert on the entire system? Who will ensure that the components developed by the teams will fit together and operate cohesively?
3. Coordination is missing. The Scrum teams cannot go merrily along developing their pieces of the solution in isolation. There must be significant coordination among the teams. How does that happen? How does everyone stay busy without wild swings in workload?
4. Vertical slicing becomes very difficult. Scrum teams are encouraged to implement features not components thus biting off vertical slices of the system. Doing so across multiple Scrum teams creates major configuration management problems. It can be done and it will be difficult.
5. Testing becomes an even bigger problem. Features and components may work properly but the complete system may not. Who tests the entire system? How are the testing efforts coordinated?

Get the idea? You may need a Scrum-of-Scrums meeting focused on software construction issues. You might need another one for testing issues. You will certainly need some kind of meeting for high-level project planning and coordination. I know — more meetings. You’ll also need to involve more people who can focus on the big picture without getting mired in the implementation details.

Running a large project (yes, it’s one project not a collection of projects) consisting of many Scrum teams is much harder than it looks. I haven’t come across anyone offering a complete and realistic way of using the Scrum of Scrums while preserving the essence of the Scrum approach. Have you?