Waterfall Projects Are Always In Trouble

Project Management Solutions conducted a survey of senior executives and project managers to determine “Strategies for Project Recovery” (pdf download). What I found most interesting is the section titled “Causes of Troubled Projects”.

They came up with these top five causes of troubled projects…

1. “Requirements: Unclear, lack of agreement, lack of priority, contradictory, ambiguous, imprecise.
2. Resources: Lack of resources, resource conflicts, turnover of key resources, poor planning.
3. Schedules: Too tight, unrealistic, overly optimistic.
4. Planning: Based on insufficient data, missing items, insufficient details, poor estimates.
5. Risks: Unidentified or assumed, not managed.”

There is no mention of any specific project methodology in the report but the causes above strongly suggest to me that waterfall was widely followed. Here’s why I say that.

1. Requirements are frequently cited as a problem area on waterfall projects — and they are. Why managers continue to believe that they can fully specify every feature and function of a software system at the outset of the project baffles me. In a dynamic business environment, it cannot be done. (In static environments, it’s possible but I’ve never worked in a static environment and I refuse to do so.)

2. Resources are often under-estimated and it’s easy to see why. If you don’t fully understand the requirements, how can you possibly define the resource needs? Unfortunately, the waterfall approach demands resource estimates up front.

3. Schedules are always overly optimistic — always. Add in items 1 and 2 above and the project is in huge trouble before it even starts. Again, waterfall needs to have schedules defined at the start.

4. Planning with accuracy is impossible in light of the above. It becomes a complete waste of effort. Waterfall advocates like to plan, re-plan and plan some more. How do they get anything done?

5. Risk management is a largely overlooked area on many projects. It seems that managers just don’t want to think about risks and their potential adverse effects. Face it, most of us don’t want to plan for catastrophic illness or death. The same is true for our projects. (This problem is not unique to waterfall. Agile projects also have this issue.)

What’s the solution?

• Forget requirements and tell stories instead.
• Bring in resources as needed including using temporary workers to cover peak demand.
• Either allow schedules to flex or fix them and allow features to flex. You can’t have it all.
• Plan less, deliver more. Let the deliverables guide the planning.
• Evaluate, prioritize and mitigate risks on every project.

Be agile.

Agile approaches to software development like Scrum, XP and Kanban do not guarantee success. Nothing does. But, agile approaches reduce risk, improve quality and keep the software development and business teams in sync. It’s worth a try, isn’t it?

Android Fragmentation Is Real and Damaging

There’s a lot of talk about Android fragmentation. Google decided to restrict access to the Honeycomb (Android 3.0) source code thus adding fuel to the discussion. Is fragmentation really a problem or could it actually benefit Google?

What is fragmentation?

Software fragmentation occurs when multiple parties take software source code, customize it as they like, release products using it, and take advantage of the underlying brand name.

In the case of Android, phone and tablet manufacturers take the Android source, change it, remove parts, add components, build devices around the customized source and market the devices as “Android”. Now we have a classic good news / bad news situation.

Good News: Having many parties working with the Android code and donating changes and additions back to the code base makes Android better at a faster pace. It also creates multiple distribution paths for the software.

Bad News: Fragmentation. Confusion. Consumers see many “Android” devices but they differ in significant ways. Some Android apps don’t run on some Android devices. Some of the devices are slick and appealing while others are drab and frustrating.

I hope you can readily see that fragmentation leads to consumer confusion that will hurt the Android brand. This is why Google has chosen to restrict access to Honeycomb. They are trying to avoid the problems that occurred when manufacturers built tables using Android 2.2 despite Google’s advisory that they should not.

Open Source Is a Double-Edged Sword

Welcome to the wild world of open source. Google cannot prevent anyone from taking open source Android components and building any kind of device they want. However, all is not lost. Google has a couple of options.

1. Google can restrict use of the “Android” trademark. Anyone can build a device using Android’s open-source code but they cannot use the term Android in their marketing without Google approval. To date, Google has been lax in controlling use of the Android name. I think they have been more interested in building brand awareness than exercising control. The brand is now well-established. The game has changed.
2. A common practice in open-source applications is to have two code bases. One is completely open source and available for anyone to use. The other consists of the open-source base plus additional proprietary components. By holding back some critical elements of Android, Google gains leverage with anyone wanting to build an Android device. They would have to agree to licensing terms to gain access to the proprietary parts.

We could devise variations of these themes. The important point is that fragmentation can be bad and Google is justified in taking steps to manage the Android user experience. What’s your opinion?

Projects Should Start With a Plan Not a Story

Let’s say that you’ve just been assigned a new project. It is just the kind of project you would like to do. It looks like the business folks are on board and ready to provide support and, most importantly, funding. What now? Where do you begin?

You could start identifying the people you’ll need to get the work done. Gather a few of them together and start writing stories. This is a common approach and a really bad idea!

Regardless of the development approach you’re following — agile development, waterfall development and everything in between — you need to begin with a high-level definition of the project.

Start With a Project Plan

Project plans are all too often ignored. I’m not talking about Microsoft Project or some other way of generating gantt charts. Nor am I referring to creating a master task list.

A project plan is a high-level definition of the project and a description of the work to be done. A good project plan should answer basic who, what, when, where, why and how questions. Here is a sampling of questions that a good project plan should answer:

Who…

• are the key stakeholders? The sponsor? The customers?
• will do the work?
• is responsible and accountable for what?

What…

• is the project supposed to accomplish?
• kind of work tasks must be performed to produce the deliverables?
• resources (facilities, tools) will be needed to do the work?

When…

• are the key milestones due?
• will the work be done?
• are the key resources needed?

Where…

• will the work be done?
• will the team members be situated?
• are the end users located?

Why…

• are the stakeholders asking for this project to be done?
• is the existing solution/approach unacceptable?
• is the project needed now?

How…

• will the deliverables be produced?
• much funding has been allocated for the project?
• much time are the stakeholders and end users willing to devote to the project?

If you don’t have the answers to these kinds of questions, you can’t possibly write stories, assign tasks and begin building the software. Do yourself a favor and start your next project with a project plan. Make sense?

Research Is Not the Same as Software Design

Does agile development suffer from lack of architecture and design? Are agile solutions inferior to those developed using non-agile approaches?

Twenty years ago the answer would have been ‘yes’. Twenty years ago the software profession was less mature than it is today. Software engineering lacked standardization back then.

Patterns Rule

Today, we have patterns — architecture patters, design patterns, solution patterns, even workflow patterns. Whatever problem your team is trying to solve, it’s likely that someone has already solved a similar problem and published a pattern for the solution.

Thus, software professionals can get more done in less time by adopting a pattern and customizing it to their needs.

Solutions at the speed of change

Some will argue that every problem is unique requiring analysis and design time to craft the optimal solution. I would counter that in today’s fast-paced business world there just isn’t time to craft the ideal solution. Furthermore, business models and technology implementations are changing so fast that solutions are outdated shortly after they are delivered.

The lack-of-design argument against agile just doesn’t hold up any longer — with one exception. If your team is blazing a new trail, attempting to solve a problem that has never been solved before, you will need significant analysis and design time before development can begin.

In this situation, I’d argue that you are conducting research, not building software. Once the research is complete, why not proceed using an agile approach?

If you want a good source of software patterns, check out the Hillside Group Patterns Library.

Do you have anything to add?

Scrum’s Deep Dark Secret

Scrum has a deep, dark secret. It all started when Scrum was originally defined. One of the key goals was to empower the software developers to self-organize and take control of their destiny. This is often viewed as anti-management but I don’t believe that was the intent.

Self-organization demands that the team take responsibility for it’s work. This in turn means that the team will be held accountable for the decisions it makes and the results it delivers.

Here’s the problem. Scrum defines two powerful roles on the team — Product Owner (PO) and Scrum Master (SM). Filling these roles with high-quality people becomes a critical success factor for the team. That’s the deep, dark secret.

The Scrum Product Owner Is Powerful

The PO has qualities like the following:

• Actively involved in the project and readily available
• Knowledgeable about the software and the needs of the end users
• Empowered to make business decisions
• Collaborative, willing to discuss and negotiate
• Decisive and able to make on-the-spot-decisions

These qualities comprise a lot to ask of a single individual. They place enormous responsibility on the PO and make the PO an easy target if the project gets into trouble.

The Scrum Master Is Benevolent

Similarly, the SM has a tough role to fill:

• Responsible for adherence to Scrum principles
• Humble in the sense that the SM serves the team not vice versa
• Encourages collaboration among team members
• Able to discuss business and technical issues
• Influential without being controlling

Again, we have enormous responsibility placed on a single individual. Another easy target.

One could surmise that Scrum was designed to protect the software developers while setting up two non-developers as scapegoats should the project get into trouble. That’s not self-organizing and it’s not teamwork.

Honestly, I don’t believe that was the original intent of Scrum at all. I think the original intent has been distorted by a minority of developers trying to avoid accountability.

Scrum needs to get back to it’s roots. PO and SM are just two of the many roles on a team. The PO and SM have no more to do with the success or failure of the project than anyone else. That’s agile!

What do you think? Is there too much emphasis on the PO and SM roles?

Enterprise Solutions Don’t Have to Take Forever

Have you ever worked on an enterprise-scale project in a big company? By ‘big’, I mean one with an IT organization of several hundred people — or more. If you have, you’ve likely had the frustrating experience of sitting in a room with 10-20 people (or more), some of them senior managers from several IT departments, trying to agree on an approach to solving a problem.

By definition, those managers will have differing agendas. Their priorities reflect the functional areas that they oversee, not the needs of the project.

Oftentimes, the group reluctantly agrees to meet periodically as they talk through the issues and try to arrive at a consensus. This is guaranteed to take forever (well, it will seem that way) and is anti-agile from any perspective.

Here’s a more agile approach.

1. Start the meeting by offering an overview of the project with its key goals and priorities.
2. Quickly focus the meeting on the problem at hand — do not get pulled into tangential issues. (If pressed, agree to follow up on those tangential issues later.)
3. Give everyone a brief opportunity to ask questions and/or express an opinion.
4. Ask for help in creating an acceptable solution. (Asking for help is important in winning support for the effort.)
5. Suggest forming a core team of experts to fully assess the problem and recommend a solution. (The core team will likely have to include at least one member from each IT department represented at the meeting.)
6. Request to have the core team meet two or three times a week. This will emphasize the importance of the issue and send the message that being on the core team is a commitment.
7. Let the whole group know how you will keep them informed and when you will meet again.

This approach will result in a better solution in less time as long as you have a strong leader on the core team. Be agile, give it a try, and send me some feedback.

Take a Simple Quiz, Learn About Being Agile

James Shore and Shane Warden developed a self-assessment quiz for software development teams. It is included in their book, The Art Of Agile Development.

Most of us seem to like surveys, quizzes, and questionnaires. They often help us gather our thoughts and explore topics in greater depth. The downside is their superficial nature. It’s hard to improve your understanding of a complex topic just by answering some questions.

Negatives aside, I like having complex topics boiled down to a series of questions. They make me think and often encourage me to research a topic in more depth.

According to Shore:

“The goal of the quiz isn’t to tell you if you’re ‘agile’ or not — it’s hard enough for a human to do that, let alone something printed on dead trees — but to give you an idea of where you could improve, and what your biggest sources of risk are.”

Nicely stated.The quiz is a series of about 50 yes/no questions. After each response, try asking yourself “why” or “why not”.

Sebastian Hermida has gone to the trouble of posting the quiz online in a simple easy to use format. Just click on over to abetterteam.org. You might get more out of the quiz by having the development team take it together. The ensuing discussion might be the best thing to come out of the effort.

Slow Down; You Move Too Fast!

There’s a common misconception floating around in enterprise companies. They believe that software release/upgrade cycles have to be long. At one time, Microsoft was criticized for releasing major Windows updates too fast. (I know, hard to believe, isn’t it?) Why do they feel that way?

• It’s expensive and time consuming to roll out new software releases.
• It’s disruptive — user training, equipment upgrades, process changes, etc.
• The user community doesn’t like it — too much hassle.

Clearly, trying to introduce agile software development into such a company will be a steep uphill climb — but there is hope.

Despite these widespread sentiments, there are large companies that have adopted regular, frequent, upgrade cycles and done it with minimal cost, disruption, and hassle. They do it by controlling the number of changes in each release.

Rather than two or three year release cycles that result in drastic changes in the software, they focus on release cycles of a few months. The magnitude of the changes is much smaller and thus easily managed.

A good example of this approach is Salesforce.com.

Background: Salesforce.com is an enterprise, cloud-computing company. They started out with a CRM solution and have diversified into custom, cloud-based applications. In 2010, Salesforce.com generated over $1.6B in revenue. They have over 92,000 customers worldwide.

When Salesforce releases a software upgrade, it’s a big deal. They have 23 production and sandbox instances. Each instance consists of multiple servers. Tens of thousands of users are impacted.

If Salesforce operated by conventional wisdom, they’d do a major release about every 5-10 years. I think they’d be out of business well before then. Their actual release cycle is 3 to 6 months — and they do it with minimal problems and little disruption to customers.

Fast doesn’t have to be disruptive.

There’s more to it than just keeping the number and scope of changes small. Here are some other things Salesforce does to make frequent releases a positive experience:

• Maintain backward compatibility at the API level. Don’t break user customizations.
• Make major new features optional. Don’t force customers to take something they don’t want.
• Conduct beta testing for significant changes. Let customers try out a feature and offer feedback.
• Provide plenty of training and documentation.

Of course, there are many other enterprise companies that operate with rapid release cycles. Google is certainly the most notable among them.

There is no point in adopting agile development techniques if your IT department is on a mission to slow down the pace of change. Work with them to identify pain points and find ways to address their concerns. Being enterprise agile is more than just daily stand-ups and short sprints.

How do you feel about rapid change?

7 Steps to Bug-Free Software

No bugs! Sounds good! Is it feasible?

Can agile software developers create software with no bugs? Image if they could. Think of all the time that would be saved testing, re-testing and testing some more. All that time spent repairing things that weren’t done correctly becomes time to spend on making the software better and faster.

It is a feasible goal.

Achieving it means thinking about the process of writing software differently. Some agile teams are proponents of pair programming believing in the theory that two heads are better than one. Pair programming can help but it alone won’t get you to zero defects.

To go all the way, consider adopting all of the following techniques:

1. Pair programming or at a minimum peer review. Some form of community participation will always result in better code than can be produced by an isolated developer.
2. Test-driven development. You’ve heard it before — write the tests before the code. You need to go one step further though. Someone other than the person writing the code should write the tests. Don’t isolate them. Have them collaborate and develop a common understanding of what the code must do.
3. Define coding standards and enforce them. We all take shortcuts when we’re rushed (and we are always in a rush). Coding standards will help ensure that the code is written properly.
4. Get your customer actively involved. Misunderstandings around requirements, stories and features are common. The only way to prevent them from becoming defects is to engage the customer.
5. Allocate time to pay down technical debt. Debt is inevitable. Defects are not. The more debt you build up, the greater the likelihood that bugs will infest the code.
6. Track the source of defects that make it through. Look for patterns or common sources of mistakes. When you identify them, take steps to prevent future occurrences.
7. Lastly, automate your regression testing. A simpler and faster testing cycle will result in more testing which will uncover more bugs before the software leaves the development team. This is a last resort and the number of bugs found at this stage should be minimal.

The most important goal is to prevent bugs from reaching the customer. It won’t be easy, particularly if you are dealing with a large base of legacy code, but it can be done. Care to comment?

It’s Not Scrum Vs. Kanban; It’s Scrum And Kanban

Scrum and Kanban have a lot in common — far more similarities than differences. Scrum is more structured and disciplined than Kanban. Kanban is more adaptive and flexible than Scrum. But the major difference that I see is simply that Scrum relies on time boxes, Kanban relies on workflow control.

Scrum Time Boxes

A time box is a fixed time period. Scrum often uses 2-4 week time boxes though shorter and longer periods are also in use. One of the major tenets of Scrum is that the time box cannot be violated. At the outset, backlog items are selected such that they can be completed within the time box. Anything that is not completed moves to the next time box or may even return to the backlog.

Time boxes establish a cadence. The team and the user community become accustomed to regular software deliveries at the end of each time box. Face it, we are all creatures of habit. We like events to occur on a regular schedule.

Kanban Workflow Control

Kanban controls workflow by limiting work-in-progress (WIP). The general idea is that stories are either waiting in the backlog (inventory), being developed (in-progress), or completed. Having too much work-in-progress leads to waste, confusion and bottlenecks.

Limiting WIP provides some flexibility. It is easier to change the workflow, add new stories or re-prioritize them using Kanban than it is with Scrum. Thus, Kanban may work better in environments that are in flux — hard to predict.

Challenges

The challenge with time-boxes is that while they sound good on paper, maintaining a regular rhythm in software development is not easy. Interruptions, unforeseen problems, new requests, emergencies, etc. are all lurking nearby just waiting to disrupt the time box.

The challenge with limiting WIP is the constant attention needed to maintain control. For example, if code development gets too far ahead of system test, adjustments need to be made to restore balance. Also, the team may feel like they never get a break as the work items just keep coming.

Which is best? I like Kanban although I have to admit that we can never get away from hard deadlines — customer demos, trade shows, industry events, financial reporting, etc. Kanban doesn’t do well with these kinds of time-boxed efforts. Maybe we need Scrumban — borrowing the best elements of each.

We could employ a lengthy time box, say 8-12 weeks, and use Kanban within the time box switching to Scrum near the end in order to synchronize the final deliverables. Our ultimate goal is to satisfy the needs of the business. Don’t ever forget that.

If you want to read more about the similarities and differences between Scrum and Kanban, take a look at the PDF download called “Kanban and Scrum – Making the most of both“. Your thoughts?