The Scrum of Scrums Approach Lacks Definition

The concept of using Scrum-of-Scrums meetings for large projects following the Scrum approach presents major problems. Mike Cohn has written on the subject. So has Tim Bowler. I think it is more complicated than they suggest.

Let’s say you have a project with 100 people assigned. I know 100 people seems like a lot but it’s not unusual in a big company. Imagine we have an average of 5 people on a Scrum team giving us 20 teams. The teams will have to be grouped in some fashion. Imagine there are 4 groups of 5 teams.

We now have 20 daily Scrum meetings, 4 daily Scrum-group meetings and 1 daily Scrum-project meeting. Here are the problems:

1. A lot of daily meetings. Granted, worst case is that someone at the top of the hierarchy must attend all three types of meetings. To lesson the burden, it may be better to reduce the frequency of the Scrum-of-Scrum meetings. Two or three times per week may be sufficient.
2. The big picture may be lost. Each Scrum team has a ScrumMaster and a Product Owner who are experts in their areas. Who is the expert on the entire system? Who will ensure that the components developed by the teams will fit together and operate cohesively?
3. Coordination is missing. The Scrum teams cannot go merrily along developing their pieces of the solution in isolation. There must be significant coordination among the teams. How does that happen? How does everyone stay busy without wild swings in workload?
4. Vertical slicing becomes very difficult. Scrum teams are encouraged to implement features not components thus biting off vertical slices of the system. Doing so across multiple Scrum teams creates major configuration management problems. It can be done and it will be difficult.
5. Testing becomes an even bigger problem. Features and components may work properly but the complete system may not. Who tests the entire system? How are the testing efforts coordinated?

Get the idea? You may need a Scrum-of-Scrums meeting focused on software construction issues. You might need another one for testing issues. You will certainly need some kind of meeting for high-level project planning and coordination. I know — more meetings. You’ll also need to involve more people who can focus on the big picture without getting mired in the implementation details.

Running a large project (yes, it’s one project not a collection of projects) consisting of many Scrum teams is much harder than it looks. I haven’t come across anyone offering a complete and realistic way of using the Scrum of Scrums while preserving the essence of the Scrum approach. Have you?

Agile Development Doesn’t Neglect System Architecture, But You Might

Agile approaches to software development are often criticized for failing to visualize the big picture from which a robust software architecture can emerge. When you are focused on small stories, short time frames, and quick deliveries, broad-based system and user design issues can easily be neglected.

But, is this a fundamental weakness of agile teams? Not necessarily. Here are a few situations where visualizing the software architecture should not be a problem:

• The software application is small in scope. The team should be able to grasp the entire system and lay it out on a whiteboard in a single design session.
• The software system is very much like something that already exists. The team can analyze or reverse engineer what exists to develop a complete understanding of the new solution space.
• The software is being built with standard components (usually based on a design pattern). Assuming the components have been tested and hardened, many architectural and deployment issues are muted.
• The team has many years of experience building solutions like the one under development. They are aware of the risks and know how to approach them.

None of the above situations will result in a project that’s easy. A project meeting such criteria will simply experience fewer and smaller risks.

Now it gets complicated.

What’s left? How about a large project, being developed mostly from custom modules, by a team with limited experience in this type of system?

Let’s agree that this type of software project will be difficult regardless of the methodology followed. Any approach can work but all approaches have an intrinsic failure risk that may not be acceptable to some teams.

For example, a waterfall approach may get bogged down in analysis. Once passed the analysis phase, the team may have to stop and regroup periodically as they learn more and realize what they missed.

An agile approach may result in rapid progress early on. However, progress could slow markedly as the system grows, defects emerge, and the backlog becomes difficult to manage.

Despite the pitfalls, I’d strongly recommend an agile approach such as Scrum, XP or Kanban especially when there are many unknowns. Agile teams learn by doing not by analyzing. Here are a few recommendations you should consider to manage the risks:

1. Know what you don’t know. That is, make a list of problems you foresee needing to be solved even though you don’t know how to solve them yet.
2. Tackle the most difficult problems as early as is practical. Don’t put them off until you have a small-scale solution built. You could be making decisions that will have to be reworked.
3. Think components. Split off software modules where feasible. Assemble separate development teams to build the modules if you can. There will be module integration issues but the overall development pace will be faster.
4. Keep your stories in epic form early on. Don’t get bogged down in defining a large number of small stories. Group the epics into themes so you can visualize how the system fits together functionally. (Build a hierarchy of ‘themes -> epics -> stories’ into your story backlog.)
5. Perform in-depth data analysis early on. Acquire some real-world data and compare it to your understanding of how the system will work. If real data is not available, simulate what you expect the data to look like. Your data model will be a critical element in building a robust solution.

Systems built using agile software development can be just as architecturally sound as those built using any other approach. Agile approaches to system development use both top down and bottom up design techniques. It’s a combination that’s proven to work.

User Stories and Use Cases – Similar But Not the Same

I often hear people equate the concepts of “Use Cases” and “User Stories”. Aren’t they pretty much the same?

Conceptually — yes. Practically — no.

Use cases, when done properly, represent a structured and rigorous approach to defining what a system will do when stimulated by external events. Use cases are broken down into scenarios where each scenario includes a detailed collection of steps showing what the actor and the system do to accomplish something of business value or process an error condition.

User stories are designed to be less rigorous. They often begin life as “epics” meaning they are high-level depictions that will need to be divided before the team can implement them. The user stories to be implemented during a sprint are very short and may not accomplish anything of business value individually. Delivering business value often requires implementing a theme, that is, a collection of user stories.

So, are “epics” or “themes” the same as use cases? Not really.

Use cases are intended to be detailed and thorough. Little is left to the imagination.

User stories are intended to be a starting point for further discussion and elaboration. They are intentionally incomplete.

Can you write use cases that are intentionally incomplete? Yes, that would be a “brief” or “casual” use case according to Alistair Cockburn. If you have a use case format that your team likes and you want to adopt an agile approach, try being brief or casual. Just resist the urge to keep going back and updating the use cases every time new information becomes available.

What about the opposite? Can you employ user stories instead of use cases? I suppose anything can work if you make the effort though in this situation the effort would be huge. Not recommended.

It’s human nature to draw parallels between concepts that appear similar to us. It’s fine to align use cases and user stories, as long as you understand the underlying differences and know what you’re getting into.

To learn more, see the use case and user story entries at Wikipedia.

Small Project Improvements May Not Be Enough

Sometimes you have make major changes, even in the middle of a project.

This post titled “How we do a retrospective” at Scrum Breakfast got me to thinking. I’m a big fan of project retrospectives or post-mortems. The general principle is to have the team identify areas that need improvement, select a very small number of them (from 1 to 5, preferably 2 or 3), and focus on those areas during the next sprint (or release or project).

It sounds good and if done meticulously can have a strong positive impact over time. There’s the potential problem — “over time”. If the project is deep trouble, or worse, the company is in danger of going under, you may not have much time.

If you face that type of difficult situation, you need to escalate the problem solving. Your focus needs to be on staying alive not just building a strong team or a great product.

Assemble a group of senior-level managers and individual contributors. Follow whatever retrospective format you are comfortable with and raise the stakes.

• Identify team strengths and find ways to take full advantage of them.
• Identify team weaknesses and either fix them fast or eliminate whatever is drawing them out.
• Reduce the scope of the project to the bare minimum required to be successful.

This won’t be easy or quick. It works best if you have a goal such as reducing the schedule by 25% or cutting the expected cost by 30%. You can’t solve big problems with small incremental steps — not when you’re dealing with a life or death situation.

An Agile Death March Is Like Any Other

Doomed projects give everyone involved a bad reputation.

All this talk of the end of the world on May 21st, made me think of “death march” software projects. I’ve been involved in a few and they are not fun. Can an agile team using Scrum, XP, Kanban, Lean or any other agile approach, completely avoid a death march?

In simple terms, a project on a death march is one that is doomed to fail — and the teams knows it. Ed Yourdon defines a death march as a project that exceeds normal parameters by at least 50%. That could mean that the schedule is compressed by 50%, the staff is reduced by 50%, the budget is half of what is needed, or the scope is twice what is reasonable.

Teams that start a project with a reasonable schedule, adequate people, ample budget and achievable scope may also become death marches over time as deadlines are missed, priorities are changed, staffing is reduced, etc. What seemed like a normal project at the outset degrades into an impossible mission over time.

The team is thus expected to work 12-14 hour days, six days a week to complete the project. Is this unreasonable? Yes. Does it happen? Yes. Can it happen to an agile team? Yes.

Don’t let emotion rule.

Death marches usually occur when a company’s management panics. Emotion governs decision-making. There may be a hidden belief that if you push people hard enough, they’ll rise to the occasion. (And they will if the potential reward is big enough.)

Some people even like death march projects. There can be a thrill factor as you strive to do the impossible and become a hero. The heroics are fine in short bursts to meet a tactical deadline but they will wear you down over time.

If you find yourself assigned to a death march project from the outset, speak up. You need to get everyone on the team to recognize the futility of launching the project with the current parameters. Then, the team needs to approach management and educate them on the problems faced by the project.

If your project has degraded into a death march, your team needs to stop and replan. Everything should be re-evaluated — schedule, people, budget, scope, technology, design, etc. The project needs a new beginning.

New project or existing one, agile or not, the team has to provide options to company management. What are the alternatives? What can be accomplished within the given parameters? What should be sacrificed?

Speak up. Be prepared to listen and compromise. Don’t just march along to the inevitable.

How to Get a Scrum Project That’s in Trouble, Out

So, your team is using Scrum and it’s not working. Symptoms include the following:

• Sprints are not delivering the stories as promised.
• Sprints are taking longer and longer.
• The number of defects found during testing is high and there’s no time for fixes.
• Morale is low as the team struggles and becomes argumentative.

The natural inclination of many managers is to revert to command-and-control. Identify the features, functions and software modules being developed. Focus on function and module delivery — not stories. Set deadlines and hold the team accountable.

Sounds like a reversion to waterfall, right?

Yes, it is and it’s the wrong approach. Here’s what should happen:

1. Identify why the sprints are not delivering the stories promised. It’s likely to be because the stories are too big and/or they were not sized accurately. Solution: Reduce the story sizes and rework the associated task estimates. You’ll also need to re-prioritize the backlog.
2. Examine the reasons for the sprints taking too long. Sprints should not be arbitrarily lengthened. When they are scheduled to end, they end. Taking more time is an admission that something is fundamentally wrong. More time will not fix the underlying problem. Solution: Stop. Re-group and re-plan. Empower the team to self-organize. Making all the sprints slightly longer may help but do not exceed 30 days.
3. A high defect rate is symptomatic of poor quality source code and/or inadequate testing. Neither is acceptable. Solution: Insist on code reviews and/or pair programming with an emphasis on quality. Also automate as much of the testing as you reasonably can to shorten the test cycles.
4. Poor morale results from frustration. Find out why the team is frustrated. Solution(s): If there is a “bad apple” on the team, remove him. If the team does not buy into the goals and objectives for the project, listen and make changes. If the team lacks tools, skills, etc., find what they need and make it happen — or eliminate the need entirely.

Reverting to the old (waterfall) way of doing things only preserves and protects what you were trying to change in the first place. It may result in a short term improvement but will make things worse in the long run.

Agile development in the form of Scrum, XP, Kanban or anything else is not easy. Agile approaches may be simpler in some respects than waterfall, but they are not easy. Keep trying. Other ideas?

XP’s Pair Programming Is Harder Than It looks

The agile development practice of “pair programming” endorsed by advocates of eXtreme Programming (XP) is controversial. On the surface, it seems that having two programmers work side-by-side sharing a keyboard will cut productivity in half. (It surely will if you take this to the limit and stuff two programmers in every cubicle. But at least you’ll need less office space and fewer computers, right?)

The counter-argument is that the quality of the code will improve resulting in fewer defects and less time spent fixing them. In addition, communication and knowledge-sharing will improve. Less rework and less refactoring should improve long-term productivity even if there is a short-term reduction.

Perhaps the larger problem is that some software developers simply don’t like pair programming. Think about it. Many developers are introverts. They like solo activities. They take pride in their work and seek peer recognition for it. Let’s find a middle ground. You don’t have to fully adopt pair programming to practice XP or any other form of agile development.

Pair programming can be a tough sell.

You may find that some of your developers like it and some don’t. That’s okay. It doesn’t have to be an all or nothing approach. Encourage those that want to pair-program to do so. Let those that don’t continue to operate solo. They may change their minds once they see how others benefit.

Try using pair programming in special situations where there is an easy sell. For example:

• It’s a great way to train new hires. They get to sit with an expert team member, learn, ask questions and dig into the code.
• It helps when implementing new technologies. The developers can help each other to understand the environment and accelerate learning for both of them.
• It’s a good way to begin development using a new custom design or a design pattern. They can jointly dig into the new system and help each other understand it.
• It may help with portions of the code that are high risk, poorly understood or (execution) time-critical. This is the “two heads are better than one” principle.

These are situations where pair programming can be used for a brief period of time to solve a problem. Some developers may grow to like the approach and want to continue using it. Others will want to go it alone.

When using pair programming, it’s a good idea to mix up the pairs from time to time. This improves cross-training and knowledge-sharing. Also, pair programming does not have to be used all day, every day. Allow the developers some flexibility to determine what works for them.

One more approach is worth considering.

There is one more variation on the pair programming approach that’s worth considering. Try having a software engineer and a test engineer work side by side. One can write the base code while the other writes the test cases. This should be more reliable than having the software engineer write both the code and the tests.

Admittedly, this is not strictly pair programming because the engineers are not working on the same code elements; they are working on related elements — but it’s still worth a try.

If you are not doing pair programming, I hope you employ design and code reviews as ways of improving software quality — because developer isolation is not a critical success factor. What do you think?

Good Engineering Is the Primary Objective

When it comes to software development, agile or otherwise, the predominant emphasis is on management disciplines. We write about planning, tracking and delivering the software. But what about writing the software?

In the end, the quality of what the software engineers write determines the outcome. I’ll go so far as to say that good software engineers can overcome incompetent managers and still deliver quality software. The reverse is not true, however. Good managers cannot overcome poor software engineering, at least not during the course of a single project.

(That’s why I’ve always advocated hiring the best software engineers available, not necessarily the best technical fit; but more on that another time.)

That said, I’d like to call your attention to the Manifesto for Software Craftsmanship. It is an adjunct to the Agile Manifesto not a replacement of it. I encourage you to read and sign it. I’ve quoted it here:

“As aspiring Software Craftsmen we are raising the bar of professional software development by practicing it and helping others learn the craft. Through this work we have come to value:

Not only working software, but also well-crafted software;

Not only responding to change, but also steadily adding value;

Not only individuals and interactions, but also a community of professionals;

Not only customer collaboration, but also productive partnerships;

That is, in pursuit of the items on the left we have found the items on the right to be indispensable.”

Regardless of the software development approach — Kanban, lean, Scrum, waterfall, XP, etc. — there is no substitute for high-quality engineering. Managers, analysts, owners, stakeholders, etc. can add value, as long as they maintain focus on the core objective of good engineering. Do you?

BTW, the site includes a world map showing the location of all of the people that have signed the manifesto. The Internet truly makes us a worldwide community.

7 New Ideas for Breaking Down User Stories

There are many books and articles about creating user stories when using an agile development approach like Scrum, XP or Kanban. The standard techniques focus on technical content and process flows when dividing high-level stories into smaller and smaller chunks. This post approaches the topic from a different perspective — that of the business.

I hope we agree that small stories are better. They are easier to understand, simpler to implement, and reduce the project risk. That said, it can be difficult to break up large functional elements of the software into small stories.

As you go through the process of evaluating, prioritizing and dividing stories, here are some critical items to consider:

• Identify obstacles to splitting the stories and eliminate them. If the same excuses for not splitting stories keep coming up, deal with the excuses.
• Don’t get caught up in design issues. There is a tendency to want to keep the system design elegant. That’s fine but building temporary logic and interfaces is okay too.
• Seek out stories that produce results having business value. Don’t just focus on completing tasks and generating code.

There are many ways to approach stories.

Stories tend to evolve around the user interfaces, the business processes, and the data flows managed by the software. Those approaches are fine and you should also consider the following unconventional ways of compiling stories:

1. Consider pre-delivery deadlines such as sales demos, trade shows and beta tests. These are often important revenue-generating events that will impact how you prioritize and implement stories.
2. Divide by stakeholder needs, for example, sales and marketing before manufacturing. You may find it simpler to focus on the needs of a stakeholder group rather than trying to please everyone.
3. Divide by target user attitude delivering to co-operative users first. If you find a user group or even a few individuals who are interesting in the software, target their needs first. You’ll get more and better feedback.
4. Deliver within a limited geographic area to avoid time zone, date, currency, language and infrastructure issues. Geographic location can introduce many complexities to the development effort. Keep the distribution area small to start.
5. Identify high-value user activities and deliver those sooner. The software is likely to have many features and functions. Work with the business organizations to find the high-value ones and focus on those first.
6. Identify high-risk software elements and implement those first. It’s natural to work on the simple things first though it’s the complex ones that carry the greatest risk. Get those out of the way early in the development effort.
7. Separate power-user or advanced features from standard ones. Software often has advanced features or configuration options. They can usually wait until later in the development cycle.

Breaking down user stories into manageable chunks is not easy. Hopefully, these approaches will give you some additional ideas. Do you have any ideas to add?

Earned “Value” Is a Fallacy

It’s not earned; it’s accrued. It’s not value, it’s cost. Big difference.

There are many proponents of the concept of “earned value” for software projects. Many of them are consulting companies and those that work on US government projects. I think earned value management (EVM) is bunk. The concept has merit but the implementation is fatally flawed.

EVM is a way to measure software project progress by comparing the amount of money spent to date to the amount projected and to the work completed. If you are spending less money than planned for the work being delivered, you are under budget. If you spend more money than planned for the work output, you are over budget.

You could be ahead of or behind the scheduled time. EVM’s focus is budget not time or quality so you’ll need to determine schedule and quality status using other metrics.

You can see why the EVM concept appeals to consulting firms. They want credit for the work completed, that is, value delivered, so they can send invoices and receive payments. EVM works great for them. What about the rest of us?

“Value” and “Cost” are not the same thing.

Problems begin with the term “value”. Many software applications have little or no value until they are largely complete. At times, they have no value even then due to changes in the marketplace or the underlying technology.

Whether real value is created or not, costs are accrued. The project cost keeps going up regardless of the value of the deliverables. Measuring cost is comparatively simple. Measuring value is guesswork.

Another problem is the degree of done or the level of completeness. Software components are designed, written, tested, integrated, verified, validated and accepted. How much value is accrued at each stage? Truth is, there is no repeatable way to know. If you wait until the ‘accepted’ stage, the earned value curve will have a classic hockey stick shape. That’s useless.

Projects accrue value above or below the accrued costs. Individual components of the software accrue value in various ways. Some components accrue high initial value while others accrue value more slowly over time, if at all.

Software value is not the same as cost. Tracking projects costs is a good idea. Estimating earned value is a complex and flawed means of measuring project progress. Don’t rely on it. Have you used EVM?