If You Can’t Measure It, You Can’t Be Agile

Applying metrics to agile projects is a controversial topic. Much of the controversy stems from the nature of agile development. Agile approaches are intended to be flexible and responsive. Metrics seem to imply rigid goals and expectations.

Fortunately, that does not have to be the case. Metrics can be valuable tools as long as they are used appropriately. The wrong metrics can be disruptive as teams will find ways to “game the system” in order to achieve the desired result.

The key point of agile metrics is to focus on team outcomes not individual activities.

Agile metrics are important from both the “how did we do” and the “how can we do better” perspectives. They help us assess the outcomes we’ve produced (lagging indicators) and the help us visualize trends (leading indicators) so that we can improve.

Some of the common measurement metrics for an agile team include:

• Burndown charts – Actual versus expected story points
• Number of story points completed during a sprint versus those expected
• Team velocity (the number of story points completed plotted over time)
• The number of defects (e.g., opened versus closed in the sprint)
• Budget (planned value) versus dollars spent (delivered or earned value)**
• Work hours allocated versus work hours spent**
• Product backlog size
• Test coverage
• Passed tests versus failed tests
• Average story size
• Estimated story development times vs actuals
• Team capacity over time (in story points)

** These only work if the problem and solution spaces are well understood. The greater the uncertainty associated with the project, the more planned value and allocated work hours will fluctuate.

The following metrics are largely ineffective for agile development:

• Simple checklists showing what has been done
• Lines of code
• Tasks completed (tasks outstanding cannot be determined in advance outside of a sprint)
• Individual developer/tester metrics instead of team metrics
• Hours worked (versus hour expected)
• Percent complete

Factors That Affect Metrics

Not every metric applies to every agile project or even to every sprint. Various agile approaches, such as Scrum, Kanban and XP, focus on different areas and place greater or lesser value on each metric above.

The nature of the work being done will also change the metric equation. A project focused on a simple upgrade to an existing software application is very different from one attempting to build something new and unproven.

The way the team is organized makes a difference. A team composed of senior, experienced developers/testers will measure themselves differently than one composed largely of inexperienced staffers.

The characteristics of the organization also come into play. A highly structured, disciplined organization will want more metrics than a more informal one.

The team’s goals and objectives are in some ways the most important factor. If the team is attempting to improve in a given area, they will need metrics that help them monitor and adjust along the way.

A Few Final Points

Finally, keep the following points in mind as you ponder which metrics are appropriate to your projects. There is a cost associated with gathering and reporting metrics. The more complex the measurement, the more expensive it will be to track and report.

Some metrics can be employed for brief periods to help teams achieve a goal or improve a process. You don’t have to commit to measuring something long term. Use metrics situationally.

Most metrics are not comparable across teams or across projects. Most measurements are ephemeral. They are only useful for a brief period of time. (Unless you like to dig back into history and reminisce about the past.)

Keep the metrics simple and gather data at regular intervals. Simpler is almost always better (and cheaper). It’s also important to keep a regular cadence thus making the measurements more meaningful.

Finally, remember that “Working software is the primary measure of progress.”, according to the Agile Manifesto. What metrics have you found valuable…or not?