The Dark Side of Metrics: Be Careful What You Wish For

diceMany managers are fond of using metrics as a way to evaluate the performance of software developers. This can be done both at the team and individual levels. However, most metrics can be “gamed”, that is, manipulated to show the desired result.

Some readers are likely thinking that their teams would never “game the system”. It’s true that most teams would never openly conspire to change a metric result. However, in the end, people will do whatever is needed to protect themselves, their teammates, and their families. If management places the team under severe and unreasonable pressure to improve a metric, each member of the team will seek out ways to improve.

Here are a few examples of metrics that are fairly easy to improve:

  •  To increase team velocity, increase the size of user story estimates.
  •  To improve the pass/fail test-case ratio, avoid testing boundary conditions.
  •  To reduce the size of the backlog, put fewer stories in it.
  •  To produce more lines of code, write sloppy code.
  •  To complete more tasks, divide the work effort into smaller tasks.
  •  To reduce time-to-market, write less software and/or reduce testing.
  •  To reduce your development costs, take on more technical debt.

These tactics may appear dishonest but it’s a matter of perspective. If I make a demand that most people view as unreasonable, I should expect that those people will satisfy the demand using equally unreasonable tactics. Fair is fair, right?

Focus on the right things.

One way around gaming is to track multiple opposing metrics. For example, you could track velocity and average story size over time. If velocity improves while story size increases, something is out of balance. But what problem have you solved? How has the company benefited?

In the end, metrics are internal measures of the past. They tell us how we did recently versus how we’ve done historically. The problem is that our customers don’t care. They want software delivered as promised; software that does what they need; software that works well; software that’s a delight to use.

The only valuable metric is your customer’s opinion of the software. That’s why the principles of agile software development make so much sense. Focus on customer collaboration and working software. Focus on the future. Use metrics sparingly to help identify and correct team weaknesses.

Metrics have their place but not as the team’s focal point. They are tools not outcomes.

photo credit: jcoterhals via photopin cc

Updated: April 6, 2013 — 8:35 pm

3 Comments

  1. agree. nice thoughts.
    But what is best to be done to get real value from the metrics? How to make them game secure?

    1. I don’t believe that any individual metric can be completely secure. It’s important to use a variety of metrics but not all at once. Mix it up. Capture different information at different times. And never judge team (or individual) performance based a a single metric. After all, metrics are tools, not results.

      1. I agree that metrics are more a trigger to think about what an issue is, or a potential issue that might occur, than a triiger to act without thinking.
        But I still think it’s worth showing that metrics change and discuss it with a team (without blaming anyone of course). Metric results review I think, are a good point to start discussion.

Comments are closed.