The headline of this post pains me but it’s true. Real agile approaches to software development will never replace the old and outmoded command-and-control approaches based on waterfall. Scrum, Kanban, XP, Lean, etc. will augment and improve waterfall approaches but never replace them. Here are two reasons for my pessimism.
The Safe Choice
Have you ever heard the old saying, “No one ever got fired for buying IBM”? It comes from a time when IBM was the king of the enterprise computing industry (some will argue that they still are). If an IT manager purchased equipment from IBM, he was making a safe decision. Even if the equipment didn’t solve the target problem, it would be assumed by many that no vendor could have provided a solution. After all, the manager bought from the best, right?
Today, companies don’t tend to dominate markets the way IBM did largely due to anti-trust laws. However, the same logic is true when it comes to any market leading vendor such as Apple, Cisco, Google, Microsoft, Symantec, etc. When you buy from the (perceived) best, you get a layer of protection from criticism.
Waterfall approaches to software development are the market leaders. They are widely used and accepted. Managers won’t be criticized for following them. Agile approaches carry more unknowns and more risks. Why would any conservative manager even consider an agile approach?
Protected By Procedure
My second rationale is more complex. It has to do with our penchant for following the rules. Most companies place plenty of rules around their software development efforts. They implement phases, gates, approvals, documents, hierarchies, reviews, standards, etc. to control every aspect of software development. Those controls aren’t inherently bad but become evil when carried to extremes.
The problem is that many people like extremes. Many managers like to tightly control how things get done. Many subordinates are eager to follow the rules. After all, you can’t be fired for following the rules, right?
Lots of rules and procedures provide a safety net. Everyone knows what they are supposed to do and how to do it. If anyone encounters a situation that’s not covered by the rules, they seek assistance so that the management team can make a decision. It’s safe and conservative. Agile approaches have few rules or procedures and relatively flat hierarchies exposing everyone to new risks. Why would any software developer even consider an agile approach?
Culture Trumps Process
These are high hurdles stemming from corporate cultures that demand conformance. If you follow the rules and make conservative decisions, you’re granted a layer of protection. If you bend the rules or make independent decisions, you could be reprimanded — even if you made the right decision.
In some organizations, the pain of the status quo will overpower the need to conform. When the software is consistently late, over-budget and crappy, management will sometimes be willing to adopt a completely new approach. That gives agile development a foot in the door.
Unfortunately, I’ve found that many organizations are willing to put up with a lot of pain to protect the status quo — self-preservation is more important than delivering good software. When the software development process battles with the corporate culture, the culture wins — every time.
Can agile software development ever replace waterfall development as the primary approach for software projects? I’m skeptical. Perhaps we’ve gone as far as we can go and it’s time for the agile community to move in a new direction. Any ideas? Your opinion is as good as mine so please leave a comment.
Hi Vin
First of all, it’s time for everyone including agile proponents to stop talking about waterfall. It’s a strawman.
What I take by what you say as waterfall, as the most degenerate form of a sequential process, it just doesn’t exist in the real world, outside of the degenerate cases.
Now in terms of agile, it is the new degenerate process. Just like the degenerate form of waterfall is harmful, so is the degenerate form of agile.
It is now the safe choice to let some soft skills BA (now labeled PO) to run a project into the ground, while developers swarm to no end on vacuous user stories, which Gilb rightly points out, that since the user stories don’t say How, they cannot even be estimated.
It is time to transcend agile and be post agile. There are a lot of choices out there and people with a brain can think and tailor to their domain, not follow the herd, whether it’s into some degnerate waterfall or into some degenerate fragile process.
PA
I just wanted to add on getting past the status quo, that that is a tough nut to crack. I’ve been on huge moribund projects where even doing a checkout and build took 2+ hours, effectively wasting 25% of the resources at hand and that had been festering for years.
Your point is well taken here — for that to get corrected, someone would have to take the fall and say yup they’d been screwing up for the past 2 years and that cost what $10 million? Who wants to improve something when it shines a light on how many roaches everyone’s been eating this time? Better to keep it dark and hope noone notices, or they can’t be eradicated, or that’s just the way we serve things around here.
Getting past that point may be impossible I don’t have an answer here. Somehow if people want to save $10 million a year say, then they have to be able to do something that does that without having their heads chopped off for blowing $10 million year over the past 10 years cuz they didn’t think of it sooner.
I can tell you one thing, though, that the higher the paycheck and the rarer the position the larger the odds that the status quo will be maintained.
If some engineer is making $100k/year they can get a job anywhere if they annoy their current employer; a VP making $250k is not going to want to rock the boat because their just aren’t that many VP slots open out there, period. It’s not simple to keep your trajectory upward if you start rocking the boat.
That’s one of my main beefs with agile and scrum. They want a perfect environment to operate in. It’s unrealistic.
Whatever methodology is out there needs to work in imperfect environments, because that is what the extant reality is.
PA
You make good points. Let’s try to avoid getting caught up in semantics. I know ‘waterfall’ is a poorly defined term but software practitioners know that the word implies a sequential process. Likewise, the term ‘agile’ is being corrupted as more companies claim to be agile but are really just using the term to make their ‘waterfall’ implementation sound cool.
I agree with your point about high-paid VP’s. Most of them are focused on making ‘safe’ decisions; not necessarily ‘best’ decisions. The ‘next big thing’ in software development processes has to make everyone feel safe.
But, not all sequential processes are bad, nor are all of them fully sequential; even Royce shows feedback between the phases in the later part of his paper.
Calling things watefall is a straw man if you mean sequential then say sequential.
Agilists present a false dichotomy that everything that isn’t agile is waterfall…it’s nonsense. There are a million ways to do projects not just so called waterfall and so called agile.
PA
Ah, someone who has read Royce.
yes, “waterfall” is a bogeyman term to use to scare people, or try to.
The IIBA has addressed this terminology issue, using the terms “plan driven” and “change driven”.
The thing is, studies like those below have shown that success rates for different types of methods are about the same.In short, the biggest issue is make sure you are developing the right thing in the first place.
http://iag.posterous.com/it-aint-easy-being-agile http://www.iag.biz/resources/library/business-analysis-benchmark.html
A sub-text to that is the kind of thing you are doing. Agile speaks to “product” and it suits product development. Developing or enhancing business information systems is a “project”, and has different impetus and needs than product development. Have a look at the following discussion which starts on a different topic but evolves to product/project as it goes.
http://www.linkedin.com/groupItem?view=&gid=1916699&type=member&item=196767530&commentID=109948014&report%2Esuccess=8ULbKyXO6NDvmoK7o030UNOYGZKrvdhBhypZ_w8EpQrrQI-
Terminology is a big problem, isn’t it? “Plan driven” implies that agile developers don’t plan. “Change driven” implies that waterfall developers can’t change. Neither perception is true yet perception is reality whether it’s correct or not. Just as software systems vary widely in their design and function, software development approaches must vary to accommodate different corporate cultures. There is no single best way.
Life isn’t either/or. In agile maybe it is. Plan driven means mostly planning with some change. Change driven means mostly change, with some planning.
Re the frighten people I cover that here:
http://postagilist.wordpress.com/2012/06/13/the-perennial-waterfall-strawmanmyth/
PA