Simple Math Doesn’t Work in Software Development

If building a particular software system is scheduled to take 5 engineers, 6 months, then simple math says that 10 engineers can build the same system in 3 months. (That’s twice the number of engineers and half the time.)

Okay, okay. I know. Some additional communication overhead will be introduced by adding people to the team. Fine. I’m in a generous mood. I’ll give you 4 months. Now get to work.

Not so fast.

Simple math doesn’t always work, particularly when you apply it to complex situations. Also, keep in mind that project estimates are prone to variability and change. A 6-month estimate could represent from 3 to 12 months of work.

Let’s examine some of the factors that affect the productivity of a group of software engineers.

• Communication skills
• Software engineering experience level
• Experience with the toolset
• Work environment
• Ability to learn
• Structure of the system
• Availability of business stakeholders and end users
• Availability of support personnel to document, deploy, etc.
• Availability of target systems to develop, test, etc.

You get the idea. Adding software developers is just the tip of the iceberg. Are all the other pieces in place to make adding developers worthwhile? If so, what’s being done to help this larger team form, storm, norm and perform?

The bigger they are, the harder they fail.

Adding people to an established team at the outset of a project is a questionable idea. As time goes on, the issues listed above only loom larger making adding people well into the project a bad idea. Adding them nearer the end of the project in a mad rush to make up lost time is foolish.

I won’t say that you should never add people to a software development project but if you do, review the list above and take appropriate actions to support the team. Finally, be realistic. Adding a person may only provide half the benefit that the simple math suggests. Adding more than one could even have a negative impact.

The Peter Principle Is Real But Agile Techniques Can Beat It

What causes the “Peter Principle” to rise up and ruin someone’s career? According to the principle, named after Dr. Laurence Peter, “In a hierarchically structured administration, people tend to be promoted up to their level of incompetence”.

That’s a scathing comment. It is often repeated and there are many examples of people who get promoted only to fail in their new jobs. How is it possible that someone who earned a promotion by doing a good job, turns out to be ineffective or incompetent in his new job? Here are a few possibilities.

1. An individual contributor is promoted to manager. It happens all the time. Many workers aspire to be managers. With proper training and coaching along with good people skills, anyone can be a manager. Unfortunately, the training and coaching part is often missing. Another critical success factor (CSF) for managers is the ability to multitask. Without it, the Peter Principle rules.
2. A subject matter expert (SME) is promoted and assumes responsibilities beyond her core area of expertise. Assuming the new area of responsibility is related to the SME’s existing knowledge base, taking on a bigger role can work. Of course, the SME will need help and guidance. A good mentor can be invaluable in making the transition. And she will need to master that multitasking CSF.
3. A great implementer is promoted into the role of designer. Design work is different from implementation work. The skills, tools and approaches are related but not identical. Training and mentoring are key ingredients. Oh, and designers tend to do much more multitasking (CSF).
4. A technical genius is promoted into a role that requires people skills (e.g. negotiation). Once someone masters the objective part of their job (technical details), that person often wants to try out the subjective (human interaction) part of the job. Fair enough, though the skill sets are usually vastly different. And, anything involving people will invariably require that CSF – multitasking.
5. Someone who is accustomed to focusing on a small number of tasks is promoted into job that demands multitasking across a broader array of work. There it is again.

Do you see a pattern here?

The lack of training, coaching and/or mentoring is a solvable problem. Why would any company invest time and money in an employee, promote the person, assign her new and more strategic responsibilities, and then drop the ball? It’s insane! Greatness doesn’t just happen. It requires constant vigilance.

As for the problem of multitasking, it’s not going away. When we multitask, we are less efficient. We make more mistakes, miss important details, and forget critical facts.

There’s a simple solution. It’s based on the fourth principle of the Agile Manifesto – “Business people and developers must work together daily throughout the project.” It’s called teamwork. The solution to the multitasking problem is teamwork. Share the workload. Divide and conquer. Many hands make light work. Many minds do great work.

The next time you get promoted do two things. 1) Get help. Insist on training and mentoring. 2) Don’t go it alone. Collaborate with your coworkers and team members.

Software Processes Are Good; Procedures Are Not

Many of us follow procedures. Not because we like them or find them useful but simply because that’s what we are told to do. Regrettably, many procedures take on a life of their own and become powerful impediments.

For purposes of this post, I’m defining a process as a systematic sequence of actions towards a goal. A procedure is a defined series of actions, carried out in a prescribed order. The distinction is subtle and important. Processes may be composed of other processes and procedures. However, procedures are at the bottom of the hierarchy. They may invoke other procedures but may not invoke processes.

It’s common to find many procedures being enforced in large organizations. IT departments routinely enforce complex procedures as a way to “control workflow and reduce risk”, or so they would have us believe. In reality, many procedures are designed solely to protect a workgroup. The procedures add no value to the company or its products and services.

Here’s a true story.

Years ago, I was a manager at a large minicomputer company (who shall remain nameless). I worked in research and development. One day we got word that everyone in R&D would need to fill out paper timesheets every week. We were told that the company wanted to track capitalized projects versus non-capitalized ones.

This was a tax reporting issue. I won’t get into the details of how projects were treated for tax purposes back then. The explanation we were given seemed plausible. We grumbled a lot but agreed to comply.

For most people, the new procedure was simple because they were assigned to a single project and they merely entered 40-hours per week on that project. For senior people and managers, it was more complex. We had to track hours spent on every project that crossed our desks. It was similar to how consultants track time across multiple clients.

This time-tracking procedure went on for a few years. One day, the management team in our group decided to gather some statistics on how many hours we had spent on projects. We believed that armed with this data, we could provide better estimates for new projects.

We thought that all the timesheets we had submitted would provide a wealth of historical data. Someone must have compiled those into a database, spreadsheet, or tracking system of some kind, right?

Wrong? When we inquired about the timesheets, we were told that they were stuffed into a folder and no one ever looked at them. Furthermore, we were told that we were one of very few groups in R&D that submitted the forms. Most groups had abandoned the timesheet directive though it was never formally withdrawn.

Amazing, isn’t it?

I have no idea how many hours were wasted filling out and collecting timesheets simply because it was mandated by someone trying to do the right thing but in the wrong way. That person’s focus was his department, not the company as a whole, and certainly not the R&D group.

You can’t be agile if you’re forced to follow rigid procedures. Being agile requires the ability to make adjustments as conditions change and new information becomes available. When it comes to software engineering, process is good; procedures are not.

How many procedures are you forced to follow in your job? Do you know why? Do the procedures benefit the company, its customers, and/or it’s investors? Or, do they simply benefit the group that enforces them? Let me know.

Not Getting the Results You Want? Perhaps You’re Measuring the Wrong Things.

Agile process metrics are a frequent topic of blog postings with the velocity metric most often being discussed. But do agile process metrics really matter? What does the business really care about?

Software development metrics are important to development teams but they mean little, if anything, to the business stakeholders and end users. Business professionals care about results and metrics around sales, profits and productivity. Sure, getting software projects done better and faster impacts business performance, but the relationships between them are nebulous at best.

Look at metrics from a different perspective.

The software we develop should help end users do something better. Customers should have a better buying experience or improved self-help options. Employees should experience better workflows, improved knowledge management, or faster communications. Those are the results that matter.

It often amazes me how managers will believe that a process, a workgroup or a tool is inefficient and wasteful. Yet, when I ask probing questions to determine how the conclusion was drawn, I find that it’s conjecture. There is no hard data to backup the statements. How can any business measure outcomes and improvements without meaningful data?

This places software projects in jeopardy. If the current business situation is misunderstood, the improvement offered by the new software might also be misunderstood. Even worse, the software development team could focus on the solving the wrong problem.

We need better metrics around the problems we’re trying to solve not the process we use to solve them. To those ends, we need to capture before and after metrics around the end user experience. What follows are some questions to ask business people. Hopefully, the answers will provide sufficient data to help determine the improvement provided by the new software and the performance of the software development team.

Food for Thought

• What is working well in the current environment? How do you know?
• What’s wrong with the current approach? How do you know?
• What information is captured today? Where is it kept? How is it used?
• How is performance tracked and reported? Are there any performance goals today?
• What are your goals for the new software?
• What would a successful outcome look like? How would you measure it?
• What information would you like to have the new software capture?
• How would you like to see the new metrics reported?
• How often do want the metrics updated?
• Do you want to improve customer or worker satisfaction? How would you measure it?

This kind of dialog will help both the business and technical professionals determine the success (or not) of their efforts. Internally-focused software development metrics like velocity are useful but they only help on the technical side. Externally-focused metrics around performance and productivity help everyone.

Scrum Isn’t Just for Product Development Projects

I was recently asked a question about applying Scrum to research projects. The core issue had to do with Scrum’s fixed time boxes. Research projects, by their very nature, aren’t driven by fixed deadlines. Depending upon the complexity of the research, the effort could take weeks, months or years. How do you apply Scrum?

While Kanban may actually be a better fit simply due to it’s flexible timing, I’m going to focus on Scrum because it was the context of the question.

Research is often a systematic investigation designed to harvest knowledge from facts and observations. Research projects frequently deal with fluctuating requirements that result from new knowledge learned via scientific inquiry. As new findings emerge, new interpretations of the original goals evolve and the process may have to begin anew.

Here are a few key areas where research projects differ from product development projects.

• The requirements may be broader and more loosely defined.
• Stories (or use cases) may be harder to estimate regardless of the measure used for units of work.
• The work plan may require more flexibility and on-the-fly planning.
• The project goals may be moving targets.
• Substantial documentation may be required.

Despite these challenges, Scrum can work. Just like product development, research is a structured and disciplined effort. Even if the research is largely based on observation, the results being observed will lead to the definition of new stories that will get added to the backlog like any Scrum project. We may need to bend Scrum rules a bit, but that’s okay. The basic Scrum principles will still work.

Scrum Guidelines for Research Projects

Here are a few guidelines for conducting research using Scrum as the process that drives the team.

• Keep Sprints short. Shorter sprints will make it easier to adjust to new information and changing “requirements”. Consider 1-2 week Sprints.
• Keep stories small. For example, setting up an experiment, doing it, and documenting the results can be separate stories. Think about activities that can be completed in 1-2 days.
• Each Sprint does not have to “deliver” something tangible as in product development. An observation made during a Sprint may be immensely valuable though not yet documented or even analyzed.
• Don’t be afraid to abort a Sprint. If something unexpected is discovered and the Sprint backlog is severely impacted, end the Sprint and regroup. (Just don’t make a habit of it.)
• The Research Owner (there is no Product Owner) and the end user may be the same person during the project. Only when the research is complete will the results be made available to outsiders.

Finally, daily standups, a Scrum Master who removes impediments, and good backlog management apply to any project using Scrum. Give it a try on your next research project.

Legacy Systems Will Have to Be Replaced — What Are You Waiting For?

In my last post, “The Cost of Doing Nothing May Exceed the Cost of Changing”, I began to examine the topic of replacing outdated, legacy systems in enterprise environments. I’m not referring to upgrades or enhancements. This is about complete replacement. It’s a daunting prospect.

I laid out two basic approaches: 1) Build a parallel system and cut over to it when ready or; 2) Replace the target system one major component at a time. Which approach to use depends upon the particulars of the situation. It’s not a simple decision.

Which ever approach you use, it’s imperative to apply agile development techniques to mitigate risks and control costs. Starting down a replacement path, getting everyone excited about the prospects, then not delivering the solution for many months (or even years) guarantees disappointment at best and failure at worst.

Some things to think about…

• Engage the stakeholders and end users early in the process and keep them engaged throughout.
• Actively seek feedback throughout the development effort and take immediate action on it. Don’t simply add suggestions to the product backlog and forget about them.
• Recognize that implementing an entirely new system will cause some disruption to daily business activities. Allocate time and effort toward minimizing it.
• Communicate and keep communicating. Keep everyone informed of what has been done, what is being done, and what’s next.
• If an unforeseen problem occurs (and it will), be responsive. Drop everything and fix it.

In the Parallel System Approach…

• Seek out ways to operate the legacy system and its replacement in parallel as early as possible. Don’t wait until the replacement is fully built out.
• Identify new capabilities and workflow improvements at the outset. Build them into the new system early to get the users interested in using the new system. Areas to explore include simplified data entry, improved workflow, and better reporting.
• Don’t expect users to repeat their entire workflows on the new system. Target specific areas for review and comment. Keep it simple.

In the Component Replacement Approach…

• Find opportunities to add value. Many of the backend changes will be transparent to the users making it difficult to engage them. Find areas where new capabilities can be introduced even if it means running a separate UI or batch process.
• Identify pain points in the legacy system and alleviate them at the earliest opportunity. This will build goodwill with the stakeholders and end users making the inevitable disruptions more tolerable.
• Breaking down the old system into many components will make it easier to build replacements but may also complicate the process of integrating the old and the new. Always evaluate risk, cost and time. Minimize two of them and let the third float.

Lastly, don’t wait for that legacy system to die or become too painful to use and maintain. What are you waiting for?

The Cost of Doing Nothing May Exceed the Cost of Changing

Resistance to change is rooted in many factors from fear of the unknown to resentment of forced mandates. Oftentimes, the underlying resistance lies in the sheer magnitude of the challenge. Many enterprise IT projects are massive undertakings with big payoffs when they succeed and huge downsides when they don’t. What follows is a situation routinely encountered at large firms and often results in an outpouring of change phobia.

It’s common to find companies using many legacy software applications running on outdated hardware. Those systems are frequently powered by old mainframes or minicomputers sold by IBM, DEC, DG, Wang, HP or Honeywell. Often these ancient beasts are interconnected in such a fragile manner that changing anything, anywhere, results in unintended consequences.

The people that developed the systems are usually long gone — laid off, retired or off to better situations. The technical documentation is usually sparse and outdated. The procedures for operating the systems are brittle and must be followed carefully and precisely.

Sounds like an accident waiting to happen, right?

Faced with such complexity, many IT departments freeze. They lock down everything to prevent change. They try to extend the life of the systems for as long as possible. Eventually, either the business folks get fed up with the lack of technological progress or hardware failures become overly disruptive and intolerable.

Is there any way out of this mess?

The biggest fear factor in these situations stems from the size of the problem. Overhauling major enterprise systems is a huge undertaking by any measure. First, ask yourself what it’s costing the company not to change. Old systems are expensive to maintain and service. In addition, the business may not be able to leverage modern cost efficiencies putting profit margins under stress.

The cost of doing nothing might be much more than you think and might even outweigh the cost of replacing those relics. Time to move on? There are really just two options for moving forward and they both carry significant, but manageable, risks and costs.

1. Select a production system to be replaced. Build a parallel system that will ultimately take over all of the functions performed by the older system. Operate both systems for a brief period of time to give everyone a chance to get accustomed to the new software. Once you’re convinced that the new system does everything it needs to do, decommission the old system.
2. Decompose the legacy system into its major component parts, for example, database, batch processing, user interface, etc. Replace one major part at a time by building a new component and connecting it to the remaining legacy components via a software wrapper or emulator. Repeat until the entire legacy system has been replaced.

Neither approach is necessarily more agile, lower risk or lower cost than the other. The most important message is to get started. You can’t finish if you don’t start.

In my next post, I’ll offer a few guidelines for approaching these types of projects in an agile way.

Focus on Team Collaboration Not Individual Skills

I’ve had scenarios similar to the following one come up several times over the years. You have someone on your software development team that writes great Java code — and does it fast — but can’t or won’t do much else. Should you remove him from the team? Before you answer that, let’s dissect the situation further.

Ideally, members of an agile development team should have both technical and non-technical skills. You see it in job postings all the time. Assuming the team is building a system using Java as the primary language, knowledge of Java, the JRE, and various Java development tools and libraries are needed skills.

The non-technical skills are more complex. They fall into buckets such as communication skills, team player, customer oriented, quick learner, results-driven, etc. (I don’t know why they leave off “walking on water”.)

Let’s face it, very few of us can do everything well. What do you do with someone who writes great code but falls far short in the non-technical (subjective) areas? I’m sure many managers would say ‘send him to training’. After all, we’re all supposed to work on our weaknesses, right?

Two major obstacles come to mind.

1. The developer may have no interest in mastering the non-technical parts of the job. Sending him to training might result in small improvements but major gains are unlikely. In fact, the situation could deteriorate if the employee rebels against the forced subjective behaviors.
2. The developer may have an interest in bettering himself but you have to expect that his coding prowess will suffer. How can he continue to write great code at a fast pace while also spending more time on other aspects of the job? It doesn’t add up.

Think carefully before creating a mold and trying to get everyone on the team to fit into it. You might be better off leveraging people’s strengths and offering assistance to cover their weaknesses. As long as the team possesses all the skills needed to deliver the results required by the company, they can be successful. Expecting every member of the team to possess every skill may sound good but it’s unrealistic.

Focus on collaboration instead.

Get team members to help each other and reinforce their abilities. Agile teams are not collections of people who work independently. They need to work together.

As for that Java developer I mentioned above, do everything you can to help him write great code. Have someone else on the team pick up the non-technical aspects of his workload. He’ll be happier and more productive. The team and the business will be grateful.

Are Multidisciplinary Software Teams the Agile Holy Grail or Achilles’ Heel?

Ideally, agile software development teams should be multidisciplinary, that is, possess all the skills needed to deliver the business solution. This is documented in the eXtreme Programming rule called ‘Move people Around’. “A team is much more flexible if everyone knows enough about every part of the system to work on it.”

It sounds good in principle but it’s not always easy or cost-effective to assemble and cross-train such a team. There are many skills needed to deliver high-quality software within an enterprise environment. Let’s assume you’re forming a new software team and you want it to be self-sufficient.

What skill sets should be on the team?

Obviously, the team needs people who can write code in whatever language(s) is/are being used. They’ll also need one or more people who can write automated tests and perform independent evaluations of code quality. Is that enough?

If the software system is small and relatively isolated, add a team leader or Scrum Master and a Product Owner and you’re good to go. The team will need to have a variety of skills (be composed of generalists) to accomplish everything needed but that’s manageable.

But, what if the software system is large, complex and part of an enterprise ecosystem? Can the same team composition take on this project? Here’s a sampling of the skills such a team might need.

• UX Designer (e.g. usability)
• Graphic Designer (e.g. user interface elements)
• Front-end developer (e.g. browser or desktop application)
• Middleware Developer (e.g. application server)
• Back-end developer (e.g. database access)
• Middleware developer (e.g. system integration)
• Database administrator (e.g. database design)
• Security Specialist (e.g. access privileges and data encryption)
• Business process analyst (e.g. workflow rules)
• Data Analyst (e.g. reporting)
• Software tester (e.g. automated regression tests)
• Technical writer (e.g. user guide, training materials)

While it’s certainly possible for a small development team to possess all of these skills, that’s unlikely to be the case. It’s also unlikely that the team’s skills would be as strong as they could be if the team members were specialists rather than generalists.

Can they still get the job done?

Assigning dedicated experts to the team is just not practical. Does the team need a full-time graphic designer or database administrator? Probably not, but they need a support structure. They need access to experts as needed.

In a small company, third-party services firms can help. In a large company with many ongoing projects, it makes sense to hire experts and have them shared across projects. But, an additional challenge is introduced.

The use of shared experts has be monitored closely. If demands on the experts’ time exceed their ability to deliver services, the teams will have to wait and deadlines will be missed.

Multidisciplinary teams are an agile development holy grail but they’re just not practical when building complex systems. Overloading the team members with required skills will create an Achilles’ Heel and reduce their effectiveness.

I believe the problem can be managed using Centers of Excellence. I’ve advocated the Centers of Excellence approach to sharing expertise in large enterprises before. It’s cultural shift that really works. You can read more about it in this post.

Technical-Debt-Driven Development: If It Works, It Ships!

Technical-Debt-Driven Development (TDDD) is about to become the hottest craze. Here’s how it works.

The software development team doesn’t waste time thinking about solution elegance or future considerations. They simply focus on getting the code working. If it works, it ships.

Technical debt is leveraged as a means of accelerating the team’s efforts just as monetary debt is leveraged as a means of building wealth. Software development is faster as a result and the rewards occur sooner.

What’s the catch?

There’s one thing I have to mention. TDDD really only works well for small, short-lived, software systems. You see, you can’t grow monetary debt indefinitely. At some point, you can no longer keep up with the interest payments and the lenders will drag you into court.

You can’t grow technical debt indefinitely either. Eventually, the team will get bogged down managing the debt. They’ll have to spend large amounts of time reworking and refactoring source code leaving little time for adding new features to the software.

So is my TDDD concept useless? Is is dead on arrival? Not entirely.

The perfectionists among us lobby for the complete elimination of technical debt (TD). They suggest that TD must be avoided at any cost. If you don’t eliminate it now, you’ll have to eliminate it later and the job will only become more difficult over time.

Really? What if we eliminated all monetary debt from our economies? Many people would not be able to afford the purchase of a home or a vehicle. Many businesses would not be able to expand and grow. The resulting loss of monetary value would cause economies around the world to collapse.

Thankfully, the comparison of monetary debt to technical debt is not quite that direct. Software systems and their companies will not collapse without TD, but they may suffer unintended consequences.

Make good choices

Good agile teams know how to make good choices. They evaluate situations and respond appropriately. If the business is in a bind and needs quick turn-around on some stories, a good team will find a way, even if means taking on technical debt.

The alternative is to give code perfection priority over the delivery of business value. Fewer stories will be delivered in the short term and business opportunities may be missed. Perfection doesn’t help if if gets the business into trouble.

On a brighter note, monetary debt has to be paid off eventually but technical debt does not. Many software systems, or parts of them, will become outdated and no longer relevant long before TD has to be paid off.

TD belongs in the product backlog like any other work item. It can be prioritized and ordered along with stories and defects. Maybe it will get attention and maybe it won’t. Either way, the team will make informed decisions.

Software is never perfect and TD is not evil. Don’t listen to those who want you to adhere to their dogmas. They only make us less agile.