Agile vs Self Fulfilling Prophecies

Most executives view agile development as simply a means to bring software to market faster than traditional methods such as waterfall. Executive understanding of how agile development works usually goes no deeper than "build it quicker". This article is not meant to bash executives. A C-suite executive should not need a deep understanding of the mechanics of agile methodologies, they have other aspects of running the company that they should be focused on. However, many agile projects fail because management at all levels do not effectively make the necessary cultural changes for agile to succeed. I believe effective agile development requires a cultural shift in the basic processes surrounding software development, and that shift must be supported from the executive C-suite all the way down to the team lead level.

The cultural shift that I am going to focus on today will be around release scope, estimation, and project planning expectations.

Self Fulfilling Prophecy of Waterfall 

A perceived advantage of a traditional waterfall development approach is that management knows what features are being developed (scope) as well as the corresponding estimates for those features (cost).  This information is plugged into a linear project plan along with the available resources and a date for project completion is yielded.

Under this linear waterfall approach there is a hidden incentive to pad the feature estimates to ensure the development team can hit their dates. These bloated estimates along with the linear project plan become a kind of self fulfilling prophecy. How often do you see waterfall projects completing early or delivering more scope than promised? It happens, but in my experience it is rare. Software development is like a gas, it will expand to fill its container and the project plan is the container. Management at all levels plan around this rigid linear timeline and as long as the milestones are being hit on time, all is well.

Working to a rigid project timeline is often embedded into the culture of the entire organization. There is little to no incentive to come in  early or deliver more scope than promised. The waterfall process can even restrict the addition of scope since once you hit the coding phase of the project all of the requirements and designs are thought to be completed.  You cannot easily add scope since the process does not usually allow for opening up requirements and designs once coding begins. Therefore culturally there are hidden pressures to simply push forward with the original plan unchanged. The culture teaches that the less change to the requirements and designs the better the project is going. This is almost always wrong.

Often times the best you can hope for with waterfall is an on time delivery of the promised scope. That sounds pretty good though doesn't it? The reality is that on time delivery of the promised scope is frequently not all that great. This linear method does not deliver the best results since mistakes in requirements or design are usually found too late to make meaningful revisions. You usually end up getting what you asked for, and often times you didn't ask for the right thing.

Agile Scope / Timeline / Cost Trap

An agile SDLC such as Scrum makes determining the exact scope and cost of a release much more difficult. Many techniques are used to "size" features that will go into each sprint, but it is very difficult to guarantee the exact features that will be included in a release. The idea of an iterative development process is to constantly refine and add to the application by learning from your mistakes during each iteration. Agile makes reaction to mistakes easier because you find them earlier in the life cycle. Therefore it is impossible to completely define what features will go into a release at the beginning of the project. This gives many executives a bad case of heartburn. They want to "know" what they are going to get, when they will get it, and what it will cost.

A common pitfall that I see in agile development is to succumb to the pressure of providing an exact scope and cost of a release at the beginning of a project. This pushes the development SDLC towards a more waterfall approach rather than iterative. Even if the development is done in iterative sprints a plan gets communicated that includes total scope and a final delivery date. Therefore reacting to unforeseen problems becomes nearly as difficult as it was under the waterfall approach.

Projects that have fallen into this detailed scope / timeline / cost trap tend to have less success with an agile methodology because they are not truly embracing the culture of agile development. Again, the biggest advantage of agile development is to catch mistakes early and continually refine your product by discovering what works in the small iterations. If all of the sprints are fully planned with scope at the beginning of a project, reacting to change can only happen by adding resources or elongating the project plan. This is counter to the goal of agile.

Under Promise and Over Deliver

I recommend all levels of management embrace a concept of under promising on features, but strive to over deliver. Instead of fully planning every feature on your wish list for the release, trim the feature list back to a bare minimum. Estimate the features as you would usually, i.e. with a large padding, and plan them into your sprints to provide a very high level project plan. Allow the team to select features from this "minimum" backlog but encourage the team to deliver more features in each sprint then what is planned. By doing this you eliminate the self fulfilling prophecy of a project plan.The team is motivated to deliver as many features as possible, vs. simply being "managed" to a rigid project plan.

I have found this strategy to work extremely well. Using this technique we are still able to deliver a high level plan complete with a delivery date, features (scope), and resources (cost) for all of executive management and our customers to plan around. Our success rate at delivering the minimum promised features on time is near 100%. But the big payoff is our ability to deliver more than what was promised, that is the over deliver part. The agile methodology along with this technique has allowed us to adapt to problems quickly and say "yes" to customers when they are asking for fixes to problems or additional scope. The frequency with which we can "squeeze" additional scope into the current release is extremely high and is measurable in our customer satisfaction results.

Professional software developers take pride in their work, I have encountered very few in my career that do not. Building the right culture in an organization and adopting good agile development methodologies will deliver great results. The key is to get the process out of the way of the development team and let them build software quickly. Once this is done right, the teams will take pride in over delivering on features and the results will show.