Project methods – Waterfall versus Agile

I have been reading a lot of discussion recently regarding the use  of different methods for planning, implementing, and managing projects.  Many methodologies exist in the industry; two of them are:

  • Traditional Waterfall (some would say PMI PMBoK but not necessarily) model
  • Agile (SCRUM, Extreme, etc.)

I read a really good article last week in the Harvard Business Review Blog — A Better Project Model than the “Waterfall” by Jeff Gothelf. In this post he talked about needing a better model than the traditional waterfall model.

Here is a summary of Jeff’s post:

The main argument is that waterfall methodologies, which have been around for a long time, have a fatal flaw–they depend on accurate requirements. The problem with this dependence is that requirements are often wrong. He discusses giving teams the ability to work with clients to find out what the real requirements are. One of the quotes from his post that I really like was from Josh Seiden, who said, “In software development, reality bats last.” What he means is no matter how much planning or optimism there is about the software product being built, the only true project success is the reality of customers using it to solve real problems.

He goes on to share examples of Agile methods, in which teams use lists of features that are prioritized by product owners (not developers) to drive the project. In addition to prioritized lists, he suggests that each project team should be handed specific success criteria. These  criteria must  be quantifiable and directed to specific outcomes that meet customer’s needs.

My own opinion of this conversation is that it is not either black or white–waterfall vs. Agile.  As both as PMP, certified in waterfall methods, and a PSM, certified in Agile Scrum methods, I contend that a blend of both methodologies is often the right answer. Agile requires small, bite sized “sprints” with smaller teams in order to produce results. However, many projects are quite large, with as many as 300 people working at any one time. In these cases, running lots of small teams with sprints can become a project management nightmare. Using an approach that allows some portions of the project to use a more traditional waterfall method, and other parts of the project to use Agile, can make the overall project more management while still achieving the desired outcomes.

As I’ve stated in prior posts, requirements are not easy to capture, especially in complex software products. There are many techniques today (including Agile) which allow us to work with real users and discover the true requirements and needs and the measurable outcomes that must be produced. One such technique, Performance DNA, is used by my organization to ensure that all requirements are identified and linked back to prioritized outcomes.

In closing, as an experienced project manager who is certified in both methodologies, I am suggesting that “either/or” is not the right discussion.  Instead, I propose that a combination of waterfall and Agile is more likely to be the right approach.

Do you have experience using a blend of waterfall and agile, or do you prefer one over the other?

Project Cost Estimating and How Scrum Projects Estimate Cost

If you want to provoke an emotional reaction from a group of software development managers, ask them about their experience doing project cost estimation – WOW. I doubt you will find any of them who look forward to the task. Some of the managers may even become downright hostile. Estimating the cost of a software development for a contract or a proposal can be a punishing experience. And, almost every manager will admit that their estimate — especially for a complex project or one with a long duration — will be wrong (but hopefully not by much).

Why? Because

  • Requirements change during development, which changes the level of effort and therefore the cost
  • The operational environment is not, and cannot be, completely understood before work begins (Unless you have the Johnny Carson “Carnack the magnificent” genie)
  • In order to create a “price to win” in competitive programs, senior management reduces estimated cost and then pushes the team to “work smarter” (HA – can I hear all the groans from project members!).
  • There is a gremlin called “Murphy” who loves to show up on projects and create all kinds of problems*.

The History of Software Project Cost Estimation
In the beginning there was intuition and wishful thinking that drove software cost estimation. A project manager, often not a practitioner or software developer, reviewed high-level requirements and created a cost estimate from that – “I think this should take a couple months.” That approach did not always work well (:-).

In an effort to be more scientific in cost estimating, professionals developed many techniques over the years for performing estimates – historical analogous estimating, parametric estimating, three point estimating (PERT), expert judgment, etc.  In fact, PMI has produced a book called “Practice Standard for Project Estimating”.

The software industry developed one costing method that assessed a cost per line of written code and then estimated how many lines of code would be required to build the software. The success of LOC estimation methods suffered from the reasons listed above as well as the obvious problem that not all lines of code are equal in difficulty. I once asked a senior software engineer, how many lines of code it would take to develop X functionality. He replied, “How many do you want it to take?” He was not being a smart aleck, he was only acknowledging the fact that there are many ways in software development to skin a cat and those options require different amounts of code.

LOC-based cost estimating was followed by function point cost estimation. A function point is an output, inquiry, input, an internal file, or an external interface. In simple terms, you develop a cost estimate by counting the number of function points in a proposed system, assigning a cost per point, and multiplying. More involved function point counting included values for difficulty and importance to the eventual system.
There were many other attempts to develop fast, cheap and simple software cost estimation methods. I could go on for a long time – However, I won’t bore you with the details.  Let’s just say there is no magic book or formula for this critical process!

Cost Estimating in Scrum
Over the last few weeks, I have been writing about using the Scrum framework for software development. Scrum practitioners do not avoid cost estimating challenges, but they have methods to assess project cost that are consistent with Scrum’s approach to project execution. They do not, however, have the magic bullet, either.

All Scrum projects begin with a story (read “high level requirements” for the PMI crowd). What does the product owner want to be able to do when the project is complete? From that vision come the initial requirements. Scrum cost estimating methods use the wisdom of the team to generate estimated effort — acknowledging from the beginning that final cost and duration will vary as uncertainty and changing requirements are reflected in actual development.

When the Scrum team meets for its initial planning, the vision is laid out by the product owner. The team then considers how to create the desired functionality, which are called story points. Each story point is rated for relative difficulty.
Cristopher McSpiritt describes the process this way in “What is Planning Poker?”

The Steps:

  1. The product owner describes a user story/feature the team is estimating. The product owner may provide clarification on the feature in response to questions.
  2. Each team member chooses a development effort card from a deck of values to represent their relative estimate of level of effort. Cards are placed upside down on the table so that they do not influence others.
  3. After all estimates are on the table, the cards are flipped over.
  4. If there is a significant range among estimates, the team members who submitted the high and low values provide a rationale for their estimate.
  5. Once the discussion on the range has been conducted, steps 2 through 4 are repeated until a consensus is reached.

This team process has the advantage of sharing the vision, building the team and gaining buy-in from the Scrum team members who have to actually do the work.

In his book, Agile Estimating and Planning, Michael Cohn reports success using a Fibonacci sequence, such as 1, 2, 3, 5, and 8 … for a relative story point difficulty value scale. The team then considers how many story points it can accomplish during a sprint – called velocity. An experienced team can generate a useful sprint / project estimate from this process. However, an estimate is just that. During actual development (the Sprint), task difficulty and team velocity are routinely reviewed and modified. As (predictable) requirements changes are requested by the product owner through the product backlog, variations in the order of the product backlog (Requirements or features) can be accommodated over the life of the entire project without affecting the current estimates and sprint.

One of the big advantages of this method is that we don’t have to estimate the entire project in excruciating detail or even know the complete set of features at the beginning of the project – this is more real world than the traditional waterfall type of project planning and estimating.
I would love to hear your opinions and thoughts about estimating for projects, both traditional and Scrum.

* Capt. Edward A. Murphy is accepted as the reason for Murphy’s law

The Lazy Project Manager's Blog

The Home of Productive Laziness Thoughts

Thoughts, experience, tips and tricks on issues affecting managers and project management

A Girl's Guide to Project Management

Project Management musings for one and all

LeadingAnswers: Leadership and Agile Project Management Blog

Thoughts, experience, tips and tricks on issues affecting managers and project management

Project Management Hut

Thoughts, experience, tips and tricks on issues affecting managers and project management

Herding Cats

Thoughts, experience, tips and tricks on issues affecting managers and project management


Pushing the Edges Out ...


Just another site