Waterfall, Lean/Kanban, and Scrum

June 10, 2010 Categories Kanban, Lean 92 Comments

There are four identified problem spaces: Simple, Complicated, Complex, and Chaotic. The identification of the problem space arises from the characteristics of the problem domain. In software development, these are the technology, the requirements, and the people doing the work.

If these characteristics, or dimensions, of the problem space are simple and well-known, the problem is defined as Simple. If the characteristics are unstable, frequently changing, and not well-known, the problem is defined as Chaotic. If the characteristics are not simple, but are more known and well-defined than not, the problem is defined as Complicated. Lastly, if the characteristics are not simple and are more unknown and undefined than otherwise, the problem is known as Complex.

Throughout the 1980’s, I worked to make projects that used the waterfall process successful. Despite creating and revising methodologies that delineated waterfall, and even automating the underlying delivery mechanisms, I was not pleased with the results. Then, in the early 1990’s, Jeff Sutherland and I formulated Scrum, an approach to software product management that was radically different from waterfall. We experienced success after success when we used it.

I needed to know why Scrum was so much more successful than waterfall. I had friends working on advanced process control at the DuPont Advanced Technology Center in Delaware. DuPont had mastered the production of simple polymers, like rayon, to the point where they could be produced anywhere in the world. However, advanced polymers like GoreTex was so complicated that it’s production couldn’t be distributed. My friends were working this problem. The head was Babatunde Ogunnaike (Tunde), one of the authors of the bible of the process control industry, “Process Dynamics, Modeling and Control.”

I showed Tunde several waterfall processes, methodologies, that I had brought with me. One was my own, another was the Coopers & Lybrand Summit D methodology. Upon inspection, Tunde observed that the processes were very prescriptive, where there was a complete plan of how to turn raw materials into finished product. He also observed that the elements of the plan weren’t well defined. The input, output, processes, and resources consumed at each point in the plan (task) weren’t of adequate detail to ensure correct flow. Tunde then asked me how successful these types of methodologies were. I told him that our project success rate was below 50% despite heroic efforts to improve and enforce the processes.

Tunde told me that defined processes, such as waterfall processes, are the basis of mass production. Although not the most cost-effective, they are dominant in volume. Experts lay out all of the workflow to turn raw material into finished products. They then hire managers to bring in resources to staff the various points in the workflow (automation is preferred for reliability, otherwise use people). The job of the manager and their supervisors was to teach the resources how to do the defined job. If they did their work correctly, the flow would be smooth and the predicted product would result. Sometimes the plan was wrong, and would be corrected. Most errors came from inadequate execution by the resources.

Tunde told me that the waterfall process aped this model. He said that this model was preferred, and always used as long as it was successful. If it weren’t succesful, regardless of attempts to tighten it up and retrain the resources, there were other, more expensive ways to create the products. He asked me what the success, or yield, was. I told him that it was less than 50%. He was aghast. He said that in modern industry a yield of less than 99.7 was considered inadequate. Fifty percent yield would be like GM throwing out every other car it built.

Tunde suggested we look at other models. First, he suggested Lean, the domain of problems where there is some complexity, but there is more that is known and stable than there is that is unknown and unstable. Tunde said that this approach was similar to the defined approach, but that it took into account complexities that might lower the yield rate to an unacceptable level. First, experts lay out the line. Managers and supervisors hire the resources and train them. Then, the line is surrounded by metrics so the experts can monitor if perturbances, or unexpected events, are occurring. Whenever they are detected, the line is stopped, the perturbance smoothed out, and production resumed. The resources were also trained to spot unexpected complexities and had the authority to stop production until the complexity was resolved.

Lean processes optimized quality and value. Sources of defects were rapidly detected and removed. Everything that wasn’t geared toward optimizing value was also removed. Waste was continually detected and removed. Decisions were delayed until the last moment to remove waste. Lean processes are the most cost-effective.

Tunde said that this approach fell apart when unexpected events overwhelmed the productivity of the line. More time was spent detecting and fixing complexities than building product. More time was spent inventing new metrics to detect complexities than was spent designing new products. He also pointed out that the Lean realm was still mass product development. It was hard to justify building production lines with controlling metrics for a run of just one product, or one release of software.

Tunde then took me to the realm of empirical process control, a technique used for small batch production when there is more unknown than known, where the complexity is greater than the predictable. He felt that this was more relevant to the field of software development. The churn in our requirements, the complexity of the many technologies, and the use of creative thinking by people made this approach most appropriate. This approach was set up to expect the unexpected through frequent inspection of progress and subsequent adaptation of next steps to optimize output. The basis of it was the container.

A container is a closed space where things can get done, regardless of the overall complexity of the problem. In the case of Scrum, a container is a Sprint, an interation. We put people with all the skills needed to solve the problem in the container. We put the highest value problems to be solved into the container. Then we protect the container from any outside disturbances while the people attempt to bring the problem to a solution. We control the container by time-boxing the length of time that we allow the problem to be worked on. We let the people select problems of a size that can be brought to fruition during the time-box. At the end of the time-box, we open the container and inspect the results. We then reset the container (adaptation) for the next time-box. By frequently replanning and shifting our work, we are able to optimize value.

There is a radical shift from prescriptive approaches (waterfall, Lean, Kanban) to empirical processes (Scrum). An organizational culture changes if required.

Prescriptive approaches rely on the ultimate ability to effectively plan and control the plan to success. Experts set up the work, managers staff and manage resources to do the work, and workers (interchangeable) do the work. Command and control optimizes productivity in predictive processes. Empirical processes rely on containers to control risk and to continually aim the results toward the highest possible value. Managers work is to pose the highest value problems, and to do anything they can to assist the people who are solving the problems. Creativity and collaboration are the hallmarks of this process.

All organizations have issues adopting Scrum. They are shifting from a prescriptive, manufacturing based approach to an agile, empirical approach. Every discrepancy between their current organizational culture and Scrum’s value driven approach is highlighted as an impediment to optimizing value. The organization is then asked to change its current culture to achieve this agility and value. When the change is too hard, they often change Scrum and keep the impediment. We call these “ScrumButs” because they are said as, “We use Scrum, BUT we found that it was very hard to have daily Scrums, so we only have them once a week.”

I was told that Kanban is frequently used when an organization cannot readily adopt Scrum. Many of Scrum most difficult aspects are then sidestepped. Managers are still in charge of telling people what to do. People can be interrupted at any time. People are still work in functional silos, preserving the jobs of functional managers. People are not allowed to work in containers, sharing skills and knowledge to bring complexity into solutions – instead they are worked on a pull (more sophisticated than push) production line.

Lean is more productive than waterfall. Its transparencies and metrics allow frequent adjustments to optimize productivity. However, that optimized productivity is creativity when people are used to solve complex problems. People are worked like cogs in a machine, and their work is optimized at the bit level, rather than being aggregated into the value level.

God help us. People found ways to have slack in waterfall, to rest and be creative. With Lean and Kanban, those hiding places are removed. We now have a progressive death march without pause.

Scrum is intended to be a place where enthusiastic people can work closely with their customers to derive the most valuable, creative products possible. Getting to this point is a hard road for organizations transforming from a waterfall, command and control, prescriptive culture. However the rewards are great and the goal compelling. Organizations that make this transition will outcompete everyone else and be places where everyone wants to work.

Those who cannot Scrum and slide back into the slickly branded ScrumBan and its like will find themselves becoming coal mines, quickly generating commodity products from a work force that cannot wait to unionize itself so it has more rest breaks.