I’ve had many, many customer ask me how they are doing, how much better they are doing than a year ago. The entrance of SAFe, the IPO of Rally, and the flood of “just in time” experts and training companies make an ability to answer that question even more important. If organizations invest heavily into “silver bullet” solutions, the credibility of the agile movement will be the one to take the hit.
Many of the efforts to “measure” agility focus on practices, such as:
• Was the retrospective held?
• Was the product backlog “ready” at the Sprint Planning meeting?
• Is QA and testing an integral part of the Scrum Team, or separate?
• Is code coverage better than before?
Unfortunately, these measurements do not reflect outcomes. Tweaking them and buying processes and products to improve them may make the outcome worse or better. In the field of evidence, these practices are known as circumstantial. That is, there are so many variables around any one or group of them, that the outcomes are indeterminate. For instance, a well-held retrospective is irrelevant in the face of a product owner that pushes for quantity over value.
If we measure outcomes, such as value, we can then look back at the practices in use. If we tweak them and value improves, we have evidence that we have done something good.
This concept of using direct rather than circumstantial evidence is called Evidence Based Management. It springs from Evidence Based Practices, increasingly used to improve medical treatment and outcomes. For more, look up the articles from Harvard Business Review, McKinsey, and – yes – Wikipedia.
Amazingly, software development has never been measured based on outcomes. It is largely seen as an expense, and only individual releases and projects are measured, traditionally by requirements, budget and date.
If we start measuring software development by evidentiary outcomes, then we will have firm grounding when we assess its value to the organization, and the value of our investments and initiatives.
I’ll be discussing this more at the ALM Forum in Seattle on April 2. In the meantime, keep your chins up. The black plague of waterfall, deterministic thinking has loosened its grip on our profession.