I’ve had many, many customer ask me how they are doing, how much better they are doing than a year ago. The entrance of SAFe, the IPO of Rally, and the flood of “just in time” experts and training companies make an ability to answer that question even more important. If organizations invest heavily into “silver bullet” solutions, the credibility of the agile movement will be the one to take the hit.
Many of the efforts to “measure” agility focus on practices, such as:
• Was the retrospective held?
• Was the product backlog “ready” at the Sprint Planning meeting?
• Is QA and testing an integral part of the Scrum Team, or separate?
• Is code coverage better than before?
Unfortunately, these measurements do not reflect outcomes. Tweaking them and buying processes and products to improve them may make the outcome worse or better. In the field of evidence, these practices are known as circumstantial. That is, there are so many variables around any one or group of them, that the outcomes are indeterminate. For instance, a well-held retrospective is irrelevant in the face of a product owner that pushes for quantity over value.
If we measure outcomes, such as value, we can then look back at the practices in use. If we tweak them and value improves, we have evidence that we have done something good.
This concept of using direct rather than circumstantial evidence is called Evidence Based Management. It springs from Evidence Based Practices, increasingly used to improve medical treatment and outcomes. For more, look up the articles from Harvard Business Review, McKinsey, and – yes – Wikipedia.
Amazingly, software development has never been measured based on outcomes. It is largely seen as an expense, and only individual releases and projects are measured, traditionally by requirements, budget and date.
If we start measuring software development by evidentiary outcomes, then we will have firm grounding when we assess its value to the organization, and the value of our investments and initiatives.
I’ll be discussing this more at the ALM Forum in Seattle on April 2. In the meantime, keep your chins up. The black plague of waterfall, deterministic thinking has loosened its grip on our profession.
Best,
Ken
Ken Schwaber's Blog: Telling It Like It Is 
Pingback: Evidence Based Management | M Anthony Librizzi
Pingback: Evidence Based Management - The Agile Product Report
The lack focus on outcomes is utterly bizarre. Most if not everyone is or has been conditioned to tick the boxes on the process and I’ve seen this constantly fail and then the revisionists, apologists, vested interests and the obsequious come in and to a political patch up job. Does it mean that most of us can’t think. I’m really not sure from what I’ve seen. I’m probably just making a statement here, but trying to loosen the grip some more.
Yes, very surprising for such a large expenditure in every organization.
Ken, you make some very valid points here. That is why, from a consulting perspective, I have always recommended to clients that they use a traditional technique/approach…a Charter…to shape the goals and objectives of what they want to achieve with Agile along with the business initiative (not just because Agile is a hip/cool thing to do). Solving a business problem where there is additional revenue or fees generated, or cost cutting achieved should be the goal, and then if those can be measured on a pilot project, it can forge the way for an organization to measure the right things and begin thinking about I/T as an extension of the business vs. a cost center. Then, the cost of the I/T initiative (preferably using a fit to purpose approach using Agile and Lean practices) can be measured against the business value achieved. Years ago, measuring tangible value vs. intangible value was very common in large companies at the start of a project and in selecting which initiatives were conducted; even at GM, where we were able to show that a business solution project (including client centralization of the mission and cost cutting) paid for itself in 2 days.
However, along with the right business measurements, I am in favor of planning and documenting Agile and Lean Practice/Technique usage as part of Agile “Release” planning, and then using that as a baseline to measure against and refine as Agile teams execute and conduct retrospectives. I also find it valuable to Assess practice usage (or lack thereof) prior to starting an Agile initiative, to highlight and address common barriers to Agile success. But, if that planning and assessment is not conducted and evaluated (Both objectively and subjectively) by someone that has really done it before, the value of the measurement is limited.
You say:
Years ago, measuring tangible value vs. intangible value was very common in large companies at the start of a project and in selecting which initiatives were conducted;
To my point, these were project based criteria, not organizational.
Again, you say:
However, along with the right business measurements, I am in favor of planning and documenting Agile and Lean Practice/Technique usage as part of Agile “Release” planning, and then using that as a baseline to measure against and refine
Again, to my point. These are not outcomes, but circumstances that may or may not produce good outcomes.
Hi Ken – we recently released an outcomes-based contract (open-source so free) that might be of interest at flexiblecontracts.com. We are starting to get people to understand that you get what you measure and pay for, so if it’s time and materials, or money for scope delivered, that’s what you get. Whereas what we really need are outcomes delivered.
Hi Ken, great post! This is a subject that comes up all the time and I continue to struggle with concrete ways of doing this. I agree with you when you say:
“If we measure outcomes, such as value, we can then look back at the practices in use. If we tweak them and value improves, we have evidence that we have done something good.”
However, our PBL’s are ordered by business value (high to low), would you agree that even though we are tweaking our practices and improving sprint-over-sprint, our measured business value outcome may not reflect our improvements since we are now delivering lower value items?
–Mike
Yes. There are many circumstances that negate prioritizing for business value, such as delivery, poor customer service, and low quality.
Pingback: ChristiaanVerwijs.nl | What is this thing called (Business) Value?
Hi Ken, various valid points brought out by you in this article. However, I still feel that the effect of waterfall is too deep, in fact it is somehow related to our educational system where people are now trained to focus on exam score result than the learning. So your last encouragement statement “The black plague of waterfall, deterministic thinking has loosened its grip on our profession”; somehow I feel that they are still out there, now started to covered themselves up with the “Scrum Process”. Furthermore, “the entrance of SAFe, the IPO of Rally, and the flood of “just in time” experts and training companies” just make the situation worst. Scrum might soon be blamed for failed project that executed with “Scrum Process” couple with waterfall spirit. In fact I have personally come across a few of them. Note that I use “Scrum Process”, because it is just what those people called it.
Thanks. good topic. I like it.
short animated video
Been for a walk, and this normally free the mind a bit and thoughts fly in freely. It occurred to me with the IPO of Rally that this is not a great look for Agile.
All vendors are looking to maximize profit. I was watching a video interview that Gabrielle Benefield did on agile contracts over on infoq. She related the story of the cynical traditional vendor winning a bid over a competing agile bid by going in outrageously cheap. They won the bid. Now I’m doing this from memory now, but the Sales person from the traditional vendor laughed this off saying that we know that they don’t know what they want – we’ll make the money back on variations which will inevitably occur.
I can relate this back to a recent experience whereby pretty much the same thing occur and the customer ended up paying 4 times as much leaving them with a sour taste and us on the frontline coping a heap of flack.
Does this relate to Rally. Maybe – I’m sure they’ll deny this for marketing purposes but inevitably listed companies put profits first oven more so they private companies. I’m afraid that this will cause some dysfunctions to start occurring in Rally. I’ve seen listed company dysfunction. However there are some small number of instances of customer focused listed companies. Will Rally manage to join this group despite the marketing hype.
The article has some very good points, but lost me at the end i.e.
SaFe 13:4 And at the end of the second month of the year, Schwaber said to the world, ‘the black plaque of waterfall is letting go of mine profession.’
13:5 The world replied to Schwaber, the high priest of the scrum tribe of the Agile religion, ‘Do thou not understand that the world is tired of black plague bashing, of my faction vs your faction, it is your own faction that is your worst enemy, so take the good origins of your thoughts, principles and values and go forth, co-operate dont attack, help where help is genuinely wanted and if thy cannot do this, then continue down the slanderous path, collect thine silver coins, certify the slander, and thy shalt risk been part of the next black plague in times ahead.’
Has anyone actually, as in act u ally implemented agile? Or Agile? Truly? Really? How can one implement 4 values and 12 principles? Religion? Doctrine? Disciples? Deification? Is there a DSM course… Deified Scrum Master? On what planet, let alone dimension is a Master a shallow academic? Think about the ETY-MO-LO-GY of master. Seriously. But in a 2 day theoretically attained piece of paper that one can stick on one’s cubicle to call one’s self a higher being than another who has yet to fulfill the doctrine’s calling… in 2 days the agile certifier can turn water into wine…
Right on target. The principles take some effort to put into effect. The values are a way of life.
Ken
Btw, there’s a great book on evidence based management by Jeffrey Pfeffer and Robert Sutton, – Hard Facts, Dangerous Half-Truths, and Total Nonsense: Profiting from Evidence-based Management.
Pingback: Evidence Based Management | www.thought-bubble.co.uk
I must tell you that it’s hard to find your articles in google, i found
this one on 12 spot, you should build some quality backlinks in order to rank your site, i know how
to help you, just type in google – k2 seo tips and tricks
Pingback: Adopting Scrum: Questions for Leadership | OpenView Labs