The Relentless Pursuit Of Improvement And The Role Of Assessment

I believe that if you are in an organization using Scrum, you should know the purpose of a time box, the reason for the Daily Scrum, the difference between a Sprint Backlog item and Product Backlog item, and the other roles, time-boxes, artifacts and rules that bind the Scrum framework together. The “Scrum Guide” contains this knowledge, and the Scrum Open, Professional Scrum Master I, and Professional Scrum Master II assessments measure understanding thereof.

If you are a developer in a Scrum team, you should know such modern engineering practices as test-driven development, acceptance test-driven development, refactoring, code coverage, and automated test harnesses. If you don’t, you may have trouble working with your teammates to build a “done” increment of product every Sprint. The “Scrum Developer Guide,” which is under development, will contain this knowledge. The Professional Scrum Developer assessment measures understanding of these topics.

An assessment is a technique for quantifying something. The best, most objective assessments are repeatable. Doctors use tests to determine how and whether patients are ill. Detectives use forensics to determine who committed a crime. And hiring managers can use Scrum assessments to determine whether an applicant knows Scrum and even knows how to build software as part of a Scrum team.

As Scrum became more widely adopted, I developed techniques to spread the knowledge of what it is and how to use it. My initial efforts were published papers and conference speeches. Next I wrote books like “Agile Software Development and Scrum” and “Agile Project Management with Scrum.”  After that I developed Scrum training courses – that is, the “Certified” Scrum Master course. I then trained others to deliver Scrum training, and I created the Scrum Alliance to scale these efforts and further promulgate knowledge of Scrum. I’m now taking the next step in this journey to improve the quality of Scrum training and consulting by assessing the quality of this training and the results that it produces. I established to make this happen.

One of the first things I did through was to collaborate with Jeff Sutherland to create the first Scrum Body of Knowledge, the “Scrum Guide.” A body of knowledge is a definitive reference on what something is and how it works. The “Scrum Guide” is a 20-page Body of Knowledge outlining the Scrum framework and its components and rules. Two other bodies of knowledge under development are the “Scrum Usage Guide” and the “Scrum Developer Guide.” Both will be of limited size, but I expect that they will soon lead to guides on even more specific topics, such as “Emergent Architecture in Scrum” and “Risk Management In Scrum.”

People will always be able to freely and easily access these guides on the web site. These bodies of knowledge are useful as definitive answers to “what and how” questions about Scrum and as reference sources for papers, discussions, and presentations.

As that work progresses, is also developing assessments, which are ways of quantifying understanding of these bodies of knowledge. The assessments test knowledge at a fundamental, intermediate, and advanced/applied level. The fundamental level is readily served by multiple choice assessments which can be delivered by computers. The intermediate level involves more subtle knowledge that can be best assessed by having experienced Scrum users grade written essays. The advanced level will most likely require all that is included in the fundamental and intermediate level, with the addition of an oral examination by Scrum experts.

The Scrum fundamental assessment (“Scrum Open”) has already been released and is openly available to anyone wishing to test his or her knowledge. This assessment is free. A separate assessment, Professional Scrum Master I, leads to certification and is available for a fee.

When I was with the Scrum Alliance, I had trouble ensuring the quality of the Certified Scrum Trainers and their training materials. This was problematic. The assessment process allows me to assess the quality of various training techniques and materials, as well as the quality of the people delivering the training. People who have no Scrum training average 72% on the Scrum Fundamentals (Scrum Open) assessment. People who have taken the Certified Scrum Master course from the Scrum Alliance average 76%. People who have attended the Scrum in Depth course average 92%. Promisingly, the standard deviation of those with no or Certified Scrum Master training is 12. The standard deviation for those with Scrum in Depth training is 5. From this, I infer that the impact of high-quality training material can be measured. This data is also available here. These types of metrics also help me measure the average score of people who are trained by different trainers. I can then identify and help improve those trainers whose students are below the assessment average. is in conversation with several organizations about private use of the assessments. These organizations wish to know the objectively measured knowledge of people who apply for work or who are already working for them. In general, people who are Certified Scrum Masters are of better skill than others. However, the disparity of knowledge and skill between them is so significant that the simple claim of having attended training is inadequate for hiring and staffing purposes. These organizations will use the assessment as the constant criterion for hiring, staffing, advancement, and remedial training instead.

I expect that as we extend these bodies of knowledge beyond the basic Scrum framework, controversy will erupt. People will openly discuss and argue about what are the best techniques and practices in a Scrum environment. As this occurs, the bodies of knowledge and corresponding assessments will emerge and improve.

Some might ask why has the right to define Scrum bodies of knowledge and to assess how well it is understood. Let me address this question. The individuals involved in these efforts have developed and promoted Scrum since its inception. We not only have the experience and expertise necessary to create these bodies of knowledge, but we also have the contacts necessary to access additional experience and expertise as necessary. has established a Scrum Council to provide formal input, and our discussion boards are a useful source of informal input.

Perhaps as importantly, we are the people who are willing to roll up our sleeves and get this work done. There is no question in my mind that this is necessary and important work, not just for Scrum but for the software development industry in general. I am working to bring to market quality-assured Scrum training and Scrum assessments because I don’t see anyone else doing so with any measure of success.

2 thoughts on “The Relentless Pursuit Of Improvement And The Role Of Assessment

  1. Pingback: Looking Back On More Than 2,500 Scrum Open Assessment Results « Ken Schwaber's Blog: Telling It Like It Is

  2. Pingback: Improving New Product Development Productivity – the Agile Approach | Jay Caplan on Medical Devices

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s