As a result of some “advanced training materials” I have been working on, I've been thinking recently about how Scrum is helping us and how we expect it to help us more. This has driven me to deeper thinking on the nature of software development. Over the past couple of Agile conferences I've come across references to the Cynefin framework. For those that have not heard about this approach Cynefin offers a way to look at the nature of organizations and decision making and of understanding levels of complexity as we make decisions.
Cynefin divides the problems into 5 domains:
When you are trying to categorize (note: this is one use of the model) the problem space you are working in, you inspect the relationship between cause and effect of the problem space. If the relationship between cause and effect is straight-forward and obvious to all, then you problem is in the simple domain. If the relationship between cause and effect is not obvious, but can be analyzed in advance, then you have a complicated problem. If the cause and effect can only be determined with the benefit of hindsight, then you are in the complex domain, while if there is no obvious relationship between cause and effect, you are in the chaotic domain. Pictorially this is represented as:
Sounds logical, right, but how does this help understand development? It turns out that each of these domains has different approach that helps us best deal with the context:
OK, great, but what now? If we look at the problem of software development, we can see we often have a complex problem. When things go wrong, it is often obvious with the benefit of hindsight what happened and why. As Steve McConnell says, “There is a limit to how well a project can go but no limit to how many problems can occur.“ The problem is that we cannot always learn enough to say “in this situation we will fix it in the following way” as there is a different problem the next time around. This is not to say the whole process is complex, but significant parts of it are.
The fallacy of the past is that we have treated software development like it is a complicated problem. We'd analyze and hope to come up with a plan that made sense. We'd then be surprised, time after time, when it didn't work out. With the benefit of hindsight, no pun intended, we can now see why this did not work. We were often dealing with a problem that is in the complex domain.
For example, take something as basic and common as the estimation process. It is pretty clear that if we estimate a piece of work and if we focus on improving our estimates over time our estimates will be often good (see Our Estimates are Terrible!). But we also know that we will be wrong and sometimes in a pretty dramatic fashion. In a software project, this causes problems because it means we are not going to be able to predict our overall plan. But is it really because we have a poor estimate, or more because of the nature of the process? If you analyze what happens you will often find that a problem with an estimate is because of some unforeseen event (change in requirement, understanding of software architecture) not as a result of poor estimating. This is not to say we should not estimate. It only says that we should understand the limitations of the data we are providing and act accordingly.
We can now we can see why the Scrum approach helps. Tools such as iterations with their planning and reviews / retrospectives offer an approach which mimics the “probe - sense - respond” approach of the complex domain. The emphasis on collaboration and communication also help us dealing with the complex environment. It also helps us understand why we cannot just “best practice” the software development process and why even identifying good practices might not be ideal in all situations.
As we work to improve the results of our software development process, we need to remember that a lot of the issues we face are “complex”. This means we need to apply the “complex” tools to help us address the problem.
As a final note I have talked about only one aspect of Cynefin and have simplified the thinking process for a particular application. There are many other aspects that can be brought to bear using this tool. By way of background:
In addition I have leaned heavily on the work of others (eg Dave Snowden mentioned above, John Helm (Agile 2013 Conference), Simon Bennett (Agile 2011 Conference)) to help me pull this together from the perspective of software development. Brilliance is a result of their thinking. Errors are mine.