As a result of some “advanced training materials” I have been working on, I've been thinking recently about how Scrum is helping us and how we expect it to help us more. This has driven me to deeper thinking on the nature of software development. Over the past couple of Agile conferences I've come across references to the Cynefin framework. For those that have not heard about this approach Cynefin offers a way to look at the nature of organizations and decision making and of understanding levels of complexity as we make decisions.

Cynefin divides the problems into 5 domains:

• 2 ordered domains, simple and complicated
• 2 un-ordered domains, complex and chaotic
• 1 domain called “disordered” when we don't know what domain we are in.

When you are trying to categorize (note: this is one use of the model) the problem space you are working in, you inspect the relationship between cause and effect of the problem space. If the relationship between cause and effect is straight-forward and obvious to all, then you problem is in the simple domain. If the relationship between cause and effect is not obvious, but can be analyzed in advance, then you have a complicated problem. If the cause and effect can only be determined with the benefit of hindsight, then you are in the complex domain, while if there is no obvious relationship between cause and effect, you are in the chaotic domain. Pictorially this is represented as:

Sounds logical, right, but how does this help understand development? It turns out that each of these domains has different approach that helps us best deal with the context:

• Simple Domain. The response in the simple context is to “sense - categorize - respond”. People assess the facts of the situation, categorize them, and then base their response on established practice. If something goes wrong, an person can usually identify the problem, categorize it and respond appropriately. This is the domain of the “best practice”.
• Complicated Domain. The response in a complicated context is “sense - analyze - respond.” This approach often requires expertise - I can detect there is a problem with my car, but I have to take it to a mechanic who can analyze the cause and get the problem resolved. Because the complicated context calls for investigating several options and there may be more than one “correct” solution, identifying “good practice” (not “best practice”) is the best approach. This is the domain of the “expert”.
• Complex Domain. The response in the complex context is “probe - sense - respond.” In a complicated context, at least one right answer exists. In a complex context there may be no known “correct” solution. Instead of attempting to impose a course of action, we must allow the path forward to reveal itself. This means we have to create environments where experiments (probe) allow patterns to “emerge.” This also means increasing levels of interaction and communication between stakeholders so that changing understanding of the situations is worked on and communicated to all.
• Chaotic Domain. The response in the chaotic domain is “act - sense - respond.” In the chaotic domain, our immediate job is not to discover patterns but to stop the pain. We must first act to establish order, then sense where stability is present and from where it is absent, and then respond by working to transform the situation from chaos to complexity, where the identification of emerging patterns can both help prevent future crises and discern new opportunities. This is the domain of rapid response - “novel practice.”

OK, great, but what now? If we look at the problem of software development, we can see we often have a complex problem. When things go wrong, it is often obvious with the benefit of hindsight what happened and why. As Steve McConnell says, “There is a limit to how well a project can go but no limit to how many problems can occur.“ The problem is that we cannot always learn enough to say “in this situation we will fix it in the following way” as there is a different problem the next time around. This is not to say the whole process is complex, but significant parts of it are.

The fallacy of the past is that we have treated software development like it is a complicated problem. We'd analyze and hope to come up with a plan that made sense. We'd then be surprised, time after time, when it didn't work out. With the benefit of hindsight, no pun intended, we can now see why this did not work. We were often dealing with a problem that is in the complex domain.

For example, take something as basic and common as the estimation process. It is pretty clear that if we estimate a piece of work and if we focus on improving our estimates over time our estimates will be often good (see Our Estimates are Terrible!). But we also know that we will be wrong and sometimes in a pretty dramatic fashion. In a software project, this causes problems because it means we are not going to be able to predict our overall plan. But is it really because we have a poor estimate, or more because of the nature of the process? If you analyze what happens you will often find that a problem with an estimate is because of some unforeseen event (change in requirement, understanding of software architecture) not as a result of poor estimating. This is not to say we should not estimate. It only says that we should understand the limitations of the data we are providing and act accordingly.

We can now we can see why the Scrum approach helps. Tools such as iterations with their planning and reviews / retrospectives offer an approach which mimics the “probe - sense - respond” approach of the complex domain. The emphasis on collaboration and communication also help us dealing with the complex environment. It also helps us understand why we cannot just “best practice” the software development process and why even identifying good practices might not be ideal in all situations.

As we work to improve the results of our software development process, we need to remember that a lot of the issues we face are “complex”. This means we need to apply the “complex” tools to help us address the problem.

As a final note I have talked about only one aspect of Cynefin and have simplified the thinking process for a particular application. There are many other aspects that can be brought to bear using this tool. By way of background:

• The framework was developed by Dave Snowden (Cognitive Edge). A good 8min introduction to the Cynefin framework is offered on the Web site. A paper written for the Havard Business Review is also available.
• Cynefin is a Welsh meaning “habitat, place” and it implies a place of multiple belongings of which we are only partially aware: cultural, religious, tribal, etc. These “belongings” affect our decision processes.

In addition I have leaned heavily on the work of others (eg Dave Snowden mentioned above, John Helm (Agile 2013 Conference), Simon Bennett (Agile 2011 Conference)) to help me pull this together from the perspective of software development. Brilliance is a result of their thinking. Errors are mine.