Introduction
This blog post will introduce the Maturity Matrix I have been developing to help report on the state of Developer eXperience for a given product.
In this first part, I'll explain why I think it's needed, what I report from it, and what those reports mean. In Part II I'll explain how I produce these numbers.
KPIs and OKRs for DevX are hard
Knowing how to describe goals and measure performance against those goals is a challenge for every DevX team.
There are some well established metrics, like "Time to Hello World" (TTHW), or tracking the elimination of support hot spots, but there are limits to how far you can go with this.
The effectiveness of these metrics depends at least a little on the volume of developers onboarding in a given period - particularly in internal DevX and platform work this can be challenging to use on short-timescales. They also don't give as anything like a complete picture of the entire experience.
Many DevX teams reach for the qualitative data at this point, and with good reason - what better indicator of a good experience than a happy customer? In practice, developers are rarely keen to repeatedly answer the same questions in such a way that change can be tracked. Decent sample sizes are hard to achieve, and even reaching the audience at all can be a challenge.
Even if you're successful in collecting all these kinds of data, and find it useful for your own needs, it can often be too complex or fuzzy to make a useful summary for management, or too unpredictable to set goals around. Whether you're asked for Key Performance Indicators (KPIs) or Objectives and Key Result (OKRs), they need to be simple and clear.
There are questions of perception too. Some consumers of the information you publish in planning and management contexts may find it hard to translate it into something they perceive as valuable. Understanding the concrete value of developer experience is a struggle for some, and challenging it is a favourite sport of those who focus only on the length of the feature list.
What I realised quite early on is that when we're talking about planning and targets it's useful to find ways to describe what we do in terms of an easily quantifiable value that can be aggregated in a way that's easy to track, and matches the "feature list" view that others might expect.
Maturity matrices
Maturity matrices are the solution I found to solve these problems.
Before I explain I need to address prior art. I honestly didn't think about the Capability Maturity Model when I started to define my maturity matrices, but I've definitely seen it before and my solution has some strong overlaps, but also some key differences. To be clear, I don't claim to have invented this solution out of the blue, I'm writing this to help others with the same issue, not for Imaginary Internet Points. Likewise, I'm sure plenty of others have found similar solutions, and I'd love to hear about them!
What is a maturity matrix and how do I use one?
The essence of the matrix is the quantification of the "maturity" of the developer experience of some feature or process.
You can report values for current maturity, iteration goal maturity (or period, or quarter or whatever unit you prefer) and the end goal for the maturity that you currently envisage. These maturity values can be aggregated up from very fine levels of detail all the way up to a single headline figure. This is exactly the kind of key metric people like to see, and it can be genuinely informative, so long as both you and your audience understand what it means.
Sadly understanding what "maturity" really means in this context is a little tricky without looking at the fine-grained detail, but let's put off that discussion, because looking at what we can read from these aggregated values will make it a little clearer. We'll explore the finer detail of how I produce this data in Part II of this blog post.
What we report
At the top level of aggregation, we can reduce this matrix down to the following columnar data:
| Product | Prior Maturity | Maturity |
|---|---|---|
| Product Cloud Platform | 40% | 48% |
| Off-The-Shelf Apps Platform | 38% | 25% |
These are nice headline figures - the executive summary. Note, that for reporting purposes I include one or more "prior" measures - I won't repeat this as we break things down, but it is relatively normal practice, and I hope it hints at why this data alone isn't sufficient.
These numbers express two things - the first and most obvious is how far DevX work has progressed towards its established goals. The second is a reflection of the fact that the goalposts move in ways that have real business impacts.
To see where this effect comes from we have to drill down one layer into the data to the next level of aggregation:
| Product | Current Maturity | Iteration Goal Maturity | Goal Maturity |
|---|---|---|---|
| Product Cloud Platform | 1003 | 1037 | 2099 |
| Off-The-Shelf Apps Platform | 255 | 300 | 1021 |
Now we see we're actually reporting a current level, a target level for the iteration/period and an overall target level. We can actually express percentages for all the relationships here.
If we say that c is the current maturity, i is the iteration goal maturity and g is the goal maturity, we can do the following calculations:
| Name | Calculation |
|---|---|
| Current vs Iteration Goal | (c ÷ i) × 100 |
| Current vs Goal | (c ÷ g) × 100 |
| Iteration Goal vs Goal | (i ÷ g) × 100 |
For the most part, I report "Current vs Goal" (as above) but there are use cases for the others. It should be obvious though that if either of the goal states goes up without a change in the current state, then the percentage resulting from those calculations will go down.
If you're not a fan of reporting a falling number to your boss (we've all had that boss at some point!), then you could also choose to express your headline figure as a ratio:
| Product | Prior Maturity | Current Maturity |
|---|---|---|
| Product Cloud Platform | 840:2099 | 1003:2099 |
| Off-The-Shelf Apps Platform | 190:500 | 255:1021 |
It's not such a nice, clean number, but this way we can see the shifting goalposts. Hold that thought for a second, and we'll talk about what this all means about your DevX performance.
Progression and expectations.
Generally speaking, you'd hope that the numbers would be relatively stable, and from time to time tick upwards. When you first start doing DevX they might even go up quickly.
However, at some point, every organisation will see the numbers drop, and then you'll find yourself handling a difficult conversation. Let's now prepare for that eventuality.
Explaining drops in aggregated DevX maturity
I already mentioned that "moving goalposts" mean that you may well see these numbers drop from review to review. That sounds like a very bad message to bring with you, and indeed it can be, but the key is understanding, explaining to those above you and making the appropriate plan as a result.
There are, to my knowledge, three reasons why we can see these maturity numbers, expressed as a percentage, go downwards from one review to the next:
- We decided that we have a higher goal for DevX maturity for some feature, process or customer types than we previously had. This is us moving our own goalposts.
- We removed something that was providing DevX maturity and didn't replace it, but the feature or process it supported is still in place.
- New feature development outpaced the DevX work needed to support it.
Case 1: the happy case!
All of these things deserve to be highlighted, and handily this maturity matrix data does exactly that!
The first case is actually good news - it means we've recognised something we'd previously failed to consider, or that some group of customer-developers is increasing in significance and we've adjusted our plan to account for this.
For example, if we had a customer base that consisted mostly of Java programmers, and we provided a higher level of support (say in the form of a dedicated SDK) to that group we could consider ourselves to have decent maturity. Now imagine that we notice a growing number of developers using Python, yet we don't provide an SDK for python, but rather let Python developers rely on API access alone. Given the growth, we might decide that providing a Python SDK is a goal. This will shift the "Goal Maturity" and possibly the "Iteration Goal Maturity", but until we've done the work the current maturity level won't shift, so the aggregated maturity percentage will drop.
How this information is encoded into the data will be covered in Part II!
Case 2: regression.
If we have a given feature or process and we do something that reduces the quality of the developer experience around it, without also removing the feature or the process (a minefield of its own!) then we have a regression.
An example of this would be removing the tutorial documentation that describes the process of onboarding with our product, whilst still accepting and expecting new developers to be boarded. This is fairly obviously bad. There are subtler cases, but they should be reflected in the numbers too. Again, you'll see how this is built into the numbers in Part II.
Case 3: failure of the DevX process.
This one is the real danger sign you need to look for, and it's where this Maturity Matrix data shines.
If you are adding functionality and processes to your product, but your developer experience work isn't keeping pace with those new features and processes, then you aren't improving your product. Developers trying to make use of those features are going to have a bad time, and that will impact their overall feeling about using your product quite severely.
If you hit this situation it should be a "stop the world" moment. A common first reaction is to assume that the problem occurs because you don't have a large enough DevX team in comparison to your product or platform team. There may well be some truth in this, but before you jump to that conclusion, you should at least consider whether your development organisation as a whole recognises that Developer eXperience is determined by the process used by the whole of that organisation, and not by an end-loaded bolt-on produced by a single team. See DevX is a quality process.