Metrics and Analyses
One of the principles of the Agile Manifesto defines ‘working software as the primary measure of progress.’ What again sounds simple often turns out to be not fully understood. Why is working software actually the most important measure of progress for agile development?
First, we need to clarify what constitutes working software. To stay within scope, we define it along three central criteria:
- The software contains no errors that hinder work
- The software runs as performantly as the work to be done requires
- The software provides real, measurable added value for the user
To make these criteria measurable, there are many possible progress indicators. Here are a few examples:
- Completed tickets / tasks
- Completed story points
- Booked times
- Project duration
- Number of meetings held
- Delivered software versions
Sounds good, right? All of these are measurable criteria that can be recorded and evaluated. Completed tickets indicate how many new features the customer has received. Booked times reveal how much work has already been done on a project. The longer a project lasts, the more progress is visible. The same applies to meetings, as more meetings potentially clarify more issues than fewer meetings. How many software versions have been delivered is almost analogous to the number of completed tickets.
Do you already notice the ironic undertone? We humans tend to lovingly and diligently create metrics, analyses, and dashboards. Whether the whole thing still has a causal connection, we rarely question. But how exactly do you measure working software? Certainly, the number of completed tasks can be a very good indicator of success. However, it should also be considered how many of the completed tasks provide real added value for the customer and user, or how many were implemented without errors at all, i.e., whether it is working software.
To summarize: Metrics are great, but we should move towards critically questioning the causality of our metrics, so as not to end up with a deceptive overall picture or even initiate the wrong countermeasures.
Consistent Pace
A good time to question your own evaluations is always when you notice an improvement. Why? Well, let’s assume a team completes an average of 10 tasks per week (for all those who already recognize that the unit of measure ‘task’ is not suitable at all: correct, I’ll get to that shortly).
After a few weeks, the team first completes 11 and then 12 tasks per week. Has the performance now increased? Or has the team perhaps just started to estimate generally smaller tasks? Have only small tasks come up? Or has the quality of work perhaps decreased and that’s why more tickets are being completed? The exact evaluation is difficult and probably even impossible.
This is due to the following reasons, among others:
- The unit of measure ‘task’ says nothing about the effort, risk, or complexity of the task.
- A decrease in quality may only become noticeable weeks later, especially if there is no continuous delivery and testing. By then, the team’s performance will long be considered to have increased.
- There are no events that would justify increased performance. When performance drops, causes are immediately investigated. Why not when performance has increased?
The Agile Manifesto states in its principles that agile teams should be able to deliver consistent performance over a long period of time. Here’s a small note from me: Performance can also improve, but there should be solid reasons for it. These can include, for example, a better development environment or better tools for delivering software. If an improvement has no apparent cause, there should be utmost vigilance to ensure that the quality of work has not decreased. Often this happens unintentionally. As always: We’re not looking for culprits, but for causes.
In the introduction, I mentioned that the unit ‘Task’ is not optimal. By now, we understand that this unit is not only suboptimal for various reasons, but even fundamentally wrong. There may be other areas where this unit of measurement works very well, but agile software development is not one of them.
Now, however, I’d like to make a slight shift and move from metrics, analyses, causal and acausal relationships to a more technical area.
Technical Excellence
We developers tend to have the following vulnerabilities: ‘Premature Optimization’ and ‘Over Engineering’. The former refers to the tendency to optimize things too early, which on one hand don’t need to be optimized and on the other hand might be deleted anyway. Over Engineering is a natural process in a developer’s life, where they deliver a solution so technically excellent and sophisticated, abstract and generic, that the time for a purposeful AND technically excellent solution would have been much shorter than for the delivered one.
While the Agile Manifesto explicitly demands technical excellence in its principles, it also points out that undone work should be maximized, precisely to limit the aforementioned phenomena.
The task as a developer is to implement customer requirements as precisely and with high quality as possible, without getting lost in unnecessary and non-value-adding details.
The task of a good Product Owner, SCRUM Master or as it is sometimes still called project manager is to keep an eye on whether a team is getting lost in details and at the same time emphasize code reviews, unit testing, etc.
In my view, these two points of the Agile Manifesto, ‘technical excellence’ and ‘simplicity of solution’, should be combined. Something like this:
‘Continuous focus on achieving technical excellence and good design with minimal effort’
Functional Metrics
So what are good metrics by which agile teams can be measured? Well, to determine that, you need to consider what exactly should be measured. If the goal is to monitor consistent pace, the so-called velocity is a proven metric. This measures the number of story points completed per sprint. The aim is to keep this number constant over a longer period of time. If you don’t use a sprint-based approach, such as Kanban, velocity must be determined over self-chosen periods.
The Burn Down Chart, on the other hand, indicates the team’s progress in the current sprint. Here, a constant course is not desired, but rather a constant decrease in the number of open tasks. The Burn Down Chart thus indicates how well the user stories were divided into sub-tasks and how well the minimization of work in progress is functioning.
Of course, these are not the only meaningful metrics, but probably the most widespread. The number of reported bugs, for example, can also be a good indicator of how stable the software product actually is.
Please try this at home or at work!
I’d like to conclude this part of the article series now, before I find myself in over-engineering, and leave you with two tasks to participate in:
- Analyze your evaluations, metrics, and KPIs. Are there acausal relationships?
- Try to implement the next user story as simply as possible. It should still meet the acceptance criteria! What would you normally have done differently, would that have already been too much?
In this spirit: Please try this at home or at work!
