Measure your software but, watch out! put the metrics into context

Posted by Tomás Espeleta on Jun 15, 2012 in Optimyth Software, Quality metrics | 0 comments

I work for a company creating quality assurance software: we give people tools to discover their software structure and inventory, as well as their code’s quality… measuring it.
In my professional life I was really lucky, meeting people who devoted their lives to the study of measurement. But let’s take it seriously.

Last week I read this quote:

“A science is as mature as its measurement tools” – Louis Pasteur

When I read it, I thought: what’s the problem with software development then? Its measurement tools are not that bad at all, they’re quite objective and precise! Why – as a software architect – I’ve got always the feeling that something is wrong, that something is not really under control, even if processes are perfectly defined and mature.

Objective metrics: code artifacts

McCabe, Halstead, LOC, FP they’re all good to measure different code properties (size, complexity, etc.). Any tool, or person knowing the algorithm can compute them, and they can be combined in fancy formulas, resulting in useful indexes and derived metrics.
Even function points, while being more difficult to learn, and sometimes difficult to count… they solve their mission: measuring functional size.
All these metrics have a perfect reproducibility: you give the same requirements specification to a counting expert in Brazil or Korea, and you’ll find more or less the same FPs. And if you use the same tool to measure cyclomatic complexity on the same set of entries, then you obtain a deterministic result. That’s good!

Time and budget

Time-tracking can be really precise (if you really want it to be). You just need to accept the small management overhead of measuring the time spent on your precisely-defined tasks during development.
Can you then imagine something more objective than the budget for a project? I’m not saying that a project is always finished on-time and on-budget. Not at all. What I’m saying is that we can objectively measure how long and how much it finally took. That’s good too!

What is “wrong”, then?

I came to the conclusion that the problem is the misuse of measurement tools and how you look at the resulting metrics. Would you use a scale to measure length? Would you use a screwdriver to fix a nail? Is the woman flying on a magic carpet? Wait! maybe not. Well, in my opinion with software development something like that is happening out there.

Let’s see an example of bad use of a metric, a pretty common one: productivity.

Quoting from Wikipedia:

“Productivity is a measure of the efficiency of production. Productivity is a ratio of production output to what is required to produce it (inputs). The measure of productivity is defined as a total output per one unit of a total input.”

So, normally, we get a size measure (LOC, FP), a time measure… et voila!

Productivity = size / time

What really bothers me is that productivity (high or low) is the base of so many blind business decisions, and I find this fact so deeply wrong that I had to write something on this blog…

Let me explain: productivity is just a metric definition, and it requires interpretation and tuning. It’s easy to imagine productivity in a car factory as “units per hour”. But, is this that easy for software development?

“Lines of code per hour”
Question: Is more productive writing more or less lines of code for the same functionality?

Less. Indeed → Why then productivity is directly proportional to size? Which is more “productive”: a programmer writing a readable code of 1000 lines, or a spaghetti cut & paste coder, writing – faster – the same functionality in 10000 unreadable lines?

Difficult question, maybe. But my answer is clear: the first one is more productive.

But the metric is not wrong by itself, you just need to take into consideration other factors, along with productivity. You need to include code quality KPIs, taking it in serious consideration during business decisions process, even making the formula more complex and including code quality inside the productivity metric definition.

“Function Points per hour”
Question: Is functionality all you need to code?

No, of course not. With new technologies, and new methodologies, you spend so much time on tasks like refactoring, restyling, never mind maintenance projects, where you can spend a whole week to change ONE single byte. Following the formula, all these tasks have very low productivity, even zero productivity. But they can represent most of your business.

So, my point here is really simple (maybe obvious to my measurement-gurus friends): productivity, alone, and based on standard size measurement, is useless. Productivity – and frankly, any metric – is useless, when you use it blindly and out of context.

Conclusions

If you want to measure and use productivity in your business decisions:

1. Be sure that you know how to measure the size of your software development tasks. Do not rely just on any automatic data gathering, but go for a “size, weighted by difficulty”, defining some simple metric that can be:

1. simple and reproducible.
2. customized: do not use standards like LOC or FP
3. based on your artifacts and technology you are using
4. take into consideration your process overhead

2. Do not use just productivity alone: consider the intrinsic quality of your result, and the importance (or perceived value) of your development.

As you can see, I would also suggest being subjective, using surveys and feedback, feelings. Do not mask your decisions behind numbers.

As I said, I work for a company devoted to metrics and data gathering, it could look like I’m saying something against it. Don’t be wrong: you need data, you need tools like ours.

“You can’t manage what you don’t measure”

That’s still true, and Louis Pasteur was with me in this, but that is not enough. Do not misuse metrics, do not hide behind them. Use them wisely.

Meet the author Tomás Espeleta

I'm a software architect, lately I'm interested in algorithms, graph theory, processes and methodologies. Since I was a boy I wanted to change the world with a computer: now computers are everywhere, and they've changed us. I love mountains, skating and a sometimes, a good zinfandel. A geek with a wide range of interests ;)

• Mail
|
• Web
|
• LinkedIn