Spurious certainties

Monday, 30 November 2009

Black or white. Yes or no. Specific. Measurable. Defined. All very useful things, in their place. All worth considering, at times. But they shouldn’t be allowed to define our approach in its entirety, because life simply isn’t as simple as that. We need to be comfortable with shades of grey in management.

On most scales, I’d consider myself to be a pretty measurement-based person. My degree was in economics (and, frightening though it is to admit, I rather enjoyed the statistics module!), and a significant chunk of my career involved work in the finance function, and included training with CIMA. So I understand the simple pleasures of accurate, measurable things, but I do get slightly frustrated when people aim to measure the immeasurable.

ROI (return on investment) is one of those things that is often used in a glib fashion to reassure people that they are spending their money wisely, and that the benefits of any expenditure significantly outweigh the actual expenditure. Of course, as a trainer, I’d love to be able to confirm to all of my clients that this is the case, but I’m honest enough to admit that (generally) I can’t - and others (generally) can’t either.

Below, I’m going to offer two training scenarios - broadly hypothetical, but based in reality, and show how it is (generally) reasonable to measure ROI in one case, and (generally) unreasonable to measure it in the other case.

The production line team

In most decent-sized manufacturing facilities these days, significant amounts of real-time data is available on team performance. In one part of my career, some of the most vital measures of performance were as follows:

• Line downtime
• Raw material utilisation
• Staffing costs
• Maintenance and repair costs
• Output rate

Different production lines and different teams performed differently, and data was available on a continuous basis, cut pretty much whichever way you wanted.

I would argue that, for a shift team running a production line, it would be relatively easy (although not perfect) to measure the impact of training. There would be enough control data (un-trained teams, history, other lines and so on), and enough rapidly measurable variables to say with reasonable certainty what the effect of training has been.

The sales team

Initial assumptions about sales teams tend to focus on how easily their results can be measured, and, having worked on sales teams myself, I wouldn’t begin to deny this. However, simply because something is easy to measure doesn’t mean that it is easy to understand what affects it. It is easy to measure rainfall in my garden, but pretty tricky to perfectly understand the causes of it.

Sales teams, depending on the sophistication of the organisation, tend to be measured on (a combination of) sales volume, sales value, gross profit, or margin. These, as with the production measures, can be collected frequently, and assigned to specific teams of people. However, there are problems.

To say that a training course followed by an increase in volume/turnover/profitability means that the training course has worked, and that you can work out the ROI of the investment in training by comparing it to the additional profits generated is, at best, misleading. In all but the simplest of cases, there are too many other factors that impact sales performance (outside the skills, abilities, knowledge and attitudes of the team members), and too ill-defined a period of time over which an effect is caused, for any measure of ROI to be truly supportable by anyone who really understands how the world works.

--

The fundamental principle behind measuring the results of training is to what extent “ceteris paribus” applies - to what extent does everything else remain the same, so that training is the only input change and therefore can be the only factor causing a change in results.

It’s rare, in the real world, for ceteris paribus to apply. We have to live with, cope with, and thrive with, the shades of grey that exist, and be comfortable with a lack of certainty.

--

Be very careful when people offer you certainties - are they really, factually certain, or are they being offered as a comfort blanket, or, worse, hiding the reality. Ask yourself why someone might offer you a spurious certainty.