So I saw a twitter thread pointing to an argument about how ROI is dead. And, well, that’s largely okay with me. However, the trigger for the post was from the results of Chief Learning Officer 2020 State of Learning report. And, when I saw them, I saw some problems. The question is whether we’re measuring impact, or not. I’d like to go through them and evaluate each.
(Back to my usual prose, as I need visual support for this. ;)
So, in the report, they indicated that the respondents indicated the demonstrated impact of training in these ways:
- General training output data
- Training output data aligned with corporate initiatives
- Learner satisfaction with training
- Employee satisfaction with training availability
- Employee engagement
- Business impact
- Employee performance data
- Planned to actual budget, expense, revenue data for training group
- Stakeholder satisfaction with training data
- ROI measures
- Net promoter score
- Employee satisfaction with company data
Yikes! Some of these are problematic at best. Let’s look at why some of these might not be good measures of impact. And, let’s be clear; impact should be about positively affecting the organization in a meaningful way. Moving needles like fewer errors, more revenue, reduced costs, happier employees and customers, etc.
So, first, what is general training output data? If it’s like what I saw in (then) ASTD’s State of the Industry report, it’s metrics like employees served per L&D employee or cost/seat/hour for training. Which might a useful measure of efficiency, if you can come up with a principled basis for what a good number would be, and then see if you’re above or below that. Unfortunately, what people do is just compare themselves to the industry average. Is that a good indicator? How do you know? Do you want to be just ‘better than average’?
Then, training output data aligned with corporate initiatives. Again, hard to say what this means (and I can’t seem to find the report). However, it sounds like it’s still efficiency, just doing that for things the business thinks are important.
And we go worse: learner satisfaction with training? Er, research I’ve read and heard cited (I think it’s from Salas, et al, but memory fails) says that’s not valid. There’s a .09 correlation between what learners think of learning impact, and it’s actual impact. That’s zero with a rounding error. That’s all about making learning ‘fun’ (instead of ‘hard fun’). Yes, you do want them to think it’s also been a good experience if you’re focusing on LXD, but that’s secondary.
Similarly, with satisfaction with training availability. What’s that matter? That’s not impact!
Some good things buried here: employee engagement should be good; more engaged employees is a good thing. As long as it’s not at the cost of something else, like, say, impact? And business impact is obviously good, as is employee performance data. Presumably positive business impact, and employee performance improving.
Planned to … stuff is all about efficiency again. And that’s ok, but only after impact. Otherwise, well, we’re not costing too much…?!?
Satisfaction again not good.
And, to the original point of the article. ROI? Yes, what it costs you to move a needle should be less than the cost of what the needle was costing you. However, I could be doing things that return the biggest ROI without doing the most important things. They can be different (e.g. a small program with a better ROI but less overall impact). So it’s only secondary.
Finally, employee satisfaction with company data? I have no idea what that means? But, again, ‘satisfaction’ isn’t really meaningful unless it’s based on real impact.
I’ve complained before about L&D measurement. Here it is, right in front of us. The answer to the question of whether we’re measuring impact or not appears to be ‘mostly not’. We’re still (largely) measuring the wrong things. And we wonder why we don’t have credibility. Please, please, start designing to improve measurable gaps, and then actually improve the outcome. Otherwise, you’ve no idea whether that bum in that seat for an hour is doing the organization any good, versus just not costing too much relative to the industry average.
I’ve browsed articles on CLO before and found either the views they express – or at least the people to whom they give a platform – to often be problematic and counter to research.
Though I have to confess that I’ve written for them! Hopefully better grounded than those you’ve seen…
It’s an opinion.
Did you ask the respondents “why” they think about impact in the way they described?
I find it better to walk on boyth sides of the street.
I couldn’t ask them why, as I didn’t conduct the survey. I’m reacting to what I’ve seen elsewhere and what on principle makes sense.
The company defines, based on their vision, mission and strategy, in which direction they want to go and provides a framework. To measure it needs hard facts… measurable and comparable. Then goals for single resources, roles, departments, subsidiaries and the whole company needs to be set (KRA, KPI).
A kind of second layer would be on how these impact each other to allow steering based on hard facts (and a lot of data points). OKRs could define on how to achieve the goals.
As a next step incorporating the results in improvement plans, Business Modelling and product development. And HR plays an also an important role in this.
Another layer would be, even fuzzy and hard to measure, behavior, meaning, what people think. Personal thoughts about “do we provide enough trainings” or “do we provide spot on training, even, if not overall many trainings” could be a very personal opinion based on experience and expectations.
This should be tracked separately to be able to deep dive into the data to not miss possible relations between hard and soft values. It’s a bit like a “just in case” data collection. And sometimes patterns appear at places you don’t expect.
I remember a customer we always had to chase to pay the bills. No meeting with the customer fixed it.
Analyzing past data on customer satisfaction, who had been the Sales Reps in the past, which trainings were moved/cancelled in the past, training survey results, lost opportunities and even, where the customer worked before revealed, that something went wrong in his previous company, which had been also our customer. And the same Sales Rep and the same Admin that created this situation in the past (that never had been fully fixed), where now the ones responsible for him again. Situation is now fixed, but without the combination of hard and soft facts we wouldn’t have been able to.
Hope not too much of above is rubbish :D… Just my experience… But I would be very interested in getting to know which data points (hard/soft) you consider as crucial, less important or meaningless and how do you measure those (direct or per relation).
Ingo, quite the response! Interestingly, the way I see it (and I had to look up OKRs ;), is that you start with OKRs, and those drive KPIs. E.g. strategy down to tactical measures. I think the fuzzy stuff about opinions matters, when you’re evaluating subjective experience: employee experience, customer experience. And there are ways to collect that in valid ways. But to your key question, the data points, in my opinion they are driven by the change you’re trying to achieve. But those should be tied to performance gaps, and the data that should be used are the evidence of the gap and how you’ll know it’s fixed. Too many errors in manufacturing? That’d be the errors per (time, piece, etc). Too long a sales cycle? Then time to close. And so on. If it’s about customer experience, it could just be a satisfaction rating. But you should have a performance you’re trying to improve, then you find the cause, and if it’s a skill gap, then it’s a course. Otherwise it might be a job aid, or a change in incentives, or messaging campaign, or… And then you can measure the impact of those. Hope this helps.