So, just as I’ve been reviewing a book on evaluation (and sharing one issue), I also get my mitts on a copy of an evaluation report. And, yes, I tend to blog about what I’m currently doing (this is learning ‘out loud’ after all!) Further, while I’ll freely admit that I’m not an evaluation expert, I do know a wee bit about methodology, who’s said what, and bad graphics. All of which were in the report. So, here’s a bit more on evaluation.
So, first of all, it’s not a bad report. Will Thalheimer did one at the beginning of the year, and having more is better (a lesson I learned from Don Taylor about AI surveys ;). It asks respondents about their practices. It also includes quotes from noted names (not including me ;), and case studies. The data are interesting. Oddly, it does not mention Will’s survey!
However, there are problems. First, as a survey, responses only characterize those who respond. Given that it’s by a vendor, that likely biases the sample. Also, surveys tend to get folks who want to respond. We get no details on how respondents were even made aware, solicited, nor the actual survey questions (tho’ you can infer). There are also, supposedly, 15 in-depth interviews. Maybe those are the noted names and case studies? To be fair, the results don’t suggest a bias towards those who want to respond, as the indications are that folks still aren’t doing a good job. Which isn’t surprising, though dismaying. Still, there should be qualifications about limitations for considering the results.
Another issue is that the report is ostensibly about Return-On-Investment (ROI). Which I consider to be the area of expertise of Patti and Jack Phillips (rightly or wrongly ;). They have extended the Kirkpatrick-Katzell model explicitly to include ROI. Thus, I find it odd that they receive almost no mention (except for one minuscule print reference). Nor, for that matter, is there a clear definition nor formula for ROI! Another strange omission.
A further omission is any talk of Will. Not only does he have a survey, but he has a worthwhile instrument! At least Phillips & Kirkpatrick (who are better known of course) get a mention, no matter how obscure. Of course, others like Brinkerhoff also don’t get a mention, so probably not a deliberate slight of Will. However, I think helping folks know that you know the players is a sign of credibility. Maybe the academic in me?
There are some hiccups in the formatting, but that’s not a big deal, except that it purports to be official output of said vendor. Which, to me, undermines credibility. If they’ll make mistakes in formal output, does that indicate a lack of diligence in other things they do? I don”t know the answer, but it’s not salutary.
I was given a chance to publicize the report, and I’m sure this post doesn’t count ;). (Once again, my principles are interfering with my success…) I’m not naming names, because the goal is not to shame, but educate. And I think it’s worth knowing what good methodology requires, and that includes references to relevant works and qualifications on what the data shows. There’s lots more on evaluation, but it’s a start.
Leave a Reply