I’ve argued before about the need for evaluation in our work. This occurs summatively, where we’re looking beyond smile sheets to actually determine the impact of our efforts. However, it also should work formatively, where we’re seeing if we’re getting closer. Yet there are some ways in which we go off track. So I want to talk about iterating and evaluating our learning initiatives.

Let’s start by talking about our design processes. The 800 lb gorilla of ADDIE has shifted from a water flow model to a more iterative approach. Yet it still brings baggage. Of late, more agile and iterative approaches have emerged, not least Michael Allen’s SAM and Megan Torrance’s LLAMA. Agile approaches, where we’re exploring, make more sense when designing for people, with their inherent complexity.

Agile approaches work on the basis of creating, basically, Minimum Viable Products, and then iterating. We evaluate each iteration. That is, we check to see what need to be improved, and what is good enough. However, when are we done?

In my workshops, when talking about iteration, I like to ask the audience this question. Frequently, the answer is “when we run out of time and money”. That’s an understandable answer, but I maintain it’s the wrong answer.

If we iterate until we run out of time and money, we don’t know that we’ve actually met our goals. As I explained about social media metrics, but applies here too, you should be iterating until you achieve the metrics you’ve set. That means you know what you’re trying to do!

Which requires, of course, that you set metrics about what your solution should achieve. That could include usability and engagement (which come before and after, respectively), but most critically ‘impact’. Is this learning initiative solving the problem we’re designing it to achieve? Which also means you need to have a discussion of why you’re building it, and how you know it’s working.

Of course, if you’re running out of time and money faster than you’re getting close to your goal, you have to decide whether to relax your standards, or apply for more resources, or abandon your work, or…but at least you’re doing so consciously. Yet this is still better than heuristically determining that three iterations is arbitrarily appropriate, for example.

I do recognize that this isn’t our current situation, and changing it isn’t easy. We’re still asked to make slide decks look good, or create a course on X, etc. Ultimately, however, our professionalism will ask us to do better. Be ready. Eventually, your CFO should care about the return on your expenditures, and it’ll be nice to have a real answer. So, iterating and evaluating should be your long term approach. Right?

I volunteer for our local Community Emergency Response Team (CERT; and have learned lots of worthwhile things). On a call, our local organizer mentioned that she was leading a section of the train-the-trainers upcoming event, and was dreading trying to make it interesting. Of course I opened my big yap and said that’s something I’m focusing on, and offered to help. She took me up on it, and it was a nice case study in making it meaningful.

Now, I have a claim that you can’t give me a topic that I can’t create a game for. I’m now modifying that to ‘you can’t give me a topic I can’t make meaningful’.  She’d mentioned her topic was emergency preparedness, and while she thought it was a dull topic, I was convinced we could do it. I mentioned that the key was making it visceral.

I had personal experience; last summer our neighbor was spreading the rumor that we were going to have to evacuate owing to a fire over the ridge. (Turns out, my neighbor was wrong.) I started running around gathering sleeping bags, coats, dog crate, etc. Clearly, I was thinking about shelter. When I texted m’lady, she asked about passports, birth certificates, etc. Doh!

However, even without that personal example, there’s a clear hook. When I mentioned that, she mentioned that when you’re in a panic, your brain shuts down some and it’s really critical to be prepared. However, she mentioned that someone else was taking that bit, and her real topic was different types of disasters. Yet my example had already got her thinking, and she started talking about different people being familiar with an earthquake (here in California).

I thought of how when talking with scattered colleagues, they disclaim about how earthquakes are scary, and I remind them that every place has its hazards. In the midwest it could be tornados or floods. On the east coast it’s hurricanes. Etc. The point being that everyone has some experience. Tapping into that, talking about consequences, is a great hook.

That’s the point, really. To get people willing to invest in learning, you have to help people see that they do need it. (Also, that they don’t know it now, and that this experience will change that.). You need to be engaged in making it meaningful!

Again, in my mind learning experience design (LXD) is about the elegant integration of learning science with engagement. You need to understand both. I’ve got a book and a workshop on learning science, and I’ve a workshop at DevLearn on the engagement side. I’ve also got a forthcoming book and an online workshop coming for more on engagement. Stay tuned!

As has become all too common, someone decided to point me to some posts for their organization. Apparently, interest was sparked by a previous post of mine where I’d complained about microlearning. While this one does a (slightly) better job talking about microlearning, it is riddled with other problems. So here’s yet another post about more marketing malarkey.

First, I don’t hate microlearning; there are legitimate reasons to keep content small. It can get rid of the bloat that comes from contentitis, for one. There are solid reasons to err on the side of performance support as well. Most importantly, perhaps, is also the benefit of spacing learning to increase the likelihood of it being available. The thing that concerns me is that all these things are different, and take different design approaches.

Others have gone beyond just the two types I mention. One of the posts  cited a colleague’s more nuanced presentation about small content, pointing out four different ways to use microlearning (though interestingly, five were cited in the referenced presentation). My problem, in this case, wasn’t the push for microlearning (there were some meaningful distinctions, though no actual mention how they require different design). Instead, it was the presence of myths.

One of the two posts opened with this statement: “The appetite of our employees is not the same therefore, we must not provide them the same bland food (for thought).” This seems a bit of a mashup. Our employees aren’t the same, so they need different things? That’s personalization, no? However, the conversation goes on to say: “It’s time to put together an appetizing platter and create learning opportunities that are useful and valuable.”  Which seems to argue for engagement. Thus, it seems like it’s instead arguing that people need more engaging content. Yes, that’s true too. But what’s that got to do with our employees not having the same appetite? It seems to be swinging towards the digital native myth, that employees now need more engaging things.

This is bolstered by a later quote: “When training becomes overwhelming and creates stress, a bite-sized approach will encourage learning.” If training becomes overwhelming and stressful, it does suggest a redesign. However, my inclination would be to suggest that ramping up the WIIFM and engagement are the solution. A bite-sized approach, by itself, isn’t a solution to engagement. Small wrong or dull content isn’t a solution for dull or wrong content.

This gets worse in the other post. There were two things wrong here. The first one is pretty blatant:

There are numerous resources that suggest our attention spans are shrinking. Some might even claim we now have an average attention span of only 8 seconds, which equals that of a goldfish.

There are, of course, no such resources pointed to. Also, the resources that proposed this have been debunked. This is actually the ‘cover story’ myth of my recent book on myths! In it, I point out that the myth about attention span came from a misinterpreted study, and that our cognitive architecture doesn’t change that fast. (With citations.) Using this ‘mythtake’ to justify microlearning is just wrong. We’re segueing into tawdry marketing malarkey here.

This isn’t the only problem with this post, however. A second one emerges when there’s an (unjustified) claim that learning should have 3E’s: Entertaining, Enlightening, and Engaging. I do agree with Engaging (per the title of my first book), however, there’s a problem with it. And the other ones. So, for Entertaining, this is the followup: “advocates the concept of learning through a sequence of smaller, focused modules.” Why is smaller inherently more entertaining? Also, in general, learning doesn’t work as well when it’s just ‘fun’, unless it’s “hard fun”.

Enlightening isn’t any better. I do believe learning should be enlightening, although particularly for organizational learning it should be transformative in terms of enhancing an individual’s ability to perform. Just being enlightened doesn’t guarantee that. The followup says: “Repetition, practice, and reinforcement can increase knowledge.” Er, yes, but that’s just good design. There’s nothing unique to microlearning about that.

Most importantly, the definition for Engaging is “A program journey can be spaced enough that combats forgetting curve.” That is spacing! Which isn’t a bad thing (see above), but not your typical interpretation of engaging. This is really confused!

Further, I didn’t even need to fully parse these two posts. Even on a superficial examination, they fail the ‘sniff test’. In general, you should be avoiding folks that toss around this sort of fluffy biz buzz, but even more so when they totally confound a reasonable interpretation of these concepts. This is just more marketing malarkey. Caveat emptor.

(Vendors, please please please stop with the under-informed marketing, and present helpful posts. Our industry is already suffering from too many myths. There’s possibly a short-term benefit, however the trend seems to be that people are paying more attention to learning science. Thus, in the long run I reckon it undermines your credibility. While taking them down is fun and hopefully educational, I’d rather be writing about new opportunities, not remedying the old.  If you don’t have enough learning science expertise to do so, I can help: books, workshops, and/or writing and editing services.)

As part of the Learning Development Conference that’s going on for the next five weeks (not too late to join in!), there have already been events. Given that the focus is on evidence-based approaches, a group set up a separate discussion room for learning science. Interestingly, though perhaps not surprisingly, our discussion ended up including barriers. One of the barriers, as has appeared in several guises across recent conversations, are the expectations on L&D. Some of them are our own, and some are others, but they all hamper our ability to do our best. So I thought I’d discuss some of these misaligned expectations.

One of the most prominent expectations is around the timeframes for L&D work. My take is that after 9/11, a lot of folks didn’t want to travel, so all training went online. Unfortunately (as with the lingering pandemic), there was little focus on rethinking, and instead a mad rush to get things online. Which meant that a lot of content-based training ended up being content-based elearning. The rush to take content and put it onscreen drove some of the excitement around ‘rapid elearning’.

The continuing focus on efficiency – taking content, adding a quiz, and putting it online – was pushed to the extreme.  It’s now an expectation that with an authoring tool and content, a designer can put up a course in 1-2 weeks. Which might satisfy some box-checking, but it isn’t going to lead to any change in meaningful outcomes. Really, we need slow learning! Yet there’s another barrier here.

Too often, we have our own expectation that “if we build it, it is good”. That is, too often we take an order for a course, we build it, and we assume all is well. There’s no measurement to see if the problem is fixed, let alone tuning to ensure it is. We don’t have expectations that we need to be measuring our impact! Sure it’s hard; we have to talk to the business owners about measurement, and get data. Yet, like other areas of the organization, we should be looking for our initiatives to lead to measurable change. One of these days, someone’s going to ask us to justify our expenditures in terms of impact, and we’ll struggle if we haven’t changed.

Of course, another of our misaligned expectations is that our learning design approaches are effective. We still see, too often, courses that are content-dump, not serious solutions. This is, of course, why we’re talking about learning science, but while one of us had support to be evidence-based, others still do not. We face a populace, stakeholders and our audiences, that have been to school. Therefore, the expectation is that if it looks like school, it must be learning. We have to fight this.

It d0esn’t help that well-designed (and well-produced) elearning is subtly different than just well-produced elearning. We can’t (and, frankly, many vendors get by on this) expect our stakeholders to know the difference, but we must and we must fight for the importance of the difference. While I laud the orgs that have expectations that their learning group is as evidence-based as the rest, and their group can back that up with data, they’re sadly not as prevalent as we need.

There are more, but these are some major expectations that interfere with our ability to do our best. The solution? That’s a good question. I think we need to do a lot more education of our stakeholders (as well as ourselves). We need to (gently, carefully) generate an understanding that learning requires practice and feedback, and extends beyond the event. We don’t need everyone to understand the nuances (just as we don’t need to know the details of sales or operations or…unless we’re improving performance on it), but we do need them to be thinking in terms of reasonable amounts of time to develop effective learning, that this requires data, and that not every problems has a training solution. If we can adjust these misaligned expectations, we just might be able to do our job properly, and help our organizations. Which, really, is what we want to be about anyway.