Archives for June 2024

Diving or surfacing?

25 June 2024 by Clark Leave a Comment

In my regular questing, one of the phenomena I continue to explore is design. Investigating, for instance, reveals that, contrary to recommendations, designers approach practice more pragmatically. That’s something I’ve been experiencing both in my work with clients and recent endeavors. So, reflecting, are and should folks be diving or surfacing?

The original issue is how designers design. If you look at recommendations, they typically recommend starting at the top level conceptualization and work down, such as Jesse James Garrett’s Information Architecture approach (PDF of the Elements of User Experience; note that he puts the highest level of conceptualization at the bottom and argues to work up). Empirically, however, designers switch between top-down and bottom-up. What do I do?

Well, it of course depends on the project. Many times (and, ideally), I’m brought in early, to help conceptualize the strategy, leveraging learning science, design, organizational context, and more. I tend to lead the project’s top-level description, creating a ‘blueprint’ of where to go. From there, more pragmatic approaches make sense (e.g. bringing in developers). Then, I’m checking on progress, not doing the implementation. I suppose that’s like an architect. That is, my role is to stay at the top-level.

In other instances, I’m doing more. I frequently collaborate with the team to develop a solution. Or, sometimes, I get concrete to help communicate the vision that the blueprint documents. Which, in working with an unfamiliar team, isn’t unusual. That ‘telepathy’ comes with getting to know folks ;).

In those other instances, I too will find out that pragmatic constraints influence the overarching conceptualization, and work back up to see how the guidelines need to be adapted to account for the particular instance. Or we need to deconnect from the details to remember what our original objective is. This isn’t a problem! In general, we should expect that ongoing development unearths realities that weren’t visible from above, and vice versa. We may have good general principles, (e.g. from learning science), but then we need to adapt them to our circumstances, which are unlikely to exactly match. In general, we need to abstract the best principles, and then de- and re-contextualize.

I find that while it’s harder work to wrestle with the details (more pay for IDs! ;), it’s very worthwhile. What’s developed is better as a result of testing and refining. In fact, this is a good argument about why we should iterate (and build it into our timelines and budgets). It’s hubris to assume that ‘if we build it, it is good’. So, let’s not assume we can either be diving or surfacing, but instead recognize we should cycle between them. Start at the top and work down, but then regularly check back up too!

Learning Debt?

18 June 2024 by Clark Leave a Comment

In our LDA conversation with David Irons, User Experience (UX) Strategist, for our Think Like A…series, he mentioned a concept I hadn’t really considered. The concept is ‘design debt’, as an extension of the idea of ‘tech debt’. I was familiar with the latter, but hadn’t thought of it from the UX side. Nor, the LXD side! Could we have a learning debt?

So, tech debt is that delta between what good technology design would suggest, and what we do to get products out the door. So, for instance, using an algorithm for sorting that’s quicker with small numbers of entries but doesn’t handle volume. The accrued debt only gets paid back once you go back and redesign. Which, too often, doesn’t happen, and the debt accumulates. The problems can mean it’s difficult to expand capabilities, or keep performance from scaling. I think of how Apple OS updates occasionally don’t really add new features but instead fix the internals. (Hasn’t seemed to happen as much lately?)

Design debt is the UX equivalent. We see expedient shortcuts or gaps in the UX design, for instance. As Ward Cunningham, an agile proponent, says:

Design debt is all the good design concepts of solutions that you skipped in order to reach short-term goals. It’s all the corners you cut during or after the design stage, the moments when somebody said: “Forget it, let’s do it the simpler way, the users will make do.”

It’s a real thing. You may experience it when entering a phone number into a field, and then hear it’s not in the proper format (though there was no prior information about what the required format is). That’s bad design, and could (and should) be fixed.

This could be true in learning, too. We could we have ‘learning debt’. When we make practice (and I should note for previous and future posts that includes any assessment where learners apply the knowledge we’ve provided) about knowledge instead of application of knowledge, for instance, we’re creating a gap between what they’ve learned to do and what they need to do. That’s a problem. Or when we put in content because someone insists it has to be there rather than a designer deciding it’s necessary for the learning. Which adds to cognitive load and undermines learning!

How often do we go back and improve our courses? If we’re offering workshops or some other instruction, we can adapt. When we create elearning, however, we tend to release it and forget it. When I ask audiences if they have any legacy courses that are out of date and unused but still hanging around their LMS, everyone raised their hands. We may update courses whose info has changed, but how many times do we go back and redo asynchronous courses because we’ve tracked and have evidence that it’s not working sufficiently? Yes, I acknowledge it happens, but not often enough. (*cough* We don’t evaluate our courses sufficiently nor appropriately. *cough*)

Ok, so everyone makes tradeoffs. However, which ones should we make? The evidence suggests erring on the side of better practice and less content. Prototyping and testing is another step that we can take to remove debt up front. With UX, lacks in design early on cost more to fix later. We don’t typically go back and fix, but we can and should. Better yet, test and fix before it goes live. Another way to think about it is that learning debt is money wasted. Build, run, and not learn, or build, test, and refine until learning happens?

There are debts we can sustain, and ones we can’t. And shouldn’t. When our learning doesn’t even happen, that’s not sustainable. Our Minimum Viable Product has to be at least viable. Too often, it’s not. Let’s ensure that viable means achieves an outcome, eh? It might not be optimal improvement, or as minimum in time as possible, but at least it’s achieving an outcome. That’s better than releasing a useless product (despite no one knowing). , even if we get paid (internally or externally). What am I missing?

Reflecting on adaptive learning technology

11 June 2024 by Clark 1 Comment

My last real job before becoming independent (long story ;) was leading a team developing an adaptive learning platform. The underlying proposition was the basis for a topic I identified as one of my themes. Thinking about it in the current context I realize that there’re some new twists. So here I’m reflecting on adaptive learning technology.

So, my premise for the past couple of decades is to decouple what learners see from how it’s delivered. That is, have discreet learning ‘objects’, and then pull them together to create the experience. I’ve argued elsewhere that the right granularity was by learning role: concepts are separate from examples, from practice, etc. (I had team members participating in the standards process.) The adaptive platform was going to use these learning objects to customize the sequence for different learners. This was both within a particular learning objective, and across a map of the entire task hierarchy.

The way the platform was going to operate was typical in intelligent tutoring systems, with a twist. We had a model of the learner, and a model of the pedagogy, but not an explicit model of expertise. Instead, the expertise was intrinsic to the task hierarchy. This was easier to develop, though unlikely to be as effective. Still, it was scalable, and using good learning science behind the programming, it should do a good job.

Moreover, we were going to then have machine learning, over time, improve the model. With enough people using the system, we would be able to collect data to refine the parameters of the teaching model. We could possibly be collecting valuable learning science evidence as well.

One of the barriers was developing content to our specific model. Yet I believed then, and still now, that if you developed it to a standard, it should be interoperable. (We’re glossing over lots of other inside arguments, such as whether smart object or smart system, how to add parameters, etc.) That was decades ago, and our approach was blindsided by politics and greed (long sordid story best regaled privately over libations). While subsequent systems have used a similar approach (*cough* Knewton *cough*), there’s not an open market, nor does SCORM or xAPI specifically provide the necessary standard.

Artificial intelligence (AI) has changed over time. While evolutionary, it appears revolutionary in what we’ve seen recently. Is there anything there for our purposes? I want to suggest no. Tom Reamy, author of Deep Text, argues that hybrids of symbolic and sub-symbolic AI (generative AI is an instance of the latter) have potential, and that’s what we were doing. Systems trained on the internet or other corpuses of images and/or text aren’t going to provide the necessary guidance. If you had a sufficient quantity of data about learning experiences with the characteristics of your own system, you could do it, but if it exists it’s proprietary.

For adaptive learning about tasks (not knowledge; a performance focus means we’re talking about ‘do’, not know), you need to focus on tasks. That isn’t something AI really understands, as it doesn’t really have a way to comprehend context. You can tell it, but it also doesn’t necessarily know learning science either (ChatGPT can still promote learning styles!). And, I don’t think we have enough training data to train a machine learning system to do a good job of adapting learning. I suppose you could use learning science to generate a training set, but why? Why not just embed it in rules, and have the rules work to generate recommendations (part of our algorithm was a way to handle this)? And, as said, once you start running you will eventually have enough data to start tuning the rules.

Look, I can see using generative AI to provide text, or images, but not sequencing, at least not without a rich model. Can AI generate adaptive plans? I’m skeptical. It can do it for knowledge, for sure, generating a semantic tree. However, I don’t yet see how it can decide what application of that knowledge means, systematically. Happy to be wrong, but until I’m presented with a mechanism, I’m sticking to explicit learning rules. So, where am I wrong?

What I’m up to

4 June 2024 by Clark Leave a Comment

Ok, so it’s been a wee bit too much about me (my books, themes), yet it occurs to me that I should document what I’m doing. (Which I’ve done before, but this is looking forward, too.) Not just for me (though it helps ;), but it’s because I realized my thinking other than books is actually getting spread out in various places. So, here’s what I’m up to…

Mostly, it’s centering around applying the cognitive and learning sciences to the design of solutions. In a variety of ways, of course. I’ve been working with Upside Learning, serving as their Chief Learning Strategist. They want to do more than pay lip service to learning science (which I laud). I’m working with them on evangelism, internal development, and more. I’m also working with Elevator 9, in this case as advisor. They’re a platform solution to complement live events, again doing so in alignment with our brains. I’m also serving as co-director of the Learning Development Accelerator. That’s a society focused on evidence-informed L&D, and we explore what this approach means in practice. In each, I’ve been advancing my own understanding, and sharing the learnings.

So, at LDA, you can find our podcasts, blog posts (some of which are free to air!), and some programs (some likewise). For members, we’re running some internal programs as well. I’ve been pleased to augment my previous program on You Oughta Know with this year’s YOK Practitioner, where I get to interview some really amazing people. Then there’s also the Think Like A…series, where we talk to representatives of adjacent fields we (should) be plagiarizing. Then there are workshops, and we’re always developing more things.

At Elevator 9, while most of the work is behind the scenes, I did author, and David Grad (the CEO) read and taped, a series of ‘liftologies’. These are short videos talking about the learning science that goes into their offering. When they redo the website, they’ll be easy to find, but right now they’re visible through the E9 LinkedIn page posts.

Upside Learning, on the other hand, has been proactive. They do a podcast with the CEO, Amit Garg (yes, I’ve been on it). They have a blog (and I’ve written some for them). I’ve also done some quick videos on myths. In addition, I’ve written some of their ebooks (topics like impact, microlearning, scenarios). And, of course, some webinars as well. These continue.

All this in conjunction with continuing as Quinnovation! I continue with a few clients, on a limited basis. These, of course, are not public, though the thoughts can percolate out (e.g. in this blog). I’m still doing some events, mostly virtually. For instance, I’ll be talking about the alignment between effective education and engaging events at LXDCon on Tues the 11th (at 7AM PT ). I’ll also be at DevLearn and Learning 2024.

That’s all I can think of at the moment. There’s more in the offing, of course. But for now, that’s what I’m up to. This blog may be (more than) enough, but the other sites prompt different thinking. They’re worth knowing about on their own, too! If you’re interested, these are places to either become evidence-based, apply it, or get it done. Obviously, it’s something I think is important for our industry. (As is knowing the human information processing loop, which I’ve made freely available.) Whatever you do, however you do it, please do avoid the myths and apply the science.