meta-learning Archives

Thinking artificially

21 February 2023 by Clark Leave a Comment

I finally put my mitts on ChatGPT. The recent revelations, concern, and general plethora of blather about it made me think I should at least take it for a spin around the block. Not surprisingly, it disappointed. Still, it got me thinking about thinking artificially. It also led me to a personal commitment.

What we’re seeing is a two-fold architecture. On one side is a communication engine, e.g. ChatGPT. It’s been trained to be able to frame, and reframe, text communication. On the other side, however, must be a knowledge engine, e.g. something to talk about. The current instantiation used the internet. That’s the current problem!

So, when I asked about myself, the AI accurately posited two of my books. It also posited one that as far as I know, doesn’t exist! Such results are not unknown. For instance, owing to the prevalence of the learning styles myth (despite the research), the AI can write about L&D and mention styles as a necessary consideration. Tsk!

The problem’s compounded by the fact that many potential knowledge bases, beyond the internet, have legacy problems. Bias has been a problem in human interactions, and records thereof can also therefore have bias. As I (with co-author Markus Bernhardt) have opined, there is a role for AI in L&D, but a primary one is ensuring that there’s good content for an AI engine to operate on. Another, I argue, is to create the meaningful practice that AI currently can’t, and is likely true for the foreseeable future. I also have yet to see an AI that can create a diagram (tho’ that, to me, isn’t as far-fetched, depending on the input).

I have heard from colleagues who find the existing ChatGPT very valuable. However, they don’t take what it says as gospel, instead they use it as a thinking partner. That is, they’ll prompt it with thoughts they’re having to see what comes up. The goal is to get some lateral input to consider (not take as gospel). It’s a way to consider ideas they may have missed or not seen, which is a valuable role.

At this point, I may or may not use AI in this way, as a thinking (artificially) partner. I’ll have to experiment. One thing I can confidently assert is that everything you read (e.g. here) that is truly from me (i.e. there’s the possibility I will be faked ) will be truly from me. I’m immodest enough to think that my writing is not in need of artificial enhancement. I may be wrong, but that’s OK with me. I hope it is with you, too!

Hyping the news

31 January 2023 by Clark Leave a Comment

I just saw another of these ‘n things you must…if you…’ headlines, and as usual it had the opposite effect they intended. I guess I’m a contrarian, because such headlines to me are an immediate warning. It happened to be in an area I know about, and I hadn’t done any of the necessary things. Yet, I have done the thing they were saying needed the prerequisites. Arguably well (do awards count?). It made me reflect on how we’re hyping the news. Some thoughts…

Yes, I know that such headlines are clickbait. ‘n‘ should be small. Yet when I tried to boil down Upside’s ‘deeper learning’ list for an infographic, it came to 14 items. Inconvenient for hype, I’m afraid, but what I’d put in the white paper. Of course there’s more, but I’m trying to be comprehensive, not ‘attractive’. Similarly, when I created my EEA alignment, I had nine elements. Not because they were convenient for marketing, but because that’s what emerged from the work.

I similarly see lists for ‘the five things’, or the ‘8 things’ (somehow 8 seems to be a maximum, at least for marketing ;). What worries me about these lists is if they’re comprehensive. Is that really all? Have you ensured that they’re necessary and sufficient? Did you even have a process? It took four of us working through months to come up with the eight elements of the Serious eLearning Manifesto. None of the above lists (Manifesto, EEA, deeper learning) are definitive, but they are the result of substantial work and thinking. Not just pulled together for a marketing push.

There are good lists, don’t get me wrong. Ones where people have worked to try to identify critical elements, or good choices based upon principled grounds. Typically, if it’s the case, there are pointers to the basis for these claims. Either there’s someone who’s known for work in the area, or they’re transparent about process. However, there are also lists where it’s clear someone’s just pulled together some random bits. Look for inconsistency, mismatches of types, etc.

In the broader picture, it’s clear that generating fear and outrage and sensationalism sell. I just want to demonstrate a resistance, and prefer a clear argument over a rant. (Here I’m trying to do the former, not the latter. ;) This goes with probably my broader prescription: I do want policy wonks making decisions. I really don’t want simple wrong answers to complicated questions no matter how appealing.

So, my short take is if you know the area, read with a critical eye. If you don’t, look for warning signs, and see what those who do know have to say about it. Caveat emptor. That’s my take on trying to stay immune to the hyping of news.

Learners as learning evaluators

24 January 2023 by Clark 7 Comments

Many years ago, I led the learning design of an online course on speaking to the media. It was way ahead of the times in a business sense; people weren’t paying for online learning. Still, there were some clever design factors in it. I’ve lifted one to new purposes, but also have a thought about how it could be improved. So here are some thoughts on learners as learning evaluators.

The challenge is the result of two conflicting challenges. For one, we want to support free answers on the part of learners. This is for situations where there’s more than one way to respond. For example a code solution, or a proposed social response. The other is the desire for auto-marking, that is independent asynchronous learning. While it’s ideal to have an instructor in the loop to provide feedback, the asynchronous part means that’s hard to arrange. We could try to have an intelligent programmed response (c.f. artificial intelligence), but those can be difficult to develop and costly. Is there another solution?

One alternative, occasionally seen, is to have the learner evaluate their response. There are positive benefits to this, as it gets learners to become self-evaluators. One of the mechanisms to support this is to provide a model answer to compare to the learners’ own response. We did this in that long-ago project, where learners could speak their response to a question, then listen to theirs and a model response.

There are some constraints on doing this; learners have to be able to see (or hear) their response in conjunction with the model response. I’ve seen circumstances where learners respond to complex questions and get the answer, but they don’t have a basis to compare. That is, they don’t get to see their own response, and the response was complex enough not to be completely remembered. One particular instance of this is in multiple response choices where you pick a collection out.

I want to go further, however. I don’t assume that learners will be able to effectively compare their response to the model response. At least, initially. As they gain expertise, they should, but early on they may not have the requisite support. You can annotate the model answer with the underlying thinking, but there’s another option.

I’m considering the value of having an extra rubric that states what you should notice about the model answer and prompts you to see if you have all the elements. I’m suggesting that this extra support, while it might add some cognitive load to the process, also reduces the load by supporting attention to the important aspects. Also, this is scaffolding that can be gradually removed, allowing learners to internalize the thinking.

I think we can have learners as learning evaluators, if we support the process appropriately. We shouldn’t assume that ability, at least initially, but we can support it. I’m not aware of research on this, though I certainly don’t doubt it. If you do know of some, please do point me to it! If you don’t, please conduct it! :D Seriously, I welcome your thoughts, comments, issues, etc.

Don’t make me learn!

10 January 2023 by Clark 1 Comment

In a conversation with a client, the book Don’t Make Me Think was mentioned. Though I haven’t read it, I’m aware of its topic: usability. The underlying premise also is familiar: make interfaces that use pre-existing knowledge and satisficing solutions. (NB: I used to teach interface design, having studied under one of the gurus.) However, in the context of the conversation, it made me also ponder a related topic: “don’t make me learn”. Which, of course, prompted some reflection.

There are times, I’ll posit, when we don’t want employees to be learning. There are times when learning doesn’t make sense. For instance, if the performance opportunities are infrequent, it may not make sense to try to have it in people’s heads. If there’s a resource people can use to solve the problem rather, than learning, that is probably a better answer. That is, in almost any instance, if the information can be in the world, perhaps it should.

One reason for this is learning, done properly, is hard. If a solution must be ‘in the head’ – available when needed and transferring to appropriate situations – there’ll likely be a fair bit of practice required. If it’s complex, much more so. Van Merriënboer’s Four Component Instructional Design is necessarily rigorous! Thus, we shouldn’t be training unless it absolutely, positively, has to be in the head when needed (such as in life-threatening situations such as aviation and medicine).

I’m gently pushing the idea that we should avoid learning as much as possible! Make the situation solvable in some other way. When people talk about ‘workflow learning’, they say that if it takes you out of the workflow, it’s not workflow. I’ll suggest that if it doesn’t, it’s not learning. Ok, so I’m being a bit provocative, but too often we err on the side of throwing training at it, even when it’s not the best solution. Let’s aim for the reverse, finding other solutions first. Turn to job aids or community (learning can be facilitated around either, as well), but stop developing learning as a default.

So, don’t make me learn, unless I have to. Fair enough?

Looking ahead

3 January 2023 by Clark Leave a Comment

A number of people are indicating that 2022 is another year to move on from. And, of course, we do need to move on (as if there were an alternative ;). Still, 2022 was a good year for Quinnovation, and here’s hoping that continues. Here’re some random thoughts looking ahead.

For one, I saw an interesting piece leveraging the financial adage (really: caution) that “past performance is not indicative of future results”. That comes with various investment opportunities; just because they’ve done well in the past doesn’t meant that will continue. The nice twist in the article was to apply it to yourself: if the past year wasn’t a great one, that doesn’t mean you’re going to continue to suffer. Things can get better despite what happened in the past (or worse), though of course taking your own proactive steps is recommended. Indeed, given that for me, 2020 and 2021 were slow years didn’t mean 2022 had to be. Fortunately!

In the broader sense, I think that despite some hiccups, we’re seeing positive trends. For instance, I increasingly see calls for greater attention to evidence-based practices. While that doesn’t mean it’s happening yet, but the notice is hopefully precedes implementation!

We’ve still some legacies slowing us down, of course. I do think that the belief in us as formal reasoning beings will continue to be a barrier. Still, the above clarion call should help us move (however slowly) to right that wrong.

I’m optimistic, by nature (despite being skeptical). Thus, I think we are working our way forward. I reckon I’ll keep working on that, at least. I am continuing with the Learning Development Accelerator, and Upside Learning, as well of course continuing to do Quinnovative things. I’m looking ahead to us having an impact, together!

Happy Holidays and the New Year!

27 December 2022 by Clark Leave a Comment

This year, my traditional Tuesday post means this is the last post of this year. The next will be in 2023! Which means it’s time for reflection, heartfelt thanks, and so on. So here’s some thoughts and wishes for happy holidays and the new year.

First, it’s been a really good year, overall. After two too-quiet years (2020 and 2021), the year has been joyously busy. Almost too, but that beats the alternative! I’ve been fortunate to be working not only with great clients, but also with Matt Richter and team for the Learning Development Accelerator (LDA), and with Amit Garg and the Upside Learning team. Both have been very fulfilling.

I’ve been serving as the co-director of the LDA, and as such helped drive a few of the initiatives. For one, the You Oughta Know weekly webinar series was a blessing! I got to interview some of my heroes in learning such as John Sweller and Rich Mayer, as well as many eminent friends. We also ran the Learning & Development Conference in a new format this year. I think it went well. We’re moving on to new ideas for this coming year (stay tuned).

Serving as Upside Learning’s Chief Learning Strategist has also been a great experience. These are folks who’ve made a welcome serious commitment to learning science. I’m helping them find the balance between rigor and commercial viability. I’ve always recognized the need to strike a pragmatic balance between principle and practicality. Thus, it’s truly ‘hard fun’ to help figure it out. More mischief is afoot (so again, stay tuned).

I’ve had the chance to realize a couple of things. For one, I’ve been fortunate to have the bandwidth to do things like publish books (my most recent also came out this year). I likely wouldn’t have had that if I had a full-time job. It was an enormous source of stress (and not a few bad decisions) to not have the security of such work, particularly when the kids were young and I was the sole bread-winner. Yet, things have turned out for the best.

Another realization is that I love working with folks to find the balance between what theory would suggest and what fits in practice. I like working through these exercises, because I learn, and I think this is where I add unique value. I also like sharing the underlying thinking, because I think we need more of it and it’s hard to scale as an individual contributor. I’m grateful I’ve had the chance for the books and to speak at various venues around the world. Also this blog!

So, thanks to my clients, my partners, and all those who strive to pay attention to what research says and do the right thing. I wish you all the best for happy holidays and the new year. May we continue to learn and grow. Stay curious, my friends!

Meta-reflections

20 December 2022 by Clark Leave a Comment

I was recently pinged about a new virtual world, a ‘metaverse‘ inspired new place for L&D. It looked like a lot of previous efforts! I admit I was underwhelmed, and I think sharing why might be worthwhile. So here are some meta-reflections.

I’ve written before on virtual worlds. In short, I think that when you need to be social and 3D, they make sense. At other times, there’s a lot of overhead for them to be useful that can be met in other ways. Further, to me, the metaverse really is just another virtual world. Your mileage may vary, of course.

This new virtual world had, like many others, the means to navigate in 3D, and to put information around. The demo they had was a virtual museum. Which, I presume, is a nice alternative to trying to get to a particular location. On the other hand, if it’s all digital, is this the best way to do it? Why navigate around in 3D? Why not treat it as an infographic, and work in 2D, leading people through the story? What did 3D add? Not much, that I could see.

My take has, and continues to be, as they say, “horses for courses”. That is, use the right tool for the job. I complained about watching a powerpoint presentation in Second Life (rightly so). Sure, I get that we tend to use new technologies in old ways first until we get on top of the new capabilities. However, I also argue that we can short-circuit this process if we look at core affordances.

The followup message was that this was the future of L&D, and we’d get away from slide decks and Zoom calls, and do it all in this virtual world. I deeply desire this not to be true! My take is that slide decks, Zoom, virtual worlds, and more all have a place. It’s a further instance of get the design right first, then figure out how to implement it. I want an ecosystem of resources.

Sure, I get that such a meta verse could be an integrating environment. However, do you really want to do all your work in a virtual world? Some things you can’t, I reckon, machining materials, for instance. Moreover, we have benefits from being out in the world. There are other issues as well. You might be better able to deal with diversity, etc, in a virtual world, but it’ll disadvantage some folks. Better, maybe, to address the structural problems rather than try to cover them over?

As always, my takeaway is use technology to implement better approaches, don’t meld your approaches to your tech. Those are, at least, my meta-reflections. What are yours?

Conference Outcomes?

24 November 2022 by Clark Leave a Comment

Two months ago, I wrote about the L&D Conference we were designing. In all fairness, I reckon I should report on how it went, now that it’s finished. There are some definite learnings, which we hope to bring forward, both for the conference (should we run it again, which we intend), and for the Learning & Development Accelerator (LDA; the sponsoring org, of which I’m co-director with Matt Richter) activities as well. So here are some thoughts on the conference outcomes.

Our design was to have two tracks (basic and advanced) and a limited but world-class faculty to cover the topics. We also were looking not just to replicate what you get at typical face-to-face conferences (which we like as well), but to do something unique to the medium and our audience. Thus, we weren’t just doing one-off sessions on a topic. Instead, each was an extended experience, with several sessions spread out over days or weeks.

The results of that seemed to work well. While not everybody who attended one of the sessions on a topic attended all, there was good continuity. And the feedback has been quite good; folks appreciate the deep dive with a knowledgeable and articulate expert. This, we figure, is an important result that we’re proud of. If someone misses a session, they can always review the video (we’re keeping the contents available for the rest of the year).

Our social events, networking and trivia, didn’t do quite so well. The networking night did have a small attendance but the trivia night didn’t reach critical mass. We attribute this at least partly to it being a later thought, and not promoting from the get-go.

We struggled a bit with scheduling. First, we spread it across changes in countries that switch to/from daylight savings time. The platform we used didn’t manage that elegantly, and we owe a lot to a staffer who wrestled that into submission. Still, it led to some problems in folks connecting at the right time. On the other hand, having the courses spread out meant we didn’t collide, you could attend any sessions you want (the tracks were indicative, not prescriptive).

The platform also had one place to schedule events, but it was as web page. As a faculty member opined, they wished they could’ve loaded all the sesssions into their calendar with one click. I resonate with that, because in moments when I might’ve had spare bandwidth to attend a session, I’m more likely to look at my calendar rather than the event page. Not sure there’s an easy solution, of course. Still, folks were able to find and attend sessions.

We also didn’t get the social interaction between the sessions we’d hoped, though there was great interaction during the sessions. Faculty and participants were consistent in that perspective. There was a lot of valuable sharing of experiences, questions, and advice.

One thing that, post-hoc, I realize is that it really helps to unpack the thinking. The faculty we chose are those who’ve demonstrated an ability to help folk see the underlying thinking. That paid off well! However, we realize that there may be more opportunities. An interesting discussion arose in a closing event about the value of debates; where two folks who generally agree on the science find something to diverge on. Everyone (including the debaters), benefit from that.

We’re going to be looking to figure out how to do more unpacking, and share the ability to do the necessary critical thinking around claims in our industry. The LDA focuses on evidence-based approaches to L&D. That requires a bit more effort than just accepting status quo (and associated myths, snakeoil, etc), but it’s worth it for our professional reputation.

So those are my reflections on the L&D Conference outcomes. Any thoughts on this, from attendees or others?

Writing books

22 November 2022 by Clark 2 Comments

I write. A lot, obviously (7 books, numerous articles, this blog, white papers, …). As a colleague pointed out, I’m lucky it comes easy. For others, that’s not the case. However, someone recently asked how to get started. As another colleague who just published posted some thoughts on what they learned, I realize it may be appropriate to toss out some thoughts on writing books. (Not least because I’m Editor-in-Chief of LDA Press, which so far has only published my own book, but hope springs eternal… ;)

I know some of the barriers to writing a book, for sure. The overwhelming scope, for one. How do you manage it? Well, like you do all big projects, you break it down. The underlying idea, then an outline, before you ever start writing a chapter or anything. I have a colleague whose supervisor never started writing without first creating a diagram. You really do need to get your idea down. I start with an outline. It won’t stay the same, of course. I’ve moved chunks around, added sections, deleted sections, etc. Not only while creating it, but while writing to it!

Set your expectations appropriately. You should expect it to take months. Not full time, but for practitioners, writing full time isn’t feasible. Certainly for non-fiction. Reward yourself for progress, too. Be easy on yourself! Set small goals: “today, I’ll write section X of chapter Y”. If you don’t make it, it’s ok.

I also have written about what makes a good book. Well-written (that is, easy to read), sensible layout, evidence-based, new perspective. A book shouldn’t be written just to exist, it should have a purpose. You learn a lot from writing a book. JD Dillon, who recently wrote The Modern Learning Ecosystem, documented his learnings. They included that it’s never finished, stories are more fun to write than tech jargon, releasing is harder than writing (depends on how you do it, I’d suggest)., and if you’re not uncomfortable, you’re not going far enough. I added: the value of editors/reviewers, creating a structure first, and nicking away a bit at a time.

You probably should not try to write a book as your first project.With speaking, you should speak within your org or to local chapters, before moving to bigger venues. Same with writing. Start small. Blog posts, or newsletter posts within your org or for your local chapter. Like drawing, I suspect, it’s just keep writing! And, importantly, get feedback! Feedback you can trust. It’s clear some folks have never paid attention to how people perceive their writing!

It sounds like a grind, but there are tangible benefits. First, you get known as someone who has an opinion worth hearing! Further, you may be invited to speak, and certainly have a basis to propose speaking. You may be asked to write more. On the other hand, you’re unlikely to get rich from your book. The old adage applies: you make more money giving it away; it’s a better business card!

Do try to get a good editor. In the publishing world, there are usually several. First, there’s your acquisition editor, who works with you to get a viable proposal to get approval. Then there’s your development editor, who works with you to stay on track and develop a clear narrative with useful examples, diagrams, and more. There’ll be copy-editing, of course, and reading initial proofs if formal. Finally, there’s your marketing editor to help get the word out and build sales.

It’s not for everyone. It’s hard. And, again, you need a unique tangible contribution. If you have one, however, don’t miss the opportunity to share it. There are real benefits. Speak and write about it small, first, to ensure it’s viable, but then, look to write it up. I hope this diatribe about writing books makes sense. Hopefully, it’ll inspire some new ones as well.