Learnlets

Secondary

Clark Quinn’s Learnings about Learning

Timing matters

by Leave a Comment

There are many phenomena associated with time. I know it was such a topic of my late friend Jay Cross that he named his vehicle for agency Internet Time. A few phenomena have arisen in my thinking, and it’s time to just review some of the reasons why timing matters.

So, for one, things seem to come in waves. It’ll be too quiet, and then too busy; too frenetic a rhythm. I might prefer a smoother rate, but…it’s not something we get to control. (At least, individually; societally we have chosen speed over quality, what with requirements for short-term returns.) Now, I do my best to cope, using the downtimes to work on background tasks. So that, of course, when it’s crunch time I feel like I can still function.

That was also true when I’ve struggled. For one, when I had back pain (before I got surgery), I had good days and bad. My solution was to get things done on the good days, so I was okay on the bad ones. That again is a coping strategy.

Overall, things are going faster than we’d like. We are dealing with increasing change, and increasing information. Things like Harold Jarche’s Personal Knowledge Mastery (PKM) are tools we increasingly need to use, to be able to adapt.

Timing matters in other ways, too. Some times the time has to be right for an idea to stick. I regularly tell the story about how I mentioned the concept of ‘explorable interfaces’ (from Jean-Marc Robert) to my PhD supervisor, to no avail. Which surprised me, because our lab was looking at usability! However, I brought the concept up a few years later, and suddenly it was a great idea! Same idea, but right time.

It’s true for learning as well. It’s not something we can rush! Yet we try to have an event and then move on, not recognizing learning takes time. We need repeated reactivation and feedback for learning to occur, yet that’s seldom the design of our organizational solutions. Yet, technology gives us the ability to address this, if only we have the will.

And, we may need to make sure it’s the right time. We need to understand why something’s relevant to us before we’re ready for the message. That’s something I think we don’t focus on enough, hence my most recent tome.

We live in interesting times, as the saying has it. There’s little reason to believe things will slow down, even if they should. So, we need to be able to deal with it. Personally, and for our learners. Timing matters, as does how we deal with it. What are your strategies?

Standing up…

by 5 Comments

…and I won’t back down. Ok, so this is a little off my usual thread, but it does have some learning in it. What I’m talking about is using your attention and your money as a way to express your values. It’s what I’m increasingly doing, and there’re lessons in it. So let’s talk about standing up for what you believe in.

It may be that I’ve stood too much on principle in the past, and paid the price. I left a (probably) secure position at a university to come back to the US to be closer to our aging parents. A job at what was positioned as a secure startup appeared to be a good choice..but I didn’t properly account for ego and greed. I even was a bit cheeky about a possible position, to my long-term shame. Consulting then, I joke, went from a euphemism for ‘unemployed’ to a way of life. I’m fortunate, that despite my lack of business nous, my curiosity and inclination to share learnings has proven to be moderately valuable. Somehow, this hasn’t been enough to dissuade me.

As I theoretically get wiser, I’m being more forthright. I’m relinquishing my accounts on platforms that have demonstrated a lack of accountability, for instance. I’ve left a few places in the past few years. I stay on LinkedIn, because it’s not awful (though getting worse), and it’s the place where folks connect for business. I’m on a few other social networks, one that is built to be able to stay independent, and one that, so far, is seeming to have good principles. That latter one I’m willing to abandon if that changes.

I’m also avoiding technologies with misrepresentation, and calling out such claims. Not always, of course, I want to educate, not punish. Still, I strive to let what science tells us to serve as a guide, not what folks want you to believe. Their intentions may be simply misguided, or worse, they may not care. It’s important to be careful, which is why we (Matt Richter and I, e.g. the LDA) wrote the research checklist, for instance. (May require membership, but it’s free!) I even avoiding indulging in an opportunity to watch an activity I enjoy, because it was part of a trend I think is harmful overall (e.g. supporting increasing compartmentalization).

I’m also shifting my purchasing. I’m trying to shop more local, and use sources that aren’t aligned with the most problematic providers. This isn’t always easy, as the ‘long tail’ means certain things are hard to come by. There are consequences, including paying more, and doing with less. Tradeoffs.

Similarly, I try to do business only with those who have approaches I favor. For instance, I’ve avoid positions where I receive compensation for promoting a product, because that would bias my recommendations. I (perhaps wrongly) believe that having that unbiased opinion (and stating when I have conflicts) is of value. I am now am working with Elevator 9, but that’s because they have demonstrated that they care about learning science.

None of this is perfect. For one, there are barriers to completely shifting. Some services you just can’t get without aligning with one platform or another. Certain products are basically just impossible to source any other way. Not everyone you know and care about will go along. You do what you can, and live with the results.

There’s learning from this. It’s harder than not. I’ve learned that trusting what people say, particularly those with vested interests, isn’t a good bet unless they’ve earned your trust in other ways first. Acquisitions, for one, rarely go the way that the acquirers promise! Also, it’s pretty obvious that this stance is an effort that not everyone can, or is willing to, make. There’s risk, for instance. On the other hand, it’s rewarding. You do feel better that you’re doing things to support what you believe.

Note that I’m being relatively opaque about my intentions. I think they’re pretty obvious, but still, the principles hold regardless; vote with your attention and your dollars. Align your actions with your values. Standing up for what you believe in is a way to show what you believe so others can see what others think. It’s a way of learning ‘out loud’ I suppose. Or maybe ‘living out loud’. Still, I won’t back down. What think you?

(And now, back to your regularly scheduled posts. BTW, my intent is to keep Tuesdays for my thoughts; if I’m touting something I think you should know about, I’ll try to keep to Thursdays. And rare. ;)

Ideas we could do without

by Leave a Comment

Saw a post on LinkedIn from a colleague, ranting about how we are regularly putting old wine in new bottles. I do believe we’re getting deeper into design and strategy, but I also agree. Similarly, I’ve seen a regular feature on a newsletter talking about terms we can do without. So, I’m combining the two here. Not surprisingly, I’m channeling previous complaints (as a commenter made mention of), but this is the first time combining them into ideas we could do without.

Microlearning. As I’ve said before, the problem here is that there are regularly two things meant here: either spaced learning or performance support. Both are good things, but lumping them under one label constitutes a problem. For one, they have different design processes and goals, so using the same term for two different things risks confusion. I like the idea of emphasizing conciseness, but…we can call it minimalism, eh?

Workflow learning. This is problematic because it implies learning, yet, as I’ve repeated, you can’t learn ‘in the workflow’. My argument rests on the fact that learning is really action and reflection, and reflection breaks the workflow. I reckon this could be definitional, as some folks might argue that such reflection is part of the workflow, but like with microlearning, they’re also many times talking about performance support. So, it’s another term with wrong usage, or at least ambiguous provenance. Let’s talk performance support or learning from the workflow.

Mobile. This may seem odd, given that I’ve been talking and writing about mobile at least since my first book on the topic, more than a decade ago! (Notably, both books are now out of print. Indicative?) Yet, I still receive requests from developers to make my mobile apps (not what I do). Also, Google declared they were going ‘mobile first’ also over a decade ago. Really, mobile has kind of just merged into digital solutions, I would suggest. Sure, we get folks asking us to use the app, but that to me is frustrating. It shouldn’t matter whether I’m using the app or a website, I have the same goals, largely. Yes, there are some location-specific things, and we (still) aren’t taking advantage well of the contextual capabilities of mobile devices, but mobile is really moot. It’s about augmenting our thinking. And, separately, taking advantage of context.

Unlearning. I’m adding this after originally writing this, because it just emerged again, and literally two days after a really nice ‘takedown‘ by Tom McDowell, who’s developed a real capability for research translation. In short, our brains can’t unlearn. That is, we don’t forget things, so we need to really build a new, alternative response to a previously learned approach. Which means that solutions designed for ‘unlearning’ won’t achieve the necessary outcome. Thus, this isn’t just a nice shortcut, but instead creates impressions that can lead folks astray. Let’s dump the phrase completely. Please?

I’ll add a new one: AI. What?

AI. As I’ve mentioned, I’ve been a big fan of artificial intelligence (AI) for literally decades. So, why am I struggling? I admit I’m getting overwhelmed when people say “AI” and mean generative AI. Generative AI is, conceptually, a small subset of AI. Sure, it’s huge right now, but that’s largely hype driven by money. It’s not real in a meaningful sense. I wish people could and would be clear, like “I’m going to call it AI, but I’m talking about generative AI and large language models (LLMs) in particular.” Which kind of undermines the hype, but what’s wrong with that? (Except for the purveyors, of course.) Sure, we should be treating all our digital endeavors similarly in strategy, e.g. as Lori Niles Hoffman’s new book points out, but AI is just one of the tools we should be tapping into.

Do I think my rant will change anything? Of course not! There’s money to be made, after all. Also, no one pays much attention to my rants here anyway ;). Still, a chance to get this off my metaphorical chest. So those are my ‘ideas we could do without’. What are yours?

A celebration of the tablet

by Leave a Comment

OK, so I’ve been around for a few turns around the sun. And, I’ve seen a fair bit of change, in technology in particular. My first job out of college was programming these new-fangled ‘personal computers’. This came after holding down jobs programming mainframes (Algol/Burroughs 7600) and mini-computers (Pascal/PDP-11s). As such, I’ve watched as the computers have shrunk to pocket and wrist-size (and more, but that’s not in common use), while increasing in capability. But there’s one device that I forget to marvel at, and so here’s a celebration of the tablet.

Why tablets, why not smartphones? There’re overlaps; they’re both much more intimate than a desktop; they”re held and controlled by touch. Yet, there’s a difference. Ok, for me. For many around the world, the phone is their only internet connection. And we are seeing larger and larger phones. But while I carry my phone with me almost all the time*, a tablet is my preferred relaxation device. I use my laptop (as a desktop, it’s usually plugged into a monitor/keyboard/mouse unless I’m traveling) for work, and my phone goes with me everywhere. But, when I’m mostly looking to be mentally engaged, my tablet is the preferred option. So, why?

I haven’t pondered that except just before going to bed, but then I’m amazed as I contemplate it. Here’s a screen big enough to engage with, on a device that’s amazingly thin. On it, I have books I can read, and games I can play. I don’t watch movies on it usually, but I have when I’ve traveled. And of course there are the information resources: web, weather, & time, and ways to generate info: notes, a way to comment on PDFs. In short, it’s intimate but pretty much fully capable.

That’s not what my phone or desktop sport as abilities. My phone (I err on the side of a smaller one, to fit easily in my pocket) is intimate, but it’s not fully capable given its limited screen. On the other hand, m laptop is powerful, but not intimate. My tablet is both. Now, I’m not going down the road of a super powerful tablet that substitutes for a laptop, that’s a bridge too far. I like having a desktop to do full work. It’s just that for many things, e.g. pondering and querying, the tablet is better. I think it’s about time frames for task, and maybe the cognitive complexity.

Still, while I like all my devices, the laptop to me is akin to the computer as I know it. The phone is a useful devices for usage on a short time frame. It’s the device in-between that, for me, is the sweet spot. A place to escape, a place to do preliminary thinking, a place to reflect. I write this on my laptop, because writing long screeds is easier with a keyboard (I eschew the keyboard cases for my tablet, as I have a laptop), but the preliminary thinking came from my intermediate device. So here’s a celebration of the tablet, the device that wasn’t going to be. I’m glad it exists.

* Recently my phone had 1% charge, I’d not completely plugged it in. Thus, for my walk I went phone-less (m’lady had hers). It was mostly ok, just a few moments of reaching for it and realizing it wasn’t there!

If it’s Thursday…

by Leave a Comment

…it must be Belgium. Ok, so there’re two things here. First, on Thursday, Feb 26, in Brussels specifically, I’ll be there. Why? I’ll be spending a full day workshopping practical learning science. In conjunction, I should note with my LDA Co-Director, Matthew Richter. And, hopefully, you!  So, what, and why, and…? For the other, yes, I know the original movie said Tuesday. And, no, I haven’t seen it.

The premise of the workshop is Designing Learning That Actually Works. This is something we both not only care about (heck, basically the whole focus of this blog!), but believe is the essential ingredient to successful L&D initiatives. Matt’s knowledge complements mine. He’s deeper into face-to-face learning and the business side, while I’m more into asynchronous learning and the cognitive side. Though, of course, we’re continually learning from each other. We also are both focused on essential practice, not theoretical background (though the latter is essential).

We’re starting with the core of how our brains work, largely at a cognitive level (not neural; despite the claims it’s largely irrelevant). We’ll cover how we process information, with a focus on learning. Then we’ll look at the emergent elements that research has identified, with a clear focus on the implications for design. There’ll be a focus on what most learning is missing, as there’s little reason to rehash what’s not going wrong!

The advantage of the location is, we feel, that it’s central. I’ve always admired how easy it is to get around Europe (I love train travel!), and thus we’re expecting that the catchment is relatively broad. I haven’t been to Europe for a number of years (thanks, Covid), so this will be a happy reunion. Also, I’ve met Matt once face-to-face, and this will be an extended chance to interact. Still, I’ve been working with him for a number of years online now and continue to be impressed with his knowledge (and jealous of his superior social skills).

So, if it’s Thursday (Feb 26) and you’re nearby, or can arrange to be there, I think this is going to be valuable. Also fun, as Matt’s possibly even more irreverent than I am, and certainly less staid ;). We hope to see you there!

By the way, I’ll be there M-F, so if you’ve some ideas for the other days, let me know!

Looking into 2026

by Leave a Comment

First, of course, happy new year! Relatively arbitrary deadline, but signification matters, and marking a new year is also a new chance. So, what’s happening? Here are some of the things I’m thinking about, looking into 2026!

So, first, a brief look back to set the stage. This was a year without any sustained engagements for Quinnovation, so that meant being a bit more agile. Not a problem, I was on lots of podcasts about a variety of things, and of course engaged with clients. I did spend considerable time and effort, however, in my side gigs.

For one, the Learning Development Accelerator (LDA) had a variety of things going on: conferences, books, webinars, podcasts, and more. Plus, they’re great people to work with! I think it’s a worthwhile investment of my time, focusing on helping people get more exposed to evidence-informed design.There are signs that we’re moving more that way (though it is a small case of two steps forward, one step back; learning styles and other myths still haunt our industry; there’s a continuing need!). I’ve also done some ‘free’ consulting to our platinum members, and that seems to be valuable for them, and I find it really rewarding!

I also continued to spend time with the Elevator 9 (E9) folks. They’re now ready for prime time (check ’em out!), but there’s been lots of work along the way. That includes developing a real platform, and I’m continuing to learn heaps about what goes into a startup. And why I haven’t been the one to do it! It’s been great, however, to be in association with folks who really do want to care about learning science; all too rare in the learning technology space (sadly).

Of course, my association with both continues.

With LDA, we’re already planning this coming year. We’re deep into thinking about what to do with the spring conference, and potential series for blog posts, and more. We already have our first Meet the Author on the schedule, and more are in the works.  There are some changes afoot, so stay tuned!

With E9, I will be using them again for my next mini-scenario workshop (with LDA) as a followon. Did it last year, as a trial, and it worked. Always room to improve, of course. Still, if you’re running a live event, and not following it up to extend the learning, why? There are other solutions – e.g. coaching – but please be doing something! There are worse solutions than E9, including nothing.

Of course, I’ll be doing more. I’ve been working on a couple of short books, likely eBooks (too short for print). I don’t want to go live yet about them, as they’re still in process. Of course they’d be with LDA Press. Besides online, I may be running a workshop or two live, too. As to conferences, well, I never say ‘never’, but right now there’s nothing I’m particularly excited about. We’ll see. And, of course, I’m always keen to help organizations, so do reach out if there are any ways I can be of assistance.

As you might expect, ideas continue to percolate. I’m always exploring more about technology, design, engagement, and more, and of course about learning. As always, you’ll probably hear about them here first, as this is where I learn ‘out loud’. There’re breadcrumbs from the past pointing forward, so it’s time to be looking into 2026. What are you seeing? In the meantime, stay curious my friends.

Aphasia analogy for LLMs?

by Leave a Comment

Something triggered, for me, an analogy. I was thinking of aphasias, and thought one might be a good analogy for LLMs. Both to understand, and to use as guidance for using. So here are some initial thoughts on an aphasia analogy for LLMs.

First, while aphasia is complex, two types reliably correlate with damage to specific areas of the brain: Broca’s, and Wernicke’s. Interestingly, they’re partners, as both deal with knowledge and language. That is: what we know, and how we’re able to communicate. Both come from damage to a specific part of the brain, but have opposite effects.

Broca’s aphasia is reasonably clear. The evidence suggests that folks retain their knowledge, but struggle to communicate. That is, what comes out is broken and ungrammatical, but meaningful. People generally have no trouble thinking, just talking about it. There of course is a region of the brain for Broca.

The companion is Wernicke’s aphasia. Here, the language is eloquent, but essentially nonsensical. People may have thoughts, but what they say has no internal cohesion. There can even be made-up words! There’s also a region of the brain known for Wernicke.

You can probably figure out where this is going: LLMs learn from large corpora of text to produce accurate language. Not accurate answers, but accurate language, an important distinction! They’re not damaged, so if most of the training set is accurate, what they say will be accurate. However, if not, they may say other things. And, of course, they can just say things that sound right that aren’t, such as making up books, court cases, and more. They’re essentially Wernicke systems!

What does this mean? It means you probably can have a good idea generation sessions with an LLM, or give it a language task like summarizing or generating language. What it also means is that there’s no way that you should trust what comes out to be accurate, and you need an expert in the loop. Given that they’ve demonstrably been corruptible, besides not always being correct, however, they shouldn’t be trusted to act on your behalf. I worry a wee bit about them being good enough that most of the time what comes out is ok. This could, of course. lull you into a sense of complacency! Hopefully folks will always feel the professional obligation to ensure that what comes out is correct.

So, understand what LLMs do, if not how they do it. Then, act accordingly. (Feel free to also investigate issues like IP, environment, biz model, and more.) Caveat emptor!

Age or experience?

by Leave a Comment

One of the things that has been a recurring theme across things I’ve been looking at lately is experience. Too often we confound age with experience. And, of course, sometimes it’s that we should be talking about it. So, a brief rant on age or experience.

First, I’ll bring up the ‘generations’ myth. It’s appealing, as our brains like buckets for things. We’re kinda wired that way. The only problem is that generations as a concept has been looked at and debunked. Heck, in Ancient Greek days they were complaining that ‘kids just have no respect”! And if you think about it, thinking that someone in Los Angeles CA of a certain age has more in common with someone in Nepal of the same age versus another Angelenõ of a different age is kinda ridiculous.

And, those ‘defining’ events? They affect every conscious person! And it’s so context dependent. A local event may not mean much to you, unless it affects you somehow, and then you share more with everyone else so affected. There’s actually a simpler explanation. Say, for instance, that “young folks want classes while old folks don’t”. That’s explainable by stage of life: when you’re young you need credentials, but later on you can point to your experience.

People share values, and gain motivation by the same underlying factors (differently across culture and personality), and more. Just look at the research on self-determination theory! Attributing to age rather than explaining by experience is a mistake. So, for instance, my kids, who arguably fit the label ‘digital natives’, still come to me (decreasingly, I’ll admit) for tech problems.

Then, there are many things that change as you develop in a domain. For instance, in our Learning Science Conference, my colleague Matt Richter was talking about feedback, and very clearly pointed out how what useful feedback is changes as you gain experience. This holds true for examples, too, the type of useful example changes. Also for practice: with more experience, you need more challenge.

Which, as we further see, is how we go wrong. We do the ‘one size fits all’, not recognizing that things need to change. To be fair, we also do the wrong practice (knowledge test rather than application to problems), give the wrong feedback to begin with (right/wrong), the list goes on. But even when we’re trying to do it right, we forget things like adapting for initial and developing experience. Yet, it’s a factor for instance in how much practice you need, how much spacing, etc.

This problem does go more broadly. We hear it in hiring (age discrimination). Of course, that’s only one problem. For example, gender, race, physical and neurological differences, and more are also present. Sadly. Okay, soapbox: DEI, done right, leads to better outcomes! Actually, that’s got an evidence-base, so probably more than soapbox. Still. So, consider experience as one of the factors distinguishing individuals. Folks can’t control their age, but they can determine their experience. So use it!

Key notes

by Leave a Comment

I’ve seen a lot of keynotes over the years. I’ve even given them! It’s time to reconcile my thoughts. So here are some key notes on keynotes.

One of the things that I’ve seen is flawless performances. Now, one of the things you’re told is that the focus is on you, and slides only should be used as an augment, not on all the time. I confess I’m not good about that (I am not very comfortable in the spotlight; imposter syndrome I suppose). Another is that you should have pauses, and jokes, and such. I do a pretty good dramatic reading (I won the dirty limerick reading contest at work once!), and occasionally even manage to raise a smile or two. The best, however, have their patter completely down. I can’t do that, because my thoughts are continually evolving, but I admire it when it happens. And I’m pretty good at tailoring a talk to the audience (having learned a few times the hard way!).

Inspiration is good, too. Letting people know there’s a way to surpass this barrier works! I try to do that too, though I confess I talk about learning design, not achieving things like climbing Mt Everest (really heard a keynote about that, and it was cool!). And I probably am a bit too conceptual, though I am learning to do better about grounding my principles in practice. However, I do recoil from too much ‘enthusiasm’! Somehow it comes across as artificial. But then, I can be a bit of a curmudgeon (apparently)…

However, what really matters to me is accuracy. I don’t mind if folks are a bit enthusiastic or polished, but I really get wound around the axle a bit when folks state stuff that’s just wrong. For instance, I heard a well-regarded personage opine about games, something I know a wee bit about (my first job, back when dinosaurs strode the earth, was on games, and it’s been a recurrent them in practice, research, and writing for literally decades). And that individual said something just dead wrong. As you may surmise, it really ground my gears. Similarly with learning science, or mobile.  In general, when people have beautifully symmetric ‘n part models’ without grounding, I want to know if those are convenient, or a necessary and sufficient list. (Too often the former.)

I also like when people tout doing things I believe in, but when it’s an area I know about, you better agree with the science, and present it accurately. If you don’t, well, I won’t be quiet about it. (I guess it’s a flaw in my character!) Still, when you say these are the five things to X, and they’re a) not completely separable, b) incomplete, c) wrong, etc, I’m going to be turned off.

Look, I love a good keynote. Many times, they get people who aren’t from the field where the keynote’s presented, and they make connections from adjacent fields. That’s acceptable, even desirable! I like a well-presented talk as well as the next person. I like ideas, and even inspiration. But I will complain about bad information. Always. Those are my key notes on keynotes, what are yours? (And I’m available, if you want L&D advice ahead of the curve but grounded in evidence. Particularly contrary takes… ;)

Analyzing analysis

by 1 Comment

Another reflection, triggered by my visit to DevLearn. One of the things that matters, and we don’t discuss enough, is analysis. That is, starting up front to determine what we need! There are nuances here, and I’m not a total expert (paging Dawn Snyder), but certain things are obvious, So let’s take some time analyzing analysis.

Analysis is the first part of the process. Yes, there’re the organizing and managing bits, but the process starts with analysis, whether ADDIE, SAM, LLAMA, or any other acronym. You need to determine what’s going on, what’s the need, and what’s the appropriate remedy.

One of the first things to note is that not everything L&D does fits. As is widely noted (e.g. here), there are lots of reasons courses aren’t the only answer. The real trigger should be a need. That is, there’s a new skillset required to do this thing we’ve identified as wrong or necessary. Or, there’s something we’re doing, but badly. At core, there are two situations: the one where we need to be, and the one where we are. The gap between is what we want to remedy.

Then, it’s matter of determining why we’re not where we want to be. The reason is, there are different interventions for different problems, as Guy Wallace talks about in his tome. It might be a lack of resources, or people get rewards for doing X, even though it’s Y they’re to be doing. These, by the way, aren’t things we deal with! That’s why you do this, so you don’t build a solution where said solution actually isn’t.

When it is a situation where knowledge in the world, or in the head, will help, then we can jump into action. Of course, we need a clear definition of what it is people need to be able to do, under what conditions, etc. BTW, what we need are performance objectives, not ‘learning’ objectives. That is, it’s about doing. Which is why, if the circumstances support, we should be providing job aids, not courses! You’ll usually find that job aids are cheaper to do than courses. If it’s not being performed very frequently, or too frequently, memory will play a role, and external memory is valuable in many such circumstances.

When you’ve determined that a course is needed, you can develop that. HOWEVER, you need certain things from the analysis phase here too. In short, you need to understand the actual performance. That includes what the performance should be, and how you can tell. Essentially, you need to know the decisions people must make to deliver the required outcomes. Which involves knowing the models that describe how the world works in this particular area, what ways people go wrong and why, and why people should care. This is where you need your subject matter experts (SMEs).  Then you can build your practices that align, and the models and examples, and then the hook and closing, and…

Whatever it is, ideally there’s a metric, that says this is what’s needed. You design to that metric, and then test until you achieve it. If you’re not achieving it faster than you’re losing resources, you can consciously evaluate. Is the lower level ok? Can we get more resources? Should we abandon ship? But doing so consciously is better than just going ’til you run out of time and/or money.

Analysis is a necessary first step. What is not is responding with acquiescence to a ‘we need a course on X’ request. Do you trust them to know that a course on X solves their problem? (Not the way to bet.) You can, and should, say, “yes, and…let’s dig in and make sure we’re solving the right problem”. Analysis is, properly, the way to start looking at problems. You understand what the gap is, then the root cause, and then align an intervention, or interventions, to address it. By analyzing analysis, we can figure out what we have to do, and why.

And, yes, I just gave a talk on designing in the real world, and you may have to do inference on resources to determine all the above, but at least you know what you need to come away with.