As has become a pattern, someone recently asked me how to evaluate soft skills. And without being an expert on soft skill or evaluation, I tried to answer on principle. So I thought about the types of observable data you should expect to find. And that yielded an initial answer. Then I watched an interesting video of a lecture by a scholar and consultant, and it elaborated the challenges. So, there‘s a longer answer too. So here‘s an extended riff on evaluating soft skills.

I started with wondering what performance outcomes would you expect for soft skills. Coupled, as well, with how could you find evidence of these observable differences. As a short answer, I suggested that there should be 3(+) outcomes from effective soft skills training.  

0) the learner should be able to perform in soft skills scenarios (c.f. Will Thalheimer’s LTEM). This is the most obvious. Put them in the situation and ask them to perform. This is the bit that gets re-addressed further down.  

1) the learner should be aware of an improvement in their ability to perform. However, asking immediately can lead to a misapprehension of ability. So, as Will Thalheimer advises in his Performance-Focused Smile Sheets, ask them 3 months later. Also, ask about behavior, not knowledge.   E.g. “Are you using the <> model in your work, and do you notice an improvement in your ability”

2) The ‘customers’ of the learner should notice the improvement. Depending on whether that’s internal or external, it might show up (at least in aggregate) in either 360 eval scores, or some observable metric like customer sat scores. It may be harder to collect this data, but of course it‘s also more valuable.  

3) Finally, their supervisors/managers should notice the improvement, whether observationally or empirically.They should be not only prepared to support the change over time, but asked to look for evidence (including as a basis to fine tune performance).  

All together, triangulating on this should be a way to establish the validity.  

Now, extending this, Guy Wallace tweeted a link to a lecture by Neil Rackham. In it, Neil makes the case that universities need to change to teaching core skills, in particular the 4 C‘s: critical thinking, creativity, communication, and collaboration. He also points out how hard it is to evaluate these without a labor-intensive effort of an individual observing performance. This is a point that others have made, that these skills have hard to observe criteria.  

There‘s some argument about so-called 21C skills, and yet I can agree that these four things would be good. The question is how to assess them reliably. Rackham argues that perhaps AI can help here. Perhaps, but at this point I‘d argue for two things. First, help students self-evaluate (which has the benefits of them understanding what‘s involved). Second, instrumenting environments (say, for instance, with xAPI) in which these activities are performed. There will be data records that can be matched to behaviors, initially for human evaluation, but perhaps ultimately for machine evaluation.  

Of course, this requires assigning meaningful activities that necessarily involve creativity, critical thinking, communication, and/or collaboration. This means project based work, and I‘ve long argued that you can‘t learn such skills without a domain. Actually, to create transferable versions, you‘d need to develop the skills across domains.  

When I teach, I prefer to give group work projects that do require these skills. It was, indeed, hard to mark these extra skills, but I found that scaffolding it (e.g. a ‘how to collaborate‘ document) facilitated good outcomes. Being explicit about the best thinking practices isn‘t only a good idea, it‘s a demonstrably useful approach in general.  

So I think developing skills is important. That means we need a means to be evaluating soft skills. We know it when we see it, but it‘s hard to necessarily find the opportunity, but if we can assign it, we can evaluate and develop these skills more readily. That, I think, is a desirable goal. What think you?

A colleague asked “is anyone talking about how deep learning requires time, attention, and focus” He was concerned with “the trend that tells us everything must be short.”   He asked if I‘d written anything, and I realize I really haven‘t. Well, I did make a call  for “slow learning” once upon a time, but it‘s probably worth doing it again.   So here‘s a riff on deep learning and expertise.

First, what do we mean by deep learning? Here, I‘m suggesting that the goal of deep learning is expertise. We‘ve automated enough of the component elements that we can use our conscious processes to make expert judgments in addressing performance requirements. This could be following a process, making strategic decisions such as diagnoses and prescriptions, and more. It can also require developing pre-conscious responses, such as we train airline pilots to respond to emergencies.  

Now, these responses can vary in their degree of transfer. Making decisions about how to remedy a piece of machinery that‘s misbehaving is different than deciding how to prioritize the new product improvements. The former is more specific, the latter is more generic. Yet, there are certain things that are relevant to both.  

Another issue is how often it needs to be performed. You can develop expertise much quicker with lots of opportunities to apply the knowledge. It‘s more challenging to achieve when there aren‘t as many times it‘s relevant in the course of your workflow. The aforementioned pilots are training for situations they never hope to see!

Before we get there, however, there‘s one other issue to address: how much has to go in the head, and how much can be in the world?   In general, getting information in the head is hard (if we‘re doing it right), and we should try to avoid it when possible. I argue  for backwards design, starting with what the performance looks like if we‘ve focused on IA (intelligence augmentation ), that is, looking for the ideal combination of smarts between technology (loosely defined) and our heads. As Joe Harless famously said “iInside every fat course there‘s a thin job aid crying to get out.”  

Once we‘ve determined that we need human expertise, we also need to acknowledge that it takes time! I put it this way: the strengthening of connections (what learning is at the neural level) can only be done so much in any one day before the strengthening function fatigues; you literally need sleep before you can learn more. And only so much strengthening can happen in that one day. So to develop strong connections, e.g. strong enough that it will be triggered appropriately, is going to have to be spaced out over time.  

This does depend on the pre-existing knowledge of the learner, but it was Anders Ericsson who posited the approximately 10K hours of practice to achieve expertise. That‘s both not quite accurate and not quite what he said, but as a rule of thumb it may be helpful. The important thing is that not just any practice will work. It takes what he called ‘deliberate practice‘, that is the right next thing for this learner. Continued, over time, as the learners‘ ability increases new practice focuses are necessary.

All that can‘t come from a course (no one is going to sit through 10000 hours!). Instead, if we follow the intent of the 70:20:10 framework, it‘s going to take some initial courses, then coaching, with stretch assignments and feedback, and joining a relevant community of practice, and….

We also can‘t assume that our learners will develop this as efficiently as possible. Unless we‘ve trained them to be good self-learners, it will take guided learning across their experience. Even if it‘s only at a particular point; most people who are pursuing a sport, hobby, what have you, eventually will take a course to get past their own limitations and accelerate development.

The short answer is that deep expertise doesn‘t, can‘t, come from a short learning experience. It comes from an extended learning experience, with spaced, deliberate, and varied practice with feedback. If you want expertise, know what it takes and do it. That‘s true whether you‘re doing it for yourself or you‘re in charge of it for others. Deep learning and expertise comes with hard work. (Also, let‘s make that ‘hard fun‘ ;).  

Asked about why I used the word pedagogy instead of andragogy, I think it’s worth elaborating (since I already had in my reply ;) and sharing. In short, I think it‘s a false dichotomy. So here‘s my analysis of andragogy vs pedagogy.

Looking at Knowles‘ andragogy, I think it‘s misconstrued. What he talks about for adults is really true for all learners, taking into account their relative cognitive capability and amount of experience. So I fear that using andragogy will perpetuate the myth that pedagogy is a different learning approach (and keep kids in classrooms listening to lectures and answering rote questions). Empirically, direct instruction works (tho‘ it‘s interpretation is different than the name might imply, I once pointed out how it and constructivism properly construed both really say the same thing ;).  

There was an article  that posited five differences, and I see a major confound; the article‘s talking about andragogy as self-directed learning, and pedagogy as formal instruction. That‘s apples and oranges. It really is more about whether you‘re a novice or a practitioner level and the role of instruction. Age is an arbitrary element here, not a defining factor. Addressing each point:

1. Adults are self-directing learners. No, in things they know they need, they can be, but also they may have their bosses or coaches pointing them to courses. Plus, for areas where the adults are novices, they still need guided instruction. Also, owing to our bad K12 and higher ed, we’re not really enabling learners to be effective and efficient self-directed learners. Further, kids are self-directed about things they‘re interested in. But we make little effort to ground what we do (particularly K6) in any reason why this is on the syllabus.  

2. The role of learner experience. Yes, this matters, but it‘s a continuum. Also, you always want to base instruction on learner experience, because elaboration requires connecting to and building on existing knowledge. Yes, we do tend to do give kids abstract problems (particularly in math), which is contrary to good learning science. “Only two things wrong in education these days, the curriculum and the pedagogy, other than that we‘re fine.” Ahem. We teach the wrong things, badly.  

3. Adults generate interest in useful information. So does everyone, but that‘s not a matter of developmental level. Kids also prefer stuff that‘s relevant. We‘ve developed a curriculum for kids that is out of date, and we don‘t motivate it. Everyone has a curriculum, and there are degrees of self-direction, but it‘s not a binary division.

4. Adult readiness to learn is triggered by relevance (yeah, kind of redundant).Kids also learn better when there‘s a reason. Hence problem-based, service-based, and other such philosophy‘s of learning. Even direct instruction posits meaningful problems. Again, the article‘s comparing an ideal human learning model compared to a broken school model.  

5. What motivates learners are real life outcomes. Really, we‘ve covered this, everyone learns better when there‘s motivation. Children learn for grades because no one‘s made it meaningful for them to care!   Kids will pursue their learning when it makes sense to them. John Taylor Gatto made the case that kids could learn the entire K6 curriculum in 100 hours if they cared! Kids do learn outside of what‘s forced on them from schooling, be it Pokemon, polka, or porcupines.  

Thus, in the comparison between andragogy vs pedagogy, I come down on the side of pedagogy. It‘s the earlier term, and while ped does mean ‘kid‘, I still think it‘s really about learning design. Learning design should be aligned to our brains, not differentiated between child and adult. Yes, there are developmental differences, but they‘re a continuum and it‘s more a matter of capacity, it‘s not a binary distinction. That‘s my take, what‘s yours?

I stumbled across the Elements of UX diagram again, and happened to wonder if it would map to LXD. Here’s my stab:

And the text, as usual.

---

In a justifiably well-known image (PDF), Jesse James Garrett (JJG) detailed the elements of (web) user experience. I‘ve been involved in the parallel development of UX and ID (and cross-fertilized them), so I wondered what the LXD version would be. So, of course, I took a stab at levels of LXD design.

To start with, JJG‘s diagram works from the bottom up. The five levels, in order, are:

1. The original objectives and user needs.
2. That leads to content requirements and/or functional specifications.  
3. The next level is an information architecture or interface design that is structured to meet those needs.  
4. Those semantic structures are then rendered as an information design with navigation or interface design.
5. The top level is the visual design, what the user actually sees or experiences.

This systematic breakdown has been well recognized as a useful development framework. The development from need to semantics to implementation syntax suggests a logical development flow. As an aside, no one‘s claiming we should develop in a linear manner, and there tends to be more up and down action in actual practice. Drilling down and then working from the bottom up as well is a well-known cycle of design!  

The learning equivalent, then, should similarly have a structured flow. We want to go from our needs, through various levels of representation, until we reach the learner experience.  

Given that we should be driven not by the goals for the interface but learner needs, I‘ll suggest we start with the performance objectives.   Then, in parallel with user needs, I‘ll stipulate that the other top-level definition comes from the user characteristics. These match the initial level stipulated.  

At the next level, I‘ll suggest that the performance objectives drive assessment specifications, and the other decision at this level is for the pedagogical approach. We need to know what learners need to able to do, and how we‘ll get them there.

As an intermediate representation equivalent to UX‘s information architecture or interface design, I suggest from the assessment we determine the necessary practice activities required, and these are coupled with the necessary content requirements: models and examples, as well as the introduction and closing. Here we‘re still at what‘s required, not how it manifests.  

The next level is where we start getting concrete. We need to pick an overall theme or look and feel, and the flow of the experience. We‘ll also, of course, need to make a consistent interface to support navigation and taking action. We know what we need to have, but we haven‘t actually rendered it yet.  

Finally, we must render the necessary media. This will be the videos, audios, text, diagrams, images, and more that comprise the experience. This includes the actions to be taken and the associated consequences of each choice.  

That‘s the equivalent structure I‘m suggesting are the different levels of LXD design. Of course, this is a thought exercise, and so I may well have made some interpretations you could disagree with. For instance, I may have slavishly followed JJG’s levels too closely. Let me know! Also, it‘s not clear whether this is a useful representation, so far it‘s sort of a ‘because it‘s there‘ effort ;). You can let me know your thoughts on that, too!  

Here’s a story about where performance support would’ve made a task much easier.

And, as always, the text.

---

The other day, I had a classic need for performance support. Of course, it didn‘t exist. So here‘s a cognitive story about when and where a job aid would help.

Our Bosch dishwasher stopped near the beginning of the cycle, and displayed an icon of a water tap. The goal was to get the dishwasher running again. What with the layer of undrained water, we figured there was some sort of problem with the drain, clogged or the pump broken. M‘lady had cleaned the drain, but the icon persisted. What now? Of course we could call a service person, but trying to be handy and frugal (and safe), we wanted to find out if it was something I could deal with. So, off to the manual.

Well, in this case, since I didn‘t know where the manual was, I went online. I accessed the site and downloaded the manual. Only to find no guide to what the icons mean. What?!? This violates what we know about our brains, in this case that our memory is limited. The support section of the site did list the error codes, but numerically, not by icon.  So, I had an indication I couldn’t map to a problem, let alone a  solution.  

This is a real flaw! If you‘re gonna use icons, provide a guide!  Don’t assume they’re interpretable. (This had happened once before with this same appliance, with an impenetrable icon and no clue.) As a result, I had to call the service line. That wait took awhile (with more people staying home, they‘re using their dishwashers more, and the appliances are therefore breaking down more). Once, the call dropped. The second time I had to stop because I had an upcoming call. The third time, however, I got through.

And a perfectly nice person listened, asked some questions, and then instructed me through a process. After hitting cancel (which automatically tries to drain everything and reset to zero) by simultaneously pressing two buttons linked by a line on the control panel, I heard noises in the sink like it was draining. After a minute, I was told to go ahead and open it up (yep, drained), turn it off and on, and then try running the cleaning cycle again. And, voila, it worked! (Yay!)

So, what‘s wrong with this picture? First of all, there should be a clear explanation of what the icon means, as indicated above. Second, it should be clearly tied to a process to address the problem, including intermediate steps.This is so common, I am quite boggled that the great engineers that made our (very good) dishwasher aren‘t complemented with a great technical communications team who write up a useful manual to support. It. Is. Just. Silly!

Note: this isn‘t a learning experience. It‘s just fine that I don‘t recall what the last time‘s icon was or what it meant, and maybe what this icon meant and what I should do. It should be infrequent enough that it‘d be unreasonable for me to have to recall. Instead, I should be able to look it up. Put information in the world!  In the long term, this should save them buckets of money because most people could self help. Clearly, they‘ve gone to numeric codes, but they could‘ve just added in the associated icons, or given a mapping from icon to numeric code. Something to help folks who have the pics.  

This is just bad design, and it‘s so obvious how to ameliorate it. People will self-help many times, but only if they can!   Just as you shouldn‘t be creating a training course when a job aid will do, you can save a help call when a job aid can address most of the problems. Use performance support when it makes sense, and doing so comes from understanding how we actually think, work, and learn. When you do, you can design solutions that meet real needs. And that‘s what we want to do, no?

On Twitter today was a question from an organization that, frankly, puzzled me. Further, I think it’s important to understand  why this was a bad question. So here let me unpack several illustrative problems.

First, the question asks “What kind of learning do you prefer?” My initial response is: why would you ask that? What learners prefer has little to do with what outcomes you need to achieve.  We should design for the learning outcomes.

Then, there’s the list of elements:

• Video-based learning
• Article-based learning
• How to guides
• Interactive quizzes

There are several problems with this list. First, why this subset? This isn’t a full suite of alternatives. What about simulations, scenarios, or games? AR or VR? Podcasts? Why this selection?

Then, the options lack full definitions. What do they mean by ‘video-based learning’?  Is it just a video, with no assessment? Is it really ‘learning’ then? Of course, if the ‘-based’ means assessment as well, how is that separate from ‘interactive quizzes’? Similarly for articles. What is included?

Yet guides and quizzes aren’t ‘-based’. Are we assuming they’re full learning solutions? That’s questionable. A how-to guide, aka performance support, might yield an outcome, but it doesn’t guarantee learning. There are lots of factors that would influence that. And interactive quizzes, without models and examples, would be a slow way to develop expertise.

Another problem is in the separation of the elements. So, for instance, a ‘how to’ guide could be a video or an article! There’s the Youtube video I used to fix my dryer, or the step by step instructions I used to figure out how to run cables on a monitor. Likewise, interactive quizzes could include video or point to an article. These aren’t mutually exclusive categories.

The point is that this is a bad question. It’s already been taken down (I wasn’t the only one to question it!). Still, there’re lessons to be learned. (Maybe the most important is to ensure your social media marketing person has enough knowledge of learning not to do such silly things, but I can’t assume that’s the locus of the problem. It’s just a hypothesis I’ve seen play out elsewhere. ;) While there are times it makes sense to ask provocative questions, there’s also a reason to have conceptual clarity.  At least, that’s my take, I welcome yours!

 

 

Sparked by a conversation, I generate some animation thoughts.

And, as always, a transcript.

---

In a conversation the other day, my colleague mentioned how she was making a practice of creating animations. I found this interesting, because while I think animations are important, I don‘t do them all that much (or so I thought). Particularly intriguing was the notion of what principles might guide animations, including when to use them. I was prompted to reflect, and so here are some animation thoughts.

First, let‘s be clear what I mean. I‘ve argued that we don‘t use graphic novel/comic formats enough, and that likewise applies to cartoons. Which are also known as animations. Yet, that‘s not really what I‘m talking about. I think we could use them more, but that‘s another reflection.

Instead, here I‘m talking about animated diagrams. And I think there are times when these are not just engaging, but cognitively important. Diagrams map conceptual relationships to spatial ones, and can add additional coding with color and shape. Animations add the dimension of time, so these relationships can change. In my categorization, these are dynamic diagrams, useful when the conceptual relationships change in important ways depending on other factors.

Interestingly, in the conversation, it came up what one form of her animations were diagram builds.  I use diagrams a lot, not only to communicate, but as a tool for my own understanding! And, I‘d done some builds, but after Will Thalheimer‘s Presentation Science course I realized I needed to do that more systematically (and now do so).  Building diagrams is helpful. Cognitively, a diagram can be overwhelming if there are too many elements. By starting at one point, and gradually adding in other elements, you can prevent cognitive overload. And in a presentation, in particular, you want to highlight important points.  

However, I also think that there are things worth indicating how they work dynamically. Like how a content system would work, e.g. context and rules combining to pull content out by description. Or how coordinates change based upon trigonometric values. I haven‘t done much of this, for the simple reason that I don‘t have a good animation tool. And, yes, I‘m aware that you do motion in PowerPoint and/or Keynote, but I haven‘t gotten into it. Time for a skill upgrade!

There are problems with animations, and guidelines. John Sweller‘s cognitive load plays out with Dick Mayer‘s work on multimedia research (as captured in his book with Ruth Clark: eLearning and the Science of Instruction), as indicated above. Thus, you shouldn‘t try to have people read text while watching visual dynamics (use audio). Also, you should help people focus attention by removing extraneous details and/or highlighting the appropriate focus.  

The general principles of media apply as well. Accessibility suggests some alternate representations. Timing suggests having a pause ability for any animation longer than a certain time, and of course the ability to replay. Similarly, the animation design should use appropriate white space, highlighting, and other aspects that make it visually clear and appealing.  

Overall, I‘d suggest that there are times when animations are the best option for conveying dynamic conceptual information. To use them, however, you have to take into account our cognitive limitations. So, these are some of my animation thoughts. I welcome yours.