Archives for April 2021

As has become a pattern, someone recently asked me how to evaluate soft skills. And without being an expert on soft skill or evaluation, I tried to answer on principle. So I thought about the types of observable data you should expect to find. And that yielded an initial answer. Then I watched an interesting video of a lecture by a scholar and consultant, and it elaborated the challenges. So, there‘s a longer answer too. So here‘s an extended riff on evaluating soft skills.

I started with wondering what performance outcomes would you expect for soft skills. Coupled, as well, with how could you find evidence of these observable differences. As a short answer, I suggested that there should be 3(+) outcomes from effective soft skills training.  

0) the learner should be able to perform in soft skills scenarios (c.f. Will Thalheimer’s LTEM). This is the most obvious. Put them in the situation and ask them to perform. This is the bit that gets re-addressed further down.  

1) the learner should be aware of an improvement in their ability to perform. However, asking immediately can lead to a misapprehension of ability. So, as Will Thalheimer advises in his Performance-Focused Smile Sheets, ask them 3 months later. Also, ask about behavior, not knowledge.   E.g. “Are you using the <> model in your work, and do you notice an improvement in your ability”

2) The ‘customers’ of the learner should notice the improvement. Depending on whether that’s internal or external, it might show up (at least in aggregate) in either 360 eval scores, or some observable metric like customer sat scores. It may be harder to collect this data, but of course it‘s also more valuable.  

3) Finally, their supervisors/managers should notice the improvement, whether observationally or empirically.They should be not only prepared to support the change over time, but asked to look for evidence (including as a basis to fine tune performance).  

All together, triangulating on this should be a way to establish the validity.  

Now, extending this, Guy Wallace tweeted a link to a lecture by Neil Rackham. In it, Neil makes the case that universities need to change to teaching core skills, in particular the 4 C‘s: critical thinking, creativity, communication, and collaboration. He also points out how hard it is to evaluate these without a labor-intensive effort of an individual observing performance. This is a point that others have made, that these skills have hard to observe criteria.  

There‘s some argument about so-called 21C skills, and yet I can agree that these four things would be good. The question is how to assess them reliably. Rackham argues that perhaps AI can help here. Perhaps, but at this point I‘d argue for two things. First, help students self-evaluate (which has the benefits of them understanding what‘s involved). Second, instrumenting environments (say, for instance, with xAPI) in which these activities are performed. There will be data records that can be matched to behaviors, initially for human evaluation, but perhaps ultimately for machine evaluation.  

Of course, this requires assigning meaningful activities that necessarily involve creativity, critical thinking, communication, and/or collaboration. This means project based work, and I‘ve long argued that you can‘t learn such skills without a domain. Actually, to create transferable versions, you‘d need to develop the skills across domains.  

When I teach, I prefer to give group work projects that do require these skills. It was, indeed, hard to mark these extra skills, but I found that scaffolding it (e.g. a ‘how to collaborate‘ document) facilitated good outcomes. Being explicit about the best thinking practices isn‘t only a good idea, it‘s a demonstrably useful approach in general.  

So I think developing skills is important. That means we need a means to be evaluating soft skills. We know it when we see it, but it‘s hard to necessarily find the opportunity, but if we can assign it, we can evaluate and develop these skills more readily. That, I think, is a desirable goal. What think you?

A colleague asked “is anyone talking about how deep learning requires time, attention, and focus” He was concerned with “the trend that tells us everything must be short.”   He asked if I‘d written anything, and I realize I really haven‘t. Well, I did make a call  for “slow learning” once upon a time, but it‘s probably worth doing it again.   So here‘s a riff on deep learning and expertise.

First, what do we mean by deep learning? Here, I‘m suggesting that the goal of deep learning is expertise. We‘ve automated enough of the component elements that we can use our conscious processes to make expert judgments in addressing performance requirements. This could be following a process, making strategic decisions such as diagnoses and prescriptions, and more. It can also require developing pre-conscious responses, such as we train airline pilots to respond to emergencies.  

Now, these responses can vary in their degree of transfer. Making decisions about how to remedy a piece of machinery that‘s misbehaving is different than deciding how to prioritize the new product improvements. The former is more specific, the latter is more generic. Yet, there are certain things that are relevant to both.  

Another issue is how often it needs to be performed. You can develop expertise much quicker with lots of opportunities to apply the knowledge. It‘s more challenging to achieve when there aren‘t as many times it‘s relevant in the course of your workflow. The aforementioned pilots are training for situations they never hope to see!

Before we get there, however, there‘s one other issue to address: how much has to go in the head, and how much can be in the world?   In general, getting information in the head is hard (if we‘re doing it right), and we should try to avoid it when possible. I argue  for backwards design, starting with what the performance looks like if we‘ve focused on IA (intelligence augmentation ), that is, looking for the ideal combination of smarts between technology (loosely defined) and our heads. As Joe Harless famously said “iInside every fat course there‘s a thin job aid crying to get out.”  

Once we‘ve determined that we need human expertise, we also need to acknowledge that it takes time! I put it this way: the strengthening of connections (what learning is at the neural level) can only be done so much in any one day before the strengthening function fatigues; you literally need sleep before you can learn more. And only so much strengthening can happen in that one day. So to develop strong connections, e.g. strong enough that it will be triggered appropriately, is going to have to be spaced out over time.  

This does depend on the pre-existing knowledge of the learner, but it was Anders Ericsson who posited the approximately 10K hours of practice to achieve expertise. That‘s both not quite accurate and not quite what he said, but as a rule of thumb it may be helpful. The important thing is that not just any practice will work. It takes what he called ‘deliberate practice‘, that is the right next thing for this learner. Continued, over time, as the learners‘ ability increases new practice focuses are necessary.

All that can‘t come from a course (no one is going to sit through 10000 hours!). Instead, if we follow the intent of the 70:20:10 framework, it‘s going to take some initial courses, then coaching, with stretch assignments and feedback, and joining a relevant community of practice, and….

We also can‘t assume that our learners will develop this as efficiently as possible. Unless we‘ve trained them to be good self-learners, it will take guided learning across their experience. Even if it‘s only at a particular point; most people who are pursuing a sport, hobby, what have you, eventually will take a course to get past their own limitations and accelerate development.

The short answer is that deep expertise doesn‘t, can‘t, come from a short learning experience. It comes from an extended learning experience, with spaced, deliberate, and varied practice with feedback. If you want expertise, know what it takes and do it. That‘s true whether you‘re doing it for yourself or you‘re in charge of it for others. Deep learning and expertise comes with hard work. (Also, let‘s make that ‘hard fun‘ ;).  

Asked about why I used the word pedagogy instead of andragogy, I think it’s worth elaborating (since I already had in my reply ;) and sharing. In short, I think it‘s a false dichotomy. So here‘s my analysis of andragogy vs pedagogy.

Looking at Knowles‘ andragogy, I think it‘s misconstrued. What he talks about for adults is really true for all learners, taking into account their relative cognitive capability and amount of experience. So I fear that using andragogy will perpetuate the myth that pedagogy is a different learning approach (and keep kids in classrooms listening to lectures and answering rote questions). Empirically, direct instruction works (tho‘ it‘s interpretation is different than the name might imply, I once pointed out how it and constructivism properly construed both really say the same thing ;).  

There was an article  that posited five differences, and I see a major confound; the article‘s talking about andragogy as self-directed learning, and pedagogy as formal instruction. That‘s apples and oranges. It really is more about whether you‘re a novice or a practitioner level and the role of instruction. Age is an arbitrary element here, not a defining factor. Addressing each point:

1. Adults are self-directing learners. No, in things they know they need, they can be, but also they may have their bosses or coaches pointing them to courses. Plus, for areas where the adults are novices, they still need guided instruction. Also, owing to our bad K12 and higher ed, we’re not really enabling learners to be effective and efficient self-directed learners. Further, kids are self-directed about things they‘re interested in. But we make little effort to ground what we do (particularly K6) in any reason why this is on the syllabus.  

2. The role of learner experience. Yes, this matters, but it‘s a continuum. Also, you always want to base instruction on learner experience, because elaboration requires connecting to and building on existing knowledge. Yes, we do tend to do give kids abstract problems (particularly in math), which is contrary to good learning science. “Only two things wrong in education these days, the curriculum and the pedagogy, other than that we‘re fine.” Ahem. We teach the wrong things, badly.  

3. Adults generate interest in useful information. So does everyone, but that‘s not a matter of developmental level. Kids also prefer stuff that‘s relevant. We‘ve developed a curriculum for kids that is out of date, and we don‘t motivate it. Everyone has a curriculum, and there are degrees of self-direction, but it‘s not a binary division.

4. Adult readiness to learn is triggered by relevance (yeah, kind of redundant).Kids also learn better when there‘s a reason. Hence problem-based, service-based, and other such philosophy‘s of learning. Even direct instruction posits meaningful problems. Again, the article‘s comparing an ideal human learning model compared to a broken school model.  

5. What motivates learners are real life outcomes. Really, we‘ve covered this, everyone learns better when there‘s motivation. Children learn for grades because no one‘s made it meaningful for them to care!   Kids will pursue their learning when it makes sense to them. John Taylor Gatto made the case that kids could learn the entire K6 curriculum in 100 hours if they cared! Kids do learn outside of what‘s forced on them from schooling, be it Pokemon, polka, or porcupines.  

Thus, in the comparison between andragogy vs pedagogy, I come down on the side of pedagogy. It‘s the earlier term, and while ped does mean ‘kid‘, I still think it‘s really about learning design. Learning design should be aligned to our brains, not differentiated between child and adult. Yes, there are developmental differences, but they‘re a continuum and it‘s more a matter of capacity, it‘s not a binary distinction. That‘s my take, what‘s yours?

I stumbled across the Elements of UX diagram again, and happened to wonder if it would map to LXD. Here’s my stab:

And the text, as usual.

---

In a justifiably well-known image (PDF), Jesse James Garrett (JJG) detailed the elements of (web) user experience. I‘ve been involved in the parallel development of UX and ID (and cross-fertilized them), so I wondered what the LXD version would be. So, of course, I took a stab at levels of LXD design.

To start with, JJG‘s diagram works from the bottom up. The five levels, in order, are:

1. The original objectives and user needs.
2. That leads to content requirements and/or functional specifications.  
3. The next level is an information architecture or interface design that is structured to meet those needs.  
4. Those semantic structures are then rendered as an information design with navigation or interface design.
5. The top level is the visual design, what the user actually sees or experiences.

This systematic breakdown has been well recognized as a useful development framework. The development from need to semantics to implementation syntax suggests a logical development flow. As an aside, no one‘s claiming we should develop in a linear manner, and there tends to be more up and down action in actual practice. Drilling down and then working from the bottom up as well is a well-known cycle of design!  

The learning equivalent, then, should similarly have a structured flow. We want to go from our needs, through various levels of representation, until we reach the learner experience.  

Given that we should be driven not by the goals for the interface but learner needs, I‘ll suggest we start with the performance objectives.   Then, in parallel with user needs, I‘ll stipulate that the other top-level definition comes from the user characteristics. These match the initial level stipulated.  

At the next level, I‘ll suggest that the performance objectives drive assessment specifications, and the other decision at this level is for the pedagogical approach. We need to know what learners need to able to do, and how we‘ll get them there.

As an intermediate representation equivalent to UX‘s information architecture or interface design, I suggest from the assessment we determine the necessary practice activities required, and these are coupled with the necessary content requirements: models and examples, as well as the introduction and closing. Here we‘re still at what‘s required, not how it manifests.  

The next level is where we start getting concrete. We need to pick an overall theme or look and feel, and the flow of the experience. We‘ll also, of course, need to make a consistent interface to support navigation and taking action. We know what we need to have, but we haven‘t actually rendered it yet.  

Finally, we must render the necessary media. This will be the videos, audios, text, diagrams, images, and more that comprise the experience. This includes the actions to be taken and the associated consequences of each choice.  

That‘s the equivalent structure I‘m suggesting are the different levels of LXD design. Of course, this is a thought exercise, and so I may well have made some interpretations you could disagree with. For instance, I may have slavishly followed JJG’s levels too closely. Let me know! Also, it‘s not clear whether this is a useful representation, so far it‘s sort of a ‘because it‘s there‘ effort ;). You can let me know your thoughts on that, too!