strategy Archives - Page 56 of 83

Assessing online assessments

9 May 2013 by Clark 3 Comments

Good formal learning consists of an engaging introductions, rich presentation of concepts, annotated examples, and meaningful practice, all aligned on cognitive skills. As we start seeing user-generated online c, publishers and online schools are feeling the pressure. Particularly as MOOCs come into play, with (decreasingly) government funded institutions giving online content and courses for free. Are we seeing the demise of for-profit institutions and publishers?

I will suggest that there’s one thing that is harder to get out of the user-generated content environment, and that’s meaningful practice. I recall hearing of, but haven’t yet seen, a seriously threatening repository of such. Yes, there are learning object repositories, but they’re not yet populated with a rich suite of contextualized practice.

Writing good assessments is hard. Principles of good practice include meaningful decisions, alternatives that represent reliable misconceptions, relevant contexts, believable dialog, and more. They must be aligned to the objectives, and ideally have an increasing level of challenge.

There are some technical issues as well. Extensions that are high value include problem generators and randomness in the order of options (challenging attempts to ‘game’ the assessment). A greater variety of response options for novelty isn’t bad either, and automarking is desirable for at least a subset of assessment.

I don’t want to preclude essays or other interpretive work like presentations or media content, and they are likely to require human evaluation, even with peer marking. Writing evaluation rubrics is also a challenge for untrained designers or experts.

While SMEs can write content and even examples (if they get pedagogical principles and are in touch with the underlying thinking, but writing good assessments is another area.

I’ve an inkling that writing meaningful assessments, particularly leveraging interactive technology like immersive simulation games, is an area where skills are still going to be needed. Aligning and evaluating the assessment, and providing scrutable justification for the assessment attributes (e.g. accreditation) is going to continue to be a role for some time.

We may need to move accreditation from knowledge to skills (a current problem in many accreditation bodies), but I think we need and can have a better process for determining, developing, and assessing certain core skills, and particularly so-called 21st century skills. I think there will continue to be a role for doing so, even if we make it possible to develop e necessary understanding in any way the learner chooses.

As is not unusual, I’m thinking out loud, so I welcome your thoughts and feedback.

Travel Tech

2 May 2013 by Clark 1 Comment

Yesterday I wrote about some products, and I thought I should also own up to the mobile apps I use while traveling (at least domestically, international is still a bloody headache). It’s something I do a fair bit, and is a natural opportunity for mobile to make your life easier and more effective.

First, the natural functions of basic apps are helpful. I put my flight details and a reminder into my calendar. 3 hours before the flight, unless it’s a connection, then 40 minutes to alert me to get to the gate (United used to have an option to automatically download it to your calendar, but that changed with the software switch on the integration with the proud bird). I also put in reservations for cars and hotels. I keep track of the confirmation number that way and don’t have to carry around an extra piece of paper. The camera is useful too, when I need to remember my parking space. Easier than entering into the calendar! And I have a password app (I use SplashID since I had it before on my Treo) where I store all my membership numbers for the loyalty programs. May as well get the benefits if you have to travel. And Google Search gets used for lots of things.

A I mentioned yesterday, Navigon is GPS software that I’ve used many a time to get from place a to place B. I try to avoid driving if at all possible (such a waste of time, give me a train any time), but when I need to in or to an unfamiliar destination, GPS is the go. These days Google Maps does a very good job too, but if you’re going somewhere with dodgy cell coverage, having maps local is nice (if battery abusive: keep a charger). Google maps in particular is very useful for walking directions and times, too.

I use the iBart app to check train schedules to and from the airport. There are lots of apps out there to facilitate using particular train systems, and I’d use Metro in other towns if I were using public transit, e.g. Boston or DC. If you live in a particular location, check and see if there’s an app for your system.

On occasions, I use SuperShuttle (I try to be frugal when time allows), and their app lets you book the trip, check on your van, etc. When needed, it’s quite useful. TaxiMagic would be used sometimes if I had trouble getting a cab (I can recall one time in Philly where it would’ve been very handy).

When I do have to drive, CheapGas helps you find the prices of petrol near you and find a provider with the best deal. Other special purpose driving apps are RoadAhead (finding things at turnouts ahead; but it would require someone else in the car with you) and the AAA and Roadside apps, which can help you find accommodation or help you with car trouble. Thankfully haven’t needed them, but nice to have.

At airports, I love GateGuru. I try to get to the airport early (I’d rather be cooling my heels with a book or an app than sweating whether I’ll make it thru security on time), and if I have time to kill or need to grab a meal or a drink, GateGuru finds the opportunities nearby and has ratings. Very helpful.

I’ve the SeatGuru app, but I tend to use the website, as it can be helpful for choosing the best seating position, particularly when you’ve got a choice and the extra considerations aren’t obvious (loud, limited recline, etc).

When I’m looking for a place to eat, Yelp can be very helpful (in fact, finding us the nice Twin Cities Grill in Minneapolis just last week). You can indicate where you are and look for what’s around. Google Maps can do this too, but Yelp’s somehow a little better, optimized as it is for this purpose. On occasion I’ll use or coordinate with UrbanSpoon.

Finally, a shoutout to United. I’ve been sucked in for years (long story, started when they were the last option when I lived in Sydney), but whether you like the service or airline or not, their app is a great example of mobile support. You can review your flights, get your boarding pass, check flight status, get your mobile QR code boarding pass, and even book a flight. Really nice job of matching user need to functionality.

So, what apps have made your life easier when you travel?

Types of thinking

30 April 2013 by Clark 5 Comments

Harold Jarche reviews Marina Gorbis’ new book The Nature of the Future, finding value in it. I was intrigued by one comment which I thought was relevant to organizations. It has to do with the nature of thinking.

In it, this quote struck a nerve: “Gorbis identifies unique human skills”. The list of them intrigued me:

Sensemaking
Social and emotional intelligence
Novel and adaptive thinking
Moral and ethical reasoning

While all are intriguing and important, the first and third really struck me. When I talk about digital technology (which I do a lot :), I mention how it perfectly augments our cognitive architecture. Our brains are pattern-matchers and meaning extractors. They’re really good at seeing insights. And they’re really bad at rote memory, and complex calculations.

Digital technology is exactly the reverse: it’s great at remembering rote information and in doing complex calculations. It’s extremely hard to get computers to do good pattern-matching or meaning making.

For the purposes of achieving meaningful outcomes, coupling our capabilities with digital technology makes a lot of sense. That’s why mobile makes so much sense: it decouples that complementary capability from the desktop, and untethers our outboard brain.

From an organizational point of view, you want to be empowering your people with digital augmentation. From a societal point of view, you want to have people doing meaningful tasks where they tap into human capability, and not doing rote tasks. They’re going to be bad at it! And, you can infer, it’s also the case that you’re going to want education to focus on how to do problem-solving and using digital technology as an augment, not on doing rote things and memory tasks. Ahem.

Designing Higher Learning

29 April 2013 by Clark 6 Comments

I’ve been thinking a lot about the higher education situation, specifically for-profit universities. One of the things I see is that somehow no one’s really addressing the quality of the learning experience, and it seems like a huge blindspot.

I realize that in many cases they’re caught between a rock and a hard place. They want to keep costs down, and they’re heavily scrutinized. Consequently, they worry very much about having the right content. It’s vetted by Subject Matter Experts (SMEs), and has to be produced in a way that, increasingly, it can serve face to face (F2F) or online. And I think there’s a big opportunity missed. Even if they’re buying content from publishers, they are focused on content, not experience. Both for the learner, and developing learner’s transferable and long-term skills.

First, SMEs can’t really tell you what learners need to be able to do. One of the side-effects of expertise is that it gets compiled away, inaccessible to conscious access. Either SMEs make up what they think they do (which has little correlation with reality) or they resort to what they had to learn. Neither’s a likely source to meaningful learning.

Even if you have an instructional designer in the equation, the likelihood that they’re knowledgeable enough and confident enough to work with SMEs to get the real outcomes/objectives is slim. Then, they also have to get the engagement right. Social engagement can go a good way to enriching this, but it has to be around meaningful tasks.

And, what with scrutiny, it takes a strong case to argue to the accrediting agencies that you’ve gone beyond what SMEs tell you to what’s really needed. It sounds good, but it’s a hard argument to an organization that’s been doing it in a particular way for a long time.

Yet, these institutions also struggle with retention of students. The learners don’t find the experience relevant or engaging, and leave. If you took the real activity, made it meaningful in the right way, learners would be both more engaged and have better outcomes, but it’s a hard story to comprehend, and perhaps harder yet to implement.

Yet I will maintain that it’s both doable, and necessary. I think that the institution that grasps this, and focused on a killer learning experience, coupled with going the extra mile to learner success (analytics is showing to be a big help here), and developing them as learners (e.g, meta-learning skills) as well as performers, is going to have a defendable differentiator.

But then, I’m an optimist.

Robinsons’ Performance Consulting

22 April 2013 by Clark Leave a Comment

As a consequence of my previous post and the commented revelation, I checked out the Robinsons’ Performance Consulting. The book really takes a different approach to what I was talking about so I suppose it’s worth delineating the difference.

What I was talking about was how learning & development groups should be looking not only to courses, but also performance support and social media as components of potential solutions to organizational needs. It naturally includes a focus on aligning with business needs, but takes a rich picture of opportunities to have impact.

The Robinsons’ book is more focused on ensuring the project you’re working on is addressing the real problem. It rightly has you stepping back to look at the business problem (the gap between how things should be, and how they are), and the reasons why these gaps exist. Then, you should be designing solutions that address all the needs, and systematically solve the problem.

There are two really good things about their approach. The focus on the real problem is designed to prevent using a solution that may be familiar, but may not really solve the problem. This was similar to my complaint. The other thing is they’re willing to go beyond courses for solutions, looking at incentives, job aids, work process redesign, etc.

On the other hand, it’s not clear to me that they would be able to incorporate potential social solutions in their repertoire. When should you have people go to their network? An interesting question, but not one that obviously flows out of their approach.

Regardless, I think the rigor of the book (and it’s nicely complemented by exercises, examples, etc.) makes it a worthy contribution. I suspect that too many L&D groups might not be willing to push as hard as needed to make sure that the solution being developed is pointed to by a thorough analysis (as many of their examples point out), and this book gives you guidance and tools to do the job. There’re also some end chapters on being a successful consultant that could be valuable to practitioners as well. Well worth a read.

Increasing our responsibility

9 April 2013 by Clark 4 Comments

I ranted a couple of weeks ago about how we need to move out of our complacency and make a positive change. As I sometimes do, I stumbled upon a diagram that characterizes the type of change I think we need to be considering.

The perspective riffs off of the concept of the relative value of formal versus informal learning methods shift as performers move from novice to expert. (And, as I’ve previously noted, what’s considered in/formal changes depending on if you’re the performer or designer.) And, too often, we tend to restrict our interventions to the formal side, yet there are lots of things we can be doing on the informal side.

Largely, however, I see learning and development (L&D) groups as focusing exclusively on novices, or to beginning practitioners, and leaving practitioners and experts on their own. Even if they’re addressing these more advanced audiences, they tend to use the ‘course’ as the vehicle, when it’s not really necessary. These audiences know what they need to know, and just want that useful information, they don’t need the full preparation that novices do. Novices don’t know what they need to know nor why it’s important, so we provide all that in a course model. We can be much more telegraphic to advanced performers, and the value of social networks starts kicking in here too.

The point I’m trying to make is that we can, and should, take responsibility for the rest of the performers. We can assist their performance, hence the term we’ve been preferring in the Internet Time Alliance: performance consultant. This implies facilitating performance across the organizational roles, top to bottom and from beginner to expert.

I’d like to suggest that L&D groups need to become focused on facilitating organizational performance, which includes but is not limited to training. It’s going to benefit the organization, it’s going to lead to greater strategic contributions and associated value, and it’s an approach that will likely preclude a long slow march to irrelevance and extinction. Better the folks that understand how we learn and perform (and if you don’t, what are you waiting for?) take responsibility than having it devolve by default to business units and/or IT, eh?

#itashare

Games & Meaningful Interactivity

8 April 2013 by Clark 5 Comments

A colleague recently queried: “How would you support that Jeopardy type games (Quizzes, etc.) are not really games?” And while I think I’ve discussed this before, I had a chance to noodle on it on a train trip. I started diagramming, and came up with the following characterization.

I separated out two dimensions. The first is differentiating between knowledge and skills. I like how Van MerriÃ«nboer talks about the knowledge you need and the complex problems you apply that knowledge to. Here I’m separating ‘having’ knowledge from ‘using’ knowledge, focusing on application. And, no surprise, I’m very much on the side of using, or doing, not just knowing.

The second dimension is whether the learning is essentially very true to life, or exaggerated in some way. Is it direct, or have we made some effort to make it engaging?

Now, for rote knowledge, if we’re contextualizing it, we’re making it more applied (e.g. moving to the skills side), so really what we have to do is use extrinsic motivation. We gamify knowledge test (drill and kill) and make it into Jeopardy-style quiz shows. And while that’s useful in very limited circumstances, it is not what we (should) mean by a game. Flashy rote drill, using extrinsic motivation, is a fall-back, a tactic of last resort. We can do better.

What we should mean by a game is to take practice scenarios and focus on ramping up the intrinsic motivation, tuning the scenario into a engaging experience. We can use tools like exaggeration, humor, drama, and techniques from game design, literature, and more, to make that practice more meaningful. We align it with the learners interests (and vice-versa), making the experience compelling.

Because, as the value chain suggests, tarting up rote knowledge (which is useful if that’s what we need, and sometimes it’s important, e.g. medical terminology) is better than not, but not near as valuable as real practice via scenarios, and even better if we tune it into a meaningful experience. Too often we err on the side of knowledge instead of skills, because it’s easy, because we’re not getting what we need from the SME, because that’s what our tools do, etc, but we should be focusing on skills, because that’s what’s going to make a difference to our learners and ultimately our organizations.

What we should do is be focusing on better able to do, moving to the skill side. Tarted up quiz shows are not really games, they’re simplistic extrinsic response trainers. Real, serious, games translate what Sid Maier said about games – “a series of interesting decisions” – into a meaningful experience: a series of important decisions. Practicing those are what will make the difference you care about.

Hire the ‘loud’?

3 April 2013 by Clark 1 Comment

In thinking about how organizations can ‘learn’, it strikes me that everyone needs to be simultaneously learning and teaching. How does that happen? I think it can be scaffolded, but it may also be an inherent trait.

A number of us are talking more about working out loud: Jane Bozarth and Harold Jarche talk about ‘narrating your work’, while I go on about ‘thinking out loud’ and ‘learning out loud’. The point is capitalizing on the benefits that come from putting your thoughts out: people can give you feedback, helping you learn; and folks can learn from you.

And, as I’ve said before, conversations are the engine of business. You need to be interacting to be advancing.

The recent story of Marissa Mayer, CEO of Yahoo, struck me as an interesting case. Here she’s bringing in folks who’ve been working remotely, or to put it another way she’s not allowing telecommuting any more. While there are obvious downsides, I can think of two justifications for that step:

to get everyone back on the same page in regards to mission and vision
to have folks sharing more

Both of these would be good outcomes for Yahoo. And I can see in both cases that it could be temporary: once you get the mission message shared, and have developed a culture of and infrastructure for sharing, folks could then again work from where they want. Of course, I have no idea whether that will actually happen.

The interesting thing for me was to contemplate those folks who don’t share. What to do? I know of folks who are happy to sit at home and do their job, and aren’t necessarily interested in the larger picture. What do you do? Sometimes these folks have useful skills. And they may have their own methods of keeping up to date. But if they’re not sharing, not contributing, what’s the overall picture?

And the thought occurred to me that those are folks that you bring in as contractors or consultants, but not as employees. Particularly in the case of a ‘no fire’ policy, who do you want on board? It seems to me that the employees you want are the ones who are continually learning and contributing to the organization’s overall knowledge.

Sure, there’s lots more you’d have to get right: safety to speak out loud, tolerating diversity, openness to new ideas, but having folks who are willing to learn together seems to me to be one criteria for an organization that will thrive.

So, is this a plausible component of a hiring policy? Those who demonstrably narrate their work are the ones to attract, develop, and reward?

#itashare

Reflections on Experience

27 March 2013 by Clark Leave a Comment

The API formerly known as Tin Can provides a consistent way to report individual activity. With the simple syntax of <who> <did> <this> (e.g. <Clark Quinn> <wrote> <a blog post>), systems can generate records across a wide variety of activity, creating a rich base of data to mine for contingencies that lead to success. While machine learning and analytics is one opportunity, there’s another, which is having people look at the data. And one person in particular.

As background, I was fortunate back in 1989 to get a post-doctoral fellowship to study at the Learning Research & Development Center at the University of Pittsburgh. One of the projects that had been developed was a series of intelligent tutoring systems (ITS) that shared an unusual characteristic. Unlike most ITS, which tutor on the domain, these three systems crossed domains (geometric optics, microeconomics, and electrical circuits, if memory serves) but the tutoring was about the systematicity in exploration. That is, the system tracked and intervened on whether you were varying one variable at a time, ensuring your data sampling was across a broad enough range of data points, etc. This reflected work done by the Valerie Shute and Jeffery Bonar some years before on your learning strategies.

I had the further benefit to work under the guidance of Leona Schauble, a very insightful researcher. One of her projects was with Kalyani Raghavan on working to make the learners’ paths in these systems visible and comprehensible to the learner, and they created the Dynamic And Reflective Notation (DARN, heh!) to capture and represent those paths.

Fast forward to today, and one of the big opportunities I see is for performers to reflect on their own paths of action. The granularity at which Tin Can can capture data, and systems might be instrumented to generate data, could be too small to be useful, so some way of aggregating activity to a reasonable level would be necessary, but looking at one’s own paths, and perhaps others, would be a useful way to reflect on process and look for opportunities to improve.

Reflection on action is a powerful learning and improvement process, but recollection isn’t as good as actual recording. The power of working out loud is really seen when those tracks are left for examination. The API has the opportunity to support more than system mining (“oh look, everyone who has this responsibility who touches this resource does way better than those who don’t”). Not that there’s anything wrong with that, but having performers do it too is a great opportunity not to be missed. As the work on protein folding has found, some patterns are better for computer solution, and others for human. We’d be remiss if we didn’t explore the opportunities to be found.

« Previous Page