I’m obviously a fan of innovation, and experimentation is a big component of innovation. However, I fear I haven’t really talked about the specifics. The details matter, because there are smart, and silly, ways to experiment. I thought I’d take a stab at laying out the specifics of experimentation.
First, you have to know what question you’re trying to answer. Should we use a comic or a video for this example? Should we use the content management system or our portal tool to host our learning and performance support resources? What’s the best mechanism for spacing out learning?
An important accompanying question is “how will we know what the answer is?” What data will discriminate? You need to be looking for a way to tell, we know, we can’t know, or we need to revise and do again.
Another way to think about this is: “what will we do differently if we find this?” and “what will we do differently if it turns out differently?” The point is to know not just what you’ll know, but what it means.
You want to avoid random experimentation. There are the ‘lets try it out’ pilots that are exploratory, but you still want to know what question your answering. Is it “what does it take to do VR” or “let’s try using our social media platform to ‘show our work'”.
Then you need to design the experiment. What’s the scope? How will you run it? How will you collect data? Who are your subjects? How will you control for problems?
One of the claims has regularly been “don’t collect any data you don’t know what you’ll do with”. These days, you can run exploratory data analysis, but still, accumulating unused data may be a mistake.
The after-experiment steps are also important. Major questions include: “what did we learn”, “do we trust the results”, and “what will we do as a result”. Then you can followup with the actions you determined up front that would be predicated on the outcomes you discover.
Experimentation is a necessary component of growth. You have to have a mindset that you learn from the experiment, regardless of outcome. You should have a budget for experimentation and expect a degree of failure. It’s ok to lose, if you don’t lose the lesson! And share your learnings so others don’t have to make the same experiment. So experiment, just like I did here; is this helpful? If not, what would it need to be useful?