Posts Tagged ‘measure’

Objection #13: How Do You Measure ROI?

Jun 18, 2008 Social Learning SIG

Flash back to yesterday’s objection, “How will you measure that it is working” should have been labeled, ‘How do you measure if it is being used and adopted.” This is what I had in mind knowing that ROI would be a separate topic today.

Christine Martell pointed this out in her comment from yesterday when she said,

“Not so sure about this being a new way. Seems like your list is looking at the activity rather than the result. Aren’t we looking for behavior change at the level of contribution to the business? Things like are they doing their jobs more effectively?”

She is so right. We do want to want this to change behavior. But now even the behavior question is in question. What behavior? With traditional learning we expect certain behaviors to result and we try to measure that. With social learning we are looking for different behaviors, but the same end results. Let’s look at both of these.

The first is more cultural. It is adopting and changing more than just what they learn, but also how they learn. They are the ones in charge of their learning and teaching each other. Instead of being forced to go to training (something we can have control over the format and measurement), they are being allowed and encouraged to learn on their own. For many, this is a new concept. Not necessarily a new practice, but in a way it is a new permission that we let them learn from each other. This involves trust. It involves the expectation that they are accountable to their own learning. Here are some more ideas to measure.

The second is more of the traditional ROI. How do you know that it is producing bottom line results? So many people say, “You can’t measure this. There is no traditional ROI model that will work for this.” I agree but disagree. There should be an adaptation, but we can still see bottom-line results. And, honestly, the numbers are not always the best ROI. Because the ROI is then used to make decisions. The results that come out of these environments may not have a hard and fast ROI at first, or may be very difficult to calculate.  The subjective results, however, can be very powerful for those who make the decisions.  From them they may be more than on board - they back it 100%.  All this, but they may never have looked at a formal ROI on it.

This isn’t always the case. But my point is that we can’t overlook the stories, the experiences that are so powerful - and then sharing them - in our quest for an ROI.

Rachel Happe suggested some measurements of ROI. A lot of them are for environments that face the customer, but some are for internal. Among those were:

# Number of new product ideas
# Idea to development initiation cycle time
# Retention/Employee turn over
# Time to hire
# Prospect identification cost
# Prospect to hire conversion rate
# Hiring cost
# Training cost
# Time to acclimation for new employees

Remember, we are looking at the final outcome, not necessarily “did they learn”. Because, honestly, we don’t care if they learn if they don’t use it for the benefit of the company. So the benefit is what we measure.  Other’s measurements might be:

# How large one’s network is
# Number of meetings taking place (or, more intuitively, are NOT taking place)
# Number of travel arrangements made (or, again, NOT made).

This is certainly not an exhaustive list. What others are you thinking of?

Tags: , , ,

Objection #12: How Will You Measure That It Is Working?

Jun 17, 2008 Social Learning SIG

When anyone throws up Objection #12 you know that they don’t quite ‘get it’ yet.  And that is OK.  Your job is to help them get it.  Some times we hear “Right now we know when people have learned.  We have them take tests.  They fill out evaluations.  We can see in the LMS that they have taken a course.  How will we measure this learning?”

Answer: It is true that you will not be able to see who has taken which courses or that they have passed a test or that they really liked a class (Level 1 evals - which is what most organizations do anyway).  We cannot measure something new using the old methods.  We can, however, measure using other methods not native to training or learning.  The methods used are web analytics.

  • When millions of people search on Google, what are they trying to do?  They are trying to learn.  You can measure the number of times employees perform a search.  They want to learn something from every search.
  • Then, look at what they are searching for.  That will give you a great insight.
  • What information are they sharing?
  • What are they commenting on?
  • How many wiki pages were created?
  • What were they created for?
  • How many blog postings were created?
  • When was the last time someone logged in? (if logging in is part of the process)
  • How many forum questions were asked?
  • How many answers were given?
  • What are the most viewed pages?
  • Define ‘active’ and measure how many people are active.
  • How many people have become active in the last _______?
  • You can analyze not only where people are going, but how they get there, how long they stay and what they do when they are there.

This may seem odd for some.  But we also need to remember that much of what they learn we cannot account for.  It happens, but we have no way to measure it.  Although this does not verify the transfer of knowledge or skills, it is a pretty good indication.  It is a new way of measuring learning.

Tags: , , , ,