by Glynn Jung
In this second instalment of learning analytics, Glynn discusses the classic approach to return on investment (ROI) for learning.
If you take the accountants’ approach to ROI for learning analysis there are five important points to note;
- The assumptions made before conducting the analysis are important and you must document them.
- It takes more than one ROI model to establish value, and not all ROI models will be valid for a given case.
- Collaboration with customers and senior management in identifying Learning benefits is critical; ROI determination is not a one-sided exercise.
- It is too easy to fall prey to the temptation to just “play with the numbers” until an acceptable result appears.
- Calculators can only “do numbers” – they can’t compute the value of the intangibles.
There are a number of classic approaches to show the financial impact that a given investment (your e-learning project) will have on a business.
The issue here is “How long will it take to get all the investment back?” Payback analysis results are expressed in months or years. This is calculated as the net investment amount divided by the average annual cash flow from the investment. The payback analysis is easy to use and easy to understand. However, it does not take into account the time value of money (which is addressed by another model, Net Present Value, or NPV). Payback also does not consider the financial performance of the investment after break-even Payback is best used to establish relative priority between potential projects.
Accounting Rate of Return (ARR)
This is another “simple” method for calculating the return on a major project. It gives a quick estimate of a project’s payback, supports comparisons between projects and it also considers returns for the entire life of the project.
Net Present Value (NPV)
Net Present Value is best used for long-term projects. It considers the time value of money- it expresses future cash flows in terms of their value today. While this is the strength of NPV, it also means that this method is not appropriate for projects that do not have clearly defined cash flows, or when the benefits of the project are not financial. NPV can be tricky!
Internal Rate of Return (IRR)
IRR is not as easy for non-accountants to understand or to calculate as NPV. I don’t even understand the terminology let alone the techniques.
Full business impact: the Balanced Scorecard
Many human performance interventions have complex effects on business results. In recent years, the best known method of impact assessment has probably been the balanced scorecard.
The balanced scorecard looks at the effect of a project in four areas:
- learning and
- internal processes.
It is holistic and long-term, and it is forward-looking. Financial results are still an important area considered, but they are not the only element.
If your organisation uses balanced scorecards it may be useful to relate the benefits of your Learning project to each of the four areas of the scorecard. Show how the program objectives relate to the objectives and important questions in each area. The emphasis is on process, not on metrics.
Measurement and accountability have long been the order of the day in most organisations. Those infamous three little words; return on investment (ROI) analysis has been a standard tool in a manager’s kit and deployed when needed. There are however a few challenges however with ROI:
- there’s a number of techniques or methods to learn and use;
- these techniques and methods sit more easily with accountants than learning and development specialists and
- determining ROI can be a long and ponderous process when often operating units within the organisation need us to address their needs very quickly.
So is there a rapid approach for an impatient world? Well yes, there are in fact two such approaches that I’ve been using: Critical Mistakes Analysis (CMA) and Fast-track Proficiencies.
But firstly “traditional ROI”: ROI is not a tool that you can use to prove the value of ALL Learning – it’s just a tactical tool, not a strategic weapon. We see it often used to justify a move from traditional training to self-directed eLearning but this is only really valid in regulatory training e.g. iterative Aviation Cabin Crew training for certification purposes. This is training which has to be done and proven to have been done. In this context we can reasonably claim to offer a return on investment via shortened time to competence and reduced or eliminated indirect costs such as shift rota cover, travel or machine / simulator time, expert support and tuition time.
Apart from those situations, I suggest that many organisations struggle to develop precise and meaningful ROI based on classic accountants’ techniques because:
- they rarely calculate the cost and value of people’s time and efforts;
- they rarely have precise targets for focusing the training – i.e. what it is that needs to change or improve and what are the benchmarks for this and
- there is little evidence of a direct relationship between training / learning and improved competence in the workplace.
We’ve struggled with this for decades. Way back in the 1970’s and 1980’s, I worked with IBM on their methodologies, (SATD and SATE) for demonstrating the value of training to the organisation. Like Kirkpatrick’s Level Four – improved organisational performance – it proved increasingly frustrating to prove the link or outcome.
Even when you have done your due diligence and used the appropriate method to calculate the expected return, getting decision-makers to accept your analysis requires presentation, negotiation, collaboration, compromise, and persuasion. Think of it as a consensus-based process. It’s certainly rarely a quick fix for a problem. Additionally some people, like me, struggle to work like an accountant.
So are there any ways of ensuring that targeted learning or training can really deliver benefits, and do so rapidly? Well, there are two approaches that I’ve been using for a number of years which can be proved to deliver a return on investment (training time, expert time, learner time et al). These two approaches are:
- “Critical Mistakes Analysis” and
- “Fast-track proficiencies”.
Critical Mistakes Analysis
CMA is a proprietary methodology offered by Cognitive Arts, a subsidiary of NIIT. CMA derived from a US Government initiative to determine if it could ever be possible to guarantee a return on investment for training. By analysing the data gathered from major Six Sigma (no, don’t switch off) projects they found that a small number of very common mistakes caused most of the damage in an organisation. This was in every sector imaginable – manufacturing, logistics, health care, education, research, pharmaceuticals etc.
They were even able to put a cost against these common mistakes and so define the value of fixing them. The multi-year research project upheld the 80/20 rule and also spawned a commercial organisation with proprietary methodology. The thing is – all organisations have loads of data, of anecdotal evidence to point to where we should focus our attention for fixing mistakes; even the IT Help Desk records – we don’t really need Six Sigma to find out what to fix. What typically is delivered as an intervention, by the way, is a 4 to 7 minute online tutorial and assessment for each fix, supported by simple online reference materials.
A CMA type approach will help reduce or eliminate the most damaging problems. Fast-Track Proficiencies on the other hand will help you deliver the key proficiencies essential to each different role. The approach came out of work done by Steve Rosenbaum and Jim Williams which resulted in a new, simple-to-implement set of techniques to help get employees up to speed in record time. It is all documented in their book: “Learning Paths: Increase profits by reducing the time it takes to get employees up-to-speed” (Pfeiffer and ASTD Press 2004), which includes a CD with templates and procedures for identifying key proficiencies. I was able to implement the techniques with a client immediately after reading the book. It even includes a rapid approach called 30/30 Learning Paths for tackling just a handful of proficiencies in a role, (not least to prove to yourselves that it really works). Again, what is typically delivered is a blend of short tutorials and online workplace support information. For more information visit http://www.learningpathsinternational.com, check out the contents of the book via the “look inside” facility on Amazon or just buy the book.
Glynn will continue to discuss the classic approach to ROI in our next blog edition. Follow us on twitter @aurionlearning for our latest blog articles and updates.