How to Measure eLearning ROI: From Smile Sheets to Hard Numbers

Investing in employee development has become a good tone for companies, and for some time it was enough to persuade the suits to launch a training program. Now, they’re asking many more questions. The most common: what is the payoff? In 2018, in the US only, total training expenditures amounted to $87.6 billion. It’s only fair that in return, company leaders want to know what value eLearning initiatives bring in terms of numbers.

In this blog post, we’ll introduce you to the ROI methodology and provide you with a detailed plan on how you can show training value to stakeholders. But first, take our short quiz to find out if your company is ready to get started with an ROI evaluation.

What’s ROI & Why Should We Care?

Return on investment, or ROI, is an ultimate measure of success that shows the ratio of what you put in and get back as a result. The idea of calculating ROI is to convert both training costs and benefits into a single value, e.g. dollars. This is actually great because you can make a direct comparison and come to a clear conclusion on the effectiveness of training.

The superpower of ROI is that the metric is familiar to any accountant or financial expert, and has credibility among executives. And that’s a big deal. If they can easily see the relation between training efforts and business results, the training project draws much more attention and thus gets more respect and resources.

The formula for calculating eLearning ROI looks like this:

A positive ROI means that the training has paid off, and a negative ROI indicates losses.

Example 1: The Beta company has invested $65,000 in employee training. As a result, they managed to increase sales and earned $175,000.

ROI = (175,000 – 65,000)/65,000*100 = 169%

So, for every dollar invested in the program, the company got $2.69.

Example 2: The Gamma company spent $25,000 on training. Trained employees brought the company an additional $7,500.

ROI = (7,500 – 25,000)/25,000*100 = -70%

In this example, the ROI is negative, which means that obtaining an additional $7,500 wasn’t worth the efforts and resources put into the training.

Now, it looks easy. What’s the holdup?

It’s easy to multiply and divide numbers when it comes to abstract examples. In practice, finding out how much you managed to earn by assigning employees a certain course can be a challenge, especially if the course is not directly related to sales or something that has a direct monetary value.

It is worth remembering that the ROI formula itself isn’t a magic ball that answers any question. Let’s say a group of analysts told you that the ROI of your last course was 7%. So what?

• Is it a good or bad ROI?
• What prevented you from getting a higher ROI?
• How can you increase the ROI?
• How do you know that the result was obtained from training, and not from new technology or marketing?

Without additional data gathered, the ROI metric doesn’t tell the whole story. For a correct interpretation of the results, we need an established methodology that includes other performance metrics as well. Let’s look at this methodology in more detail.

Performance Evaluation Models: Kirkpatrick vs Phillips

The most renowned model for measuring training effectiveness is Donald Kirkpatrick’s 4-level model. The stumbling block in Kirkpatrick’s methodology was that he supposed that trying to measure training results in money was often impossible or didn’t make sense. That’s why Jack Phillips upgraded Kirkpatrick’s model and presented his 5-level ROI Methodology.

A comparison of the Kirkpatrick and Phillips models

Phillips refined the model and added Level 0 (Inputs) and Level 5 (ROI) — the hard numbers stakeholders are craving. He also developed approaches for isolation of learning effects from the influence of other factors, such as marketing. This seriously increases the credibility of the whole evaluation.

“Yes, all the solutions are working in harmony, but you need to know the relative amount of each contribution. If you do not know what caused the results, you can’t manage it. You can rest assured that marketing will show the effects of advertising, promotions, and even pricing strategies. Marketing is ahead of us on this issue and they work hard at it. If we do nothing, as the Kirkpatricks suggest, we lose credibility and support.”

Jack Phillips

Response to LinkedIn Post: Kirkpatricks vs. Phillips

How to Calculate eLearning ROI

According to Phillips’s methodology, ROI evaluation consists of four stages: planning, data collection, analysis, and reporting.

The ROI Methodology Process Model. Source: ROI Institute

1. Set learning objectives

Clear goals put down on paper guarantee that you’ll be asking the right questions and won’t miss anything while collecting information. Patti Phillips recommends elaborating objectives from the top and then go down the list, starting from ROI metrics and finishing with learners’ reactions to the training.

2. Work out a plan

At this step, you develop and work through three documents: a data collection plan, a data analysis plan, and an ROI calculation plan. This will be your project scope. It’ll allow you to monitor the progress and let you evaluate your own success in the evaluation process.

The project scope should contain information about the objectives of the training, how you’re going to achieve them, a list of tools, and the deadlines. You also need to determine who is responsible for the implementation of each of the plan items.

You can download a free project plan template and other super-useful resources on the ROI Institute website.

If you invest some effort in this task, you’ll get a transparent step-by-step guide for implementing the project. Sign it off with the stakeholders; this will help you make sure that your expectations coincide, and agree on the criteria for marking each stage of the plan as completed.

3. Collect data during and after the program

At this step, we apply the traditional Kirkpatrick approaches: we evaluate the learners’ reaction to the studied content and how well it was digested. After the training is completed, we monitor how the acquired knowledge is applied in practice and what result it brings.

Here are some tools that can be used for gathering information:

• reaction surveys or smile sheets
• polls
• interviews
• focus groups
• tests
• case solving
• check sheets
• timesheets and observations

An example of evaluating changes in employees’ behavior (Level Four).

 4. Isolate the training effect

The business outcome is affected by a large number of different factors, from pricing strategies to the personal charm of a certain sales rep. Isolation is an essential step that helps to determine the amount of improvement directly related to the program. It ensures the credibility of the whole evaluation methodology.

Here are the most common isolation techniques:

• Control groups: You run a soft launch for a pilot group and compare results with those who didn’t participate in training.
• Learners’ estimates: Participants estimate the amount of improvement that is related to the program.
• Supervisors and managers’ estimates: Team leaders estimate the impact of the program on the output.

5. Convert results to monetary values

To implement this step, you need to put a dollar tag on each result connected with the training program, so that you can compare it with the program costs. As a rule, hard data such as performance, time, and quality can be easily converted:

• Extra sales → profit from these sales
• Employee time savings → cost of the saved time calculated on the basis of the average rate per hour
• Reduction of equipment downtime → can be calculated in comparison with the results of the past periods
• Reduction of employee turnover → money that you save on recruitment and onboarding
• Quality improvement → money that was saved thanks to reducing the number of defective items

6. Estimate training costs

There is a Step 0: Inputs at the bottom of the Phillips pyramid; now it’s time to revise and fix the final costs of the program. This may include both direct and indirect expenses: instructors’ salaries, the cost of an LMS, the cost of a PR campaign for the program, the purchase of equipment, and so on. And don’t forget to include the cost of the very evaluation of the program performance.

7. Calculate ROI

Now, it’s time to get back to the formula we already mentioned:

As a rule, ROI is calculated per year. If there are no desired results in the first year, there’s probably no chance there will be in the future. But in some cases, when you launch a long-term large-scale program, such as a corporate university, it may take several years to achieve the maximum effect. You need to consider that as well.

8. Identify intangibles

In most cases, training programs bring intangible benefits along with monetary results. For example, they can improve the public image of the company and increase the team spirit and loyalty of the employees.

The challenge is that soft data can be subjective, so attempts to convert it into monetary values and use in an ROI calculation can seriously affect the accuracy of the final results. Therefore, it’s better to show a lower ROI metric and provide a strong report on intangible benefits than to demonstrate higher ROI while risking the credibility of the entire program evaluation.

If you have doubts on whether to convert or not to convert, use the guidance from the ROI Institute:

To convert or not to convert. Source: ROI Institute

9. Report results

The last but not least step in evaluating training performance is reporting. This step may require some additional planning, as the future of the project depends on it, and the stakes are high.

At this point, you have two major tasks:

a) Prepare reports for different audiences

Think about what kind of reports you need to prepare. Different audiences need different information, but most likely you’ll need two versions of each report: a detailed one and a basic one.

b) Communicate results

How are you going to present the report to a certain audience? Make a list of channels you can use, and adapt reports for them. For example:

The Key Takeaways

• The ROI metric shows the payoff of training programs by converting training costs and benefits into money.
• Calculating ROI isn’t a goal in itself. Without an established methodology, you won’t be able to interpret the results correctly.
• It’s critical to use isolation techniques to ensure the credibility of the evaluation.
• It’s better to have a strong report on intangibles and lower ROI than to show a higher ROI and risk the credibility of the evaluation.
• Evaluation is meaningless without proper communication of results.

Further Reading on the eLearning ROI:

• Implementing the Four Levels: A Practical Guide for Effective Evaluation of Training Programs, Donald L. Kirkpatrick, James D. Kirkpatrick

The classic book on using Kirkpatrick’s model and measuring learning outcomes at four levels.

• The Bottomline on ROI, Patricia Pulliam Phillips

The absolute must-read book if you’re going to measure ROI for training programs. The author generously shares use cases, ready-made tables, and forms that will save you plenty of time. She even gives an example of a speech to defend the project at the stakeholders’ review.

• The Phillips ROI Methodology e-course on Lynda

Everything you need to know about the methodology in less than an hour.

• Understanding LMS Reports: 12 Things to Look at in Your eLearning Statistics

To calculate ROI, you will need to collect and analyze a huge amount of information. The good news is that this process can be automated at least partially. In this article, you’ll learn what kind of statistics can be gathered by an LMS, and what conclusions can be made with its help.