Are You Letting Your LMS Decide How You Measure Learning?

We all know that metrics are essential to understanding the performance of learning programs. But what should we be measuring in order to assess whether we’re making progress, providing value, and supporting the needs of the organization with learning programs?

Most learning management system (LMS) solutions come with stock reports that allow you to see how people are using the system and interacting with the content. These include:

• Utilization—What content is being consumed and by whom?
• Completion Percentage—Do learners drop out at a certain point?
• Evaluation—Are the programs rated highly by learners?
• Time Spent—How much time is being spent on learning programs?

The reason these are the standard metrics is that they are easy things for the system to measure. These stock reports can be created with the click of a button, but unfortunately, they provide very little insight into the ROI of learning programs themselves. But because they are easy metrics to produce, organizations often fall back to using them as key performance indicators (KPIs) instead of finding ways to measure what is meaningful for the business.

Measuring the value of learning programs (which is ultimately the KPI you are looking for) is more challenging and depends on having a clear understanding of the goals of those programs. An important first step is understanding the different types of programs, because each one needs to be measured in a different way:

• Policy compliance programs: Often presented as “read and acknowledge,” the goal of these programs is to make sure employees are aware of a policy, and to create a record for reporting purposes. For these, the value from the LMS is the ability to easily report on each of these programs so that when an auditor arrives and asks to see the records, they can be quickly and easily provided. Often the “training” value of these compliance programs is minimal or nonexistent.
• Skill-based learning programs: Ranging from click-through PowerPoints to complex simulations and virtual reality-based learning, the majority of programs we see delivered through learning management systems fall into this category. The goal here is skill building. As such, the measure of success is whether learners are able to acquire the skill, and then demonstrate their competence. Typically, this means the programs include knowledge checks or other demonstrations of understanding or application. When creating KPIs for these programs, consider completion percentage, as well as performance on the knowledge checks. In addition, it’s also important to track learners’ evaluation of the programs in terms of whether they felt the content was valuable, whether it helped them increase their skills on the job, etc. And finally, you want to be able to in some way connect these skill-based programs to organizational capability models to ensure you are developing the skills you need in the organization, both now and for the future.
• Soft skills and leadership learning programs: Soft skills training is often the hardest to measure. Well-designed leadership and soft skills programs typically are delivered as a hybrid of synchronous live learning (either classroom-based or live virtual sessions), online programs, supplemental materials, and observational learning. As such, only some of the elements of the program would be delivered through an LMS, and the true measure of success ultimately would be measured via engagement surveys or other types of leader evaluations. It is possible to gather KPIs on completion and have knowledge checks similar to the skill-based learning model, but the ultimate behavior change would be visible only to managers and colleagues on the job.

Going Beyond Built-In Reports

As we look at the measurement of learning programs in this context, we can start to see the need to evolve beyond the built-in reports that are provided. While they can be valuable for the first category (compliance programs) in order to easily report on who has completed those policy document acknowledgements, they have significant limitations in understanding total value. For example, we would want to be able to break down the total cost of program development, including any vendors involved. It is also important to calculate how much time learners are spending on these programs.

Once we have determined cost, we can consider the value received from the program. In the case of compliance programs, this can be measured in terms of a reduction in noncompliance events, successful audits, or other measures. In the case of skill-building programs, this may be measured against the cost to purchase a program from an external vendor, or it may be measured in terms of talent strategy metrics such as recruitment and retention costs. In the case of leadership and soft skills, these typically are measured through engagement surveys, manager feedback tools, or other pulse surveys.

While it can be tempting to focus on what’s easy for a learning platform to measure as you think about metrics for your learning programs, it’s rare that you’ll be able to accurately assess value using just those tools. As you pursue the goal of effective KPIs, be guided by the needs of the business—not the reporting capabilities of the system.

Katy Tynan is a principal analyst at Forrester. She works at the intersection of people and technology. With a strong focus on leadership and organizational development, Tynan supports CIOs, CMOs, and chief human resources officers in their work to optimize customer experience through a highly engaged, inclusive workforce. She helps Forrester clients build a science-based approach to delivering industry-leading employee experience. The author of numerous books and publications, Tynan has deep expertise in the practice of leadership at all levels of the organization. Her most recent book, “How Did I Not See This Coming? A New Manager’s Guide to Avoiding Total Disaster,” was published by ATD Press in 2016. Tynan holds a BA in psychology from the University of Massachusetts at Amherst and is currently pursuing an MA in leadership and organizational communication at Northeastern University.