Before an Organization begins deciding on what to train, an understanding of the organization’s goals should be in order. The goals come from the organization’s KPIs (Key Performance Indicators). KPI’s are like a compass that tells us the direction the organization is headed based on data. Kirkpatrick’s 4th Level of Evaluation of Training is based on Return on Expectations (ROE) or Return on Investment (ROI) for the Company.

At one time it was thought that the best a Training Organization could do was provide a Return on Expectations. But what are those expectations? How are they decided and what is the budget to make it all happen?

KPIs (Key Performance Indicators) are the goals a company wishes to achieve. Some examples of a KPI are:

Sales KPIs
» Number of New Contracts Signed Per Period
» Dollar Value for New Contracts Signed Per Period
» Number of Engaged Qualified Leads in Sales Funnel
» Hours of Resources Spent on Sales Follow Up
» Net Sales – Dollar or Percentage Growth
» New and Expansion Monthly Recurring Revenue (MRR)
» Number of Sales Calls Held
» Number of Sales Calls Scheduled
» Number of Customers Onboarded
» Number of Sales Calls per Representative
» Average Purchase Value
» Sales by Department
» Same Store Sales Growth
» Sales Per Area – Geography
» Sales Per Area – Square Feet

Marketing KPIs
» Monthly Website Traffic
» Number of Qualified Leads
» Conversion Rate for Call-To-Action Content
» Keywords in Top 10 Search Engine Results
» Blog Articles Published This Month

And so forth.

Key Performance Indicators (KPIs) are the elements of your plan that express what you want to achieve by when. They are the quantifiable, outcome-based statements you’ll use to measure if you’re on track to meet your goals or objectives. Just like Learning and Development is outcome based, Learning and Development can only prove a ROI if it is linked to KPIs. If an organization doesn’t know where it is going and what that place looks like then obviously Learning will take you there and that is not necessarily a good thing. It is just a thing and at best it gives a Return on Expectations (ROE) vs. a Return on Investment (ROI) no matter the results of the first three Levels of Evaluation. Training, just like KPIs, requires data. If we train someone to be more effective in increasing number of sales calls, then we need data on which team members had the training and which didn’t. Only with this data can we link training to KPIs and therefore declare Training ROI.

The next step is to conduct Gap Analysis. In many cases, the Instructional Designer assigned to a project may be in the dark on what are the true KPI’s of an organization and instead rely solely on the advice of departmental Functional Area Managers, Technical Area Managers or Curriculum Development Managers or Subject Matter Experts on what should be developed. Each is going to have an opinion and that opinion may or may not be based on data or even a KPI at all. If the desire for a type of training is demanded by a group then a KPI should match it at the organizational level or you will never achieve the 4th Level of Evaluation. In a survey conducted by Mathews et al. (2001) which focused on organizations in the United Kingdom, Finland and Portugal that were implementing quality management practices such as ISO 9001:2000, the following elements were ranked by importance in determining training needs.

  • Senior Management Decisions
  • Supervisor’s opinions
  • Skills inventory
  • Employee surveys
  • Analysis of projected business or service plans
  • Customer opinions
  • Training audits
  • Requests from work groups
  • External consultant
  • Advisory committee

Why did senior management decisions and supervisor opinions rank highest? After-all, these people sit in meeting after meeting (except for the Subject Matter Experts that do what they do well day after day and may or may not be in frequent communication with the FAMs or TAMs) discussing the needs of the organization and they should know right? Actually, it is more likely just perception. At one time that was true and should remain true today, however, xAPILMS leads us closer than ever to linking real-time data to not only ROE but ROI as well. xAPILMS is designed on providing dashboards of information created through Business Intelligence: Dashboards that are updated daily on the latest information in the Learning Record Store (LRS). Since these Dashboards are dynamic; Leaders, Designers and FAMs can access real time information.

An example of a Business Intelligence dashboard inserted into the Course: Texas Energy Overview created by Randy Miller as a demonstration of the xAPILMS dashboard functionality. Would you like to know the top 5 children in a School District of 28,000 kids, who asks the most questions of their teachers? An LRS and a Dashboard can slice and dice it all as this kind of information is constantly streaming into the database in real time as a student or team member moves through various training scenarios.

An example Dashboard used in the course Texas Energy Overview

Let’s say that Number of Sales Calls Per Month is a KPI for the company. Learning and Development is tasked with increasing this KPI result. Data would be crucial in understanding how some salespeople are making 100 calls per month and others are only making 30. It would also be nice to know how many calls are actually converted to sales. Maybe the KPI is misplaced and volume of calls is not correlated with actual sales and in our analysis phase of training development we discover this. Learning and Development therefore should be dependent on data not opinion. If these linkages are actually made, organizational training will move toward the ROI model vs. the ROE model.

The more we understand the underlying data accumulated in a Learning Record Store (LRS), the more we can develop training that moves the participant in a particular fashion. xAPI then becomes important for the entire training effort which should include the Program Management Office (PMO) including learning analysts, the Training Manager, Staffing, Functional Area Mangers from the line organizations (Manufacturing, Data, Drilling, Sales, Avionics, etc. etc.) and Subject Matter Experts finally chosen to assist the Instructional Designers in making the training happen.