‘Evaluation’ generally refers to an examination of the extent to which something seems suitable for achieving a desired result. In the evaluation of a training session, one aims to find out whether the training and its supporting measures are able to bring about desired success on a personal as well as company level. Therefore, evaluating training is part of a systematic measure to assure learning transfer success, as well as a means of assessing performance. The best-known model of training evaluation is the Kirkpatrick four-level model.
In 1959, the American economist Donald Kirkpatrick published his dissertation and later developed a series of articles in which he described the four-level model, a model which is still in use today in learning assessment. Evaluation based on these four levels can determine whether a completed training has indeed achieved desired results, i.e. whether critical behaviours have changed.
The Kirkpatrick Model
Level 1 – Reaction
At this level, participant engagement is measured and participants are asked to express whether they found the training relevant and whether they were satisfied to the level that they would recommend it to others.
Level 2 – Learning
AAt the learning level, an assessment is made as to whether increases in the dimensions of ‘knowledge’, ‘attitude’, and ‘skill’ have taken place. Participants are also asked whether they are confident they will be able to apply what they have learned and if they are motivated to actually put it into practice.
Level 3 – Behaviour
This level is an exploration of whether behaviour in the workplace has changed.
Level 4 – Results
Whether behavioural change has led to a positive change in the company’s overall performance is evaluated at this level.
Working the model in two directions: When setting up the evaluation plan, follow the motto ‘Have the end in mind’ – start with Level 4 and work your way down to Level 1. After the training has been developed, you begin measuring at all levels (now in the direction from 1 to 4 and then draw up the results and reflect this data back to your clients.
If the training design and training implementation has been planned along the four levels, it is easy to see how the training can benefit the company.
Evaluation according to Kirkpatrick
When creating the evaluation plan, you start at Level 4, i.e. the results. The typical questions at this level are: What are the overall goals and expectations of the company? What concrete results should be achieved in the future and how can they be measured? How can training have a positive impact on achieving these goals?
This process, therefore, is not concerned with the isolated world of the participant, rather it ascertains whether training will generate results for the company. Results are presented in the form of a Return on Expectations (ROE). The ROE demonstrates to clients/stakeholders the value of the training initiative by clarifying expectations of the training and, later, by measuring the extent to which they have been met. The ROE can be expressed in terms of e.g. ‘efficiency’, ‘productivity’, and ‘customer satisfaction’.
In order to be able to measure the organizational effects resulting from training, appropriate leading indicators are needed. Over time, these indicators reveal whether the training has had an impact on critical behaviours; these behaviours, in turn, positively influence the desired results. Measuring such leading indicators demonstrates in due course whether the training is able to consistently bring about the desired change. These processes define measurement criteria that demonstrate both change in the short term and over a longer period of time. Thus, with ongoing analyses of these different data groups, one can quickly intervene if necessary.
Internal leading factors may include, for example, rates of failure, production, and waste, as well as those of employee satisfaction and fluctuation and workplace safety. External leading indicators may include the number of new customers, customer satisfaction, referral rates, brand perception, complaints, and sales volume.
At the third level, the focus is on behaviour. Working together with the client, one to three critical behaviours that employees need to change to achieve desired results will be identified. To make that determination, it helps to work with the ‘Hans and Franz’ metaphor:
Hans is an employee who has participated in the training and implemented what he has learned. Franz, on the other hand, is an employee who did not participate in training and works as he always has. Questions to the client then focus precisely on the differences between the two: What does the trained employee, Hans, do? How does he work? What does Franz do differently? Do you recognize working differences between the trained Hans and Franz, the employee who did not attend the training? From this discussion, one can discern behaviours that need to be changed.
After training is carried out, measures are taken and checked to see whether participants put learned skills into practice, which goes hand in hand with the transfer of learning in the workplace.
To change behaviours, so-called performance drivers are used which are divided into two categories: monitoring and support. For each behaviour, monitoring options are sought. Interviews, observation, and dashboards are used to find out where the organization stands. In response to the results of monitoring, executive staff and the organization as a whole can take action where support is needed.
Employees need support in implementing behavioural changes. Different methods may help to strengthen new behaviours (e.g. follow-up modules, role modelling), help to encourage them (e.g. coaching), and reward them (e.g. recognition, bonuses).
The second level puts learning in the foreground. Five elements are important here and three of them are closely linked: ‘knowledge’, ‘attitude’, and ‘skill, synonymous with ‘head’, ‘heart’, and ‘hand’.
Helpful questions can be: Did the participants improve their knowledge and acquire relevant skills? What do employees need to know and use after training? What change in attitude is needed? What do employees need to learn to behave this way?
In addition, elements of ‘confidence’ and ‘commitment’ are queried – Am I confident that I am able to apply what I have learned? Am I committed to applying it? – and these are very interesting questions with regard to evaluation and retrospective reflection. Both questions can be answered in writing on the feedback form, but it is much more interesting to ask them in the room at the end of the training. If there are any doubts about feasibility expressed by the participant group, it is necessary to find out where these concerns are coming from and address how to alleviate any possible obstacles.
The answers to these two questions often have nothing to do with the training; instead, they examine the company and the resources at its disposal and the training participants themselves.
Measurement at this second level can be used to determine if the methods used to achieve the learning goal were effective. Learning success is measured from the point of view of the participants, and can prove effective when assessed immediately at the end of the seminar: How do I assess myself now compared to at the start of the seminar with regard to my current knowledge?
Satisfaction is also evaluated as the last, lowest level. The three elements are ‘customer satisfaction’ (in terms of participant satisfaction), ‘engagement, and ‘relevance’.
How is it ensured that the training is satisfying for all involved? How should the learning environment be designed so that the participants can learn optimally? How did the participants perceive the measures? Did the coach contribute to their learning success? Was the topic thoroughly addressed? Was the participant engaged? Was the content of the program relevant to the job?
Answers to these questions indicate whether the measures were even accepted by the participants – clearly a prerequisite for effectiveness. Positive reviews indicate that the training design and content presentation function in good order. A good measurement criterion at this point is the rate at which the training is recommended.
Based on all of the above, a specific type of training evaluation can be formulated, which Kirkpatrick refers to as a hybrid evaluation tool – i.e. a plan on all four levels.
The Kirkpatrick Evaluation requires the evaluation form to be prepared beforehand, after which the training can be developed and carried out. Starting with the first training, measurements are taken on a continual basis. During the training, feedback can also be gathered on an ongoing basis, easily done through questioning or, for example, a Plus-Delta review. Other options include knowledge checks and recaps, role-playing games, and group activities; i.e. all tools with which the trainer can recognize whether the participants are learning well. At the end of the training, a feedback form can be handed out. A further ongoing review of whether the training brings the desired behavioural changes can be carried out via post-testing, observation, feedback sessions with the manager, as well as in interviews or surveys.
The results of this process are discussed with the client at regular intervals and changes are made if necessary.
If you want to dive deeper into the subject, I recommend the book Four Levels of Training Evaluation (Kirkpatrick, J. and Kirkpatrick, W., 2016, ATD press).