You usually measure a return on investment (ROI) in dollars or Dirhams, not how many times you walked to the bank. Why, then, do people still insist on counting how many people turned up for a workshop (the traditional ‘bums on seats’ number) and then offering that number as a measure of the value added for the time (and money) spent?
The fact that all 5/50/500/whatever employees in your organization had three training days this year has virtually no bearing on the learning that may (or may not) have occurred.
One of my favourite thoughts about learning is, “Learning is measured by a change in behavior.” If you are still doing what you always did, not a lot of learning has occurred regardless of how many days you spent in the training room! But how do you know if a change in behavior has occurred?
Well, you need to know where people started from, and that means some form of selection or assessment prior to any training taking place. Then, you need some form of measurement of where they are following the training, and the ability to relate any change in behavior to the requirements of the organization (including, as we shall see later, the expectations of the organization).
Kirkpatrick and Philips
Anyone seriously involved in evaluating training or assessing ROI has at some time used or referenced the work of Prof Donald Kirkpatrick (Kirkpatrick, 1959; 1998; 2007) and/or Dr Jack Philips (Philips, J. J., 1983; 1998.).
Kirkpatrick’s model of 4 Levels of Evaluation identified Reaction (1), Learning (2), Behavior (3), and Results (4) are relevant labels for assessment. Philips’ 5 Level model added ROI as a fifth level to Kirkpatrick’s 4-level work, introducing cost-benefit calculations as an additional form of assessment.
These models are almost sufficient in themselves for any caring organization to rely on for a reasoned assessment of where their investment went.
The practitioner or assessor must take some care, though, in the application of the overall model and any individual levels. Kirkpatrick’s first level, Reaction, more commonly referred to as the “Happy Sheet,” is not necessarily the most relevant evaluation criteria for an expensive long-term series of workshops.
A competent evaluator can encapsulate Philips’ fifth level of ROI based upon cost-benefit analysis into the fourth level of Kirkpatrick if Kirkpatrick’s Results level is broadened to include bottom-line figures and accountability. That is, to take the measurement of training away from the trainers and put it squarely back in the hands of the managers responsible for P&L. To do this, the first action of the manager must be to insist on management input being central to the training content. It is at this point that a new factor enters the equation: we meet expectation as well as investment.
ROE, not just ROI!
If included as an element of the final assessment of learning, Return on Expectations (ROE) inserts qualitative into the more-usual quantitative models of accounting. Kirkpatrick and his organization, initially through the efforts of his son, Jim, and daughter-in-law, Wendy, broadened their understanding of the need for the business objectives and goals to form an integral part of the learning outcomes. This meant meeting with on-the-job supervisors and considering the wider responsibility of stakeholder expectations.
Jack Philips mirrors this to some extent in his model wherein he supports isolating the effects of learning. He also broadens Kirkpatrick’s level 3 and level 4 definitions, and, in tandem with the work of Dr Patti P Philips, Jack’s wife, partner, co-author and author in her own right, Jack Philips supports quantitative assessment using advanced mathematical tools and external comparisons. However, the stakeholder chooses to measure the amount of learning, though, it is always useful for those involved to share a common language, be it a qualitative or quantitative (or mixed!) one.
Measurement – Bloom’s Taxonomy
Possibly the most authoritative source of measurement terminology is that contained in the works of Bloom et al (1956) in what is usually referred to as “Bloom’s Taxonomy.”
It is hard to envision a more relevant source of terms to describe organizational goals, and the fact that many formal schools and colleges still use this source in their curriculum design is testament to its longevity. Satisfying the head, the heart, and the hands must still remain the purpose of the trainer, and of the learner.
“Allo?
I am reminded of the government department official who, when discussing how far their customer service training had progressed, responded to my challenge to ‘prove it’ by agreeing to allow us (I was facilitating a group of about 12 senior officials on management behavior) to put a colleague’s phone on speaker as he called the department reception.
I argued that the response would be a single-word salutation, probably something like “allo?” The official was adamant that his people had passed that stage. The call connected and, “allo?” rang out across the room. The official insisted it must be the new girl, and we should try a different office. We dialled the new number: “allo?” rang out once more and, while I had not made a friend of that official (the laughter of his colleagues assured me of that!), I had made the point that it is a change in behavior not attendance at a workshop that showed whether learning had taken place. And measurement of a change in behavior had taken no more than a single phone call.
The pleasant corollary to this story was that, at the next refreshment break, the official did approach me, somewhat ruefully, and assured me that he would be looking into the training content, and the training provider, of the recent ‘customer service’ programme.
