Evaluating Your CLE’s Impact with Kirkpatrick

Things guaranteed to start an argumentChicago style vsNew York-style pizzaStar Trek vs. Star WarsPepsi vs. CokeIn the world of learning design, these debates are joined by one morecourse evaluation surveys 

If the thought of course evaluation surveys – those questionnaires you fill out at the end of a CLE, often to receive CLE credit — causes anger, thats understandableBut while these surveys aren’t everyone’s cup of tea, they do concern all of us indirectly.  

Course evaluation surveys often focus on how you enjoyed the presentation, whether you learned anything, and if you would recommend the presentation to others. Now think back to law school. Imagine if your professors’ primary metric of success was whether you enjoyed the class (I’m sure some of you feel that this definitely wasn’t their primary metric) or they asked whether you had learned something as you were leaving for your next class. Is that the best time to evaluate how the class has impacted your law school career? Postsession CLE surveys can feel a little like that – speak now or forever hold your peace. 

The Kirkpatrick Model of Evaluation 

The Kirkpatrick Model is one of the most well-known models for analyzing and evaluating the results of training and education programs. Its levels of training evaluation have become somewhat canon in the training field, although Kirkpatrick has some detractors, including those who feel the levels are correct but the order should be flipped.  

Kirkpatrick is divided into four levels, each measuring a different impact of training: 

(www.kirkpatrickpartners.com/Our-Philosophy/The-Kirkpatrick-Model.) 

If youve ever filled out a survey at the end of a CLE, then youre likely giving feedback designed to inform these levels of training evaluationIt’s also likely you were asked how you found the training, what changes you would make, if you learned anything, and if you’ll apply what you learned in your professional role. These questions appear to cover Levels 1  3 and are certainly more informative than going with the presenter’s gut.  

Collecting data on the impact of training is, of course, essential. However, the assumption that a post-session course evaluation survey is all that can be done to understand that impact can cause an uproar at training and development conferences. Ideally, CLE is a solution that impacts individual and organizational challengesso the measurement of success needs to be both individual and organizational.    

What do these levels mean in practice? 

Learning professionals love measuring things, and levels allow us to put things in nice categories. But what do they actually mean in practice and how do they help us make sense of what needs to change? Let’s look at the levels again: 

Level 1 “Reaction is easy – did you enjoy the course and was it relevantIt could be argued that this metric isn’t as important, but learner engagement and learning success are connected: bored learners equal lower retention.  

Attendees’ opinions are the primary data source for “reaction” and rightly so. But once you get past Level 1, the emphasis moves away from the individual and toward the training’s organizational goalswhatever they happen to be.  

Level 2 Learning can be summarized as “Did you acquire the intended knowledge and skills?” It appears to be aimed at learnershowever, when we think about job performance, who determines if you have the requisite knowledge and skills for your role? You or the organization? This is where your goals and your organization’s goals should align – if you are striving for different things, that is a problem.  

Level 3 Behavior,” or whether or not youre exhibiting intended behaviorssits firmly in the performance review arena. Hopefully, a performance review is a conversation between you and your boss around a set of mutually agreed-upon guidelines, whether in a competency model, job description, or established performance goals.  

Level 4 Results is directly aimed at targeted business outcomes (e.g., hours billed, client satisfaction, cases wonand are metrics that are typically gathered at an organizational level.  

Course Evaluation using Kirkpatrick  

CLE providers do a great job in collecting and analyzing data from course evaluation surveys, offering feedback to presenters, and striving to respond to learnersIf we measure CLE success beyond the participants’ immediate reactions to professional and institutional change, how can we collect this data?  

In larger organizations, it’s the internal training department, which can hopefully draw a line between the problem the training is trying to address and outcomes achieved. However, a significant proportion of CLE is provided by professional associations and third-party providers who may not be able to collect meaningful data beyond the immediate event.  

So, what can be done to bridge this gap between immediate reaction and longerterm changeHere are some suggestions about how Kirkpatrick can help attorneys, firms, and providers in course evaluation and identify a training’s value: 

  • Providers can reach out to attorneys and firms to establish more integrated relationships. For example, learning pathways of connected courses spaced over time typically result in deeper learning and longer-term changes in behaviors. (Level 2 – Learning) 
  • Firms and providers can follow up with learners to determine a CLE’s impact and a resulting behavior change. This can be done with a three-to-five question survey and summary of takeaways at two weeks and three months after the course.  (Even a 10% response rate would give you some solid data.) (Level 3 – Behavior) 
  • Firms and attorneys can collaborate to identify professional development goals by establishing metrics for success then seeking out CLEs that have identified those metrics in the learning objectives. Reviewing the goals after the training to see if the metrics have been met is key to understanding the impact the CLE had. (Level 3/4 – Behavior/Results) 
  • Firms and attorneys can view CLE as part of a longer-term learning process, e.g., identifying an issue to resolve, measuring it, targeting training, measuring it again, and comparing the results. (Level 4 – Results) 

These suggestions take time, something that few of us have. But even taking five minutes to write down three skills you want to improve and measurements for success will help you move CLE from a passive one-and-done experience to something that resembles a coherent individual development pathway.

Staying up to date on issues impacting the legal profession is vital to your success. Subscribe here to get the Commission’s weekly news delivered to your inbox.

Dan Davies
Dan Davies is the Education Manager at the Illinois Supreme Court Commission on Professionalism. He manages the Commission’s educational programming aimed at promoting a more professional, civil and inclusive legal profession. Prior to joining the Commission, Dan was a learning manager at the YMCA of the USA, where he designed learning solutions across topics including diversity and inclusion, achievement-gap programming and data-based decisions, and fiscal management. In his free time, Dan can be found spending time with his wife and two teenage sons, coaching travel soccer and riding a motorcycle, when he has the chance.

Share this:

Dan Davies
Dan Davies is the Education Manager at the Illinois Supreme Court Commission on Professionalism. He manages the Commission’s educational programming aimed at promoting a more professional, civil and inclusive legal profession. Prior to joining the Commission, Dan was a learning manager at the YMCA of the USA, where he designed learning solutions across topics including diversity and inclusion, achievement-gap programming and data-based decisions, and fiscal management. In his free time, Dan can be found spending time with his wife and two teenage sons, coaching travel soccer and riding a motorcycle, when he has the chance.

Leave a Reply

Your email address will not be published. Required fields are marked *