EVALUATION
Learning Impact Evaluation is one of the core competencies of every learning specialist. During my tenure as the Capacity Building Program manager at the Inter-American Development Bank, I led the design and implementation of a learning impact evaluation following the Kirkpatrick model. This report was developed in collaboration with Fundacion CEDDET.
​Conducting a learning evaluation is critical for many reasons. How do we know if the training filled that gap? Are the learners performing better after the learning? Was it worth investing precious resources in a particular activity? Any learning solution aims to fill a learning gap that will improve the learners' performances. In that sense, the Kirkpatrick model has proposed a solid framework to answer some of these questions. We can also incorporate the return on investment (ROI) to estimate if the learning experience has been cost-effective.
The first level of the Kirkpatrick model measures satisfaction levels (in our case, we achieved an average of 84% satisfaction rate in the 200 courses implemented). This level was evaluated utilizing the end-of-the-course surveys and short interviews with the learners.
The second level measures the learning delta from the moment learners start the course until the end of the course. We used a continuous assessment model composed of weekly activities, including forum discussions, group projects, and individual projects. Additionally, we implemented pre and post-tests at the beginning and the end of the course (in order to fulfill the program criteria, learners must submit all weekly activities and score more than eight out of ten on the final test).
Lastly, we partnered with the ROI Institute in Chile to conduct the third level of evaluation: the transfer of knowledge. This level evaluates how learners used the new knowledge acquired in their jobs or how their performance has improved due to the learning. This third leg of our evaluation is still undergoing.
A very comprehensive evaluation!
Leading this evaluation has allowed me to practice all I learned in my Learning Impact Evaluation certification by ATD. Delivery proper evaluation required a lot of hard work and a qualified team. We devoted countless hours in drafting questions for the surveys, reaching out to learners during and after the courses, incorporating new data to our databases, analyzing that information and creating user-friendly reports to communicate the information to key stakeholders. We also triangulate this information with other sources of information such as the Annual Program Meeting in Miami or when the specialist engaged with the different stakeholders to gather feedback.
However, all the effort yielded its fruit. The reports were key to engaging with the program stakeholders, including beneficiaries (learners), senior management, course instructors, and facilitators. It was also critical to improving the quality of the learning experience (formative evaluation). As a result of this evaluation, some courses dramatically changed to better respond to the beneficiaries needs.
Developing learning impact evaluation is time-consuming, but it is worth the investment. Many little things can be done to insert an evaluation mindset into the team. For example, we collected data from the edX platform to improve our MOOCs (Massive Open Online Course). We evaluated the videos that were used and concluded that some of them were too long or not instructional solid. In this sense, think about how you can use your data to make better decisions for your learning programs.
You will see it is totally worth it!