Measuring Effectiveness and Analytics in Compliance Training at Pharma Company
Compliance Training Business Challenge
Top Pharma company needed to ensure that its Global Health Care Compliance (HCC) Annual Training Program met the updated guidelines from the U.S. Department of Justice (DOJ) on corporate compliance program effectiveness. The challenge was twofold:
- Traditional learning measurement models, such as ROI, did not sufficiently demonstrate knowledge transfer.
- The company lacked statistically defensible evidence that employees had truly mastered compliance concepts beyond just completing the compliance training.
A 2019 pressure test of the training program revealed that they could not confidently determine whether employees were capable of applying compliance principles in practice, putting the company at risk of failing to meet regulatory expectations.
"51% reliability in 2019 to 85% in 2020. By leveraging measurement effectiveness we are now able to confidently say we meet DOJ guidelines."
CLIENT TESTIMONY
Training Solution
To address these concerns, the company’s Health Care Compliance Global Policy, Training, and Communications team developed the Measuring Effectiveness Initiative, a rigorous, six-step methodology to evaluate and enhance learning transfer. This initiative included:
- Partnering with psychometric experts to create a defensible learning measurement model.
- Implementing xAPI tracking technology to collect and analyze detailed learner interaction data.
- Using psychometric analysis (including Classic Test Theory and Item Response Theory) to evaluate the reliability and validity of final assessments.
- Developing a Toolkit for Building Effective Assessments, which guided the creation of scenario-based, statistically defensible test questions.
- Establishing a change management strategy to address cultural resistance and ensure stakeholders understood the importance of the new learning measurement approach.
Example questions:
- Have employees truly learned during training?
- What specific knowledge and skills were acquired?
- Was the training design effective?
- Were the testing methods employed effective?
- Was any misconduct in learning detected and prevented?
Learning Program Results
The Measuring Effectiveness Initiative significantly improved the defensibility and impact of the Pharma company’s compliance training:
- Improved Assessment Reliability: The psychometric analysis of the final assessment improved from 51% reliability in 2019 (statistically random) to 85% in 2020, aligning with academic standards.
- Defensible Learning Transfer: The initiative provided quantifiable evidence that employees not only completed training but also demonstrated compliance capability in practice.
- Behavioral Insights: Analysis of learner behavior revealed key factors that influenced success, such as structured progression through course content and optimal pacing.
- Scalability & Continuous Improvement: The model became an ongoing tool for refining compliance training, with a repeatable methodology that informs risk mitigation and future learning strategies.
By pioneering a data-driven, psychometric approach to compliance training, this Pharma company established itself as a leader in corporate learning measurement, ensuring its workforce is both knowledgeable and capable of maintaining regulatory compliance
.
Brandon Hall Gold 2024
Category: Best Use of Learning Measurement
Medal: Gold
Year: 2024
"The initiative provided quantifiable evidence that employees not only completed training but also demonstrated compliance capability in practice."
We Are Your Partner in Health Care Compliance Training
Meaningful Collaboration. Measurable Impact.
At LHT Learning, our healthcare compliance training specialists design impactful, sustainable learning programs that drive long-term success. From DOJ guidance and ADA compliance to the challenges of global implementation, we turn complex regulations into clear, engaging training that sticks and is statistically defensible using measurement effectiveness.
Ready to elevate your training? Let’s create a solution that delivers measurable business results.
