Full Case Study: Measurement as Design Not Afterthought
Calculating ROI for a National Onboarding Program
Can you relate?
You ask for data on your new onboarding program’s effectiveness. You’re given satisfaction scores and quiz results but these provide little insight into how individuals are actually performing. After digging deeper, you realize the program wasn’t actually designed to produce the data you need.
The Real-Life Situation
We were contracted to revamp onboarding for a high-stakes financial services role. The client wanted a best-in-class level of quality that increased overall sales volume and decreased time to productivity. The L&D team needed to be able to prove that the program we designed was worth the investment.
The Role(s) I Played
I was one of two lead LXDs on the project. Specific to focus of this case study, I led stakeholder discovery sessions and helped translate early measurement discussions into design decisions.
Approaching the Challenge
In speaking with project stakeholders, we learned that they needed to be able to spot struggling hires early but that data on sales volume and time to productivity wouldn’t be available or meaningful until several months after onboarding. As such, we had to come up with other metrics of success.
It turned out to be a good thing that we held these discussions at the beginning of the project, because the options we came up with had significant design implications.
*Note: The collaborative nature of these planning sessions was such that I can’t isolate my specific contributions.
Comprehension & Confidence
We paired untracked self-serve content with practice-based eLearning. Knowledge checks and confidence ratings gave insight into initial comprehension levels.
Knowledge Acquisition
After each major section of the program, we included a challenging, scenario-based assessment to measure knowledge acquisition.
Skill Development
We shifted workshops from instruction-based to practice-based so facilitators could observe performance and application of learning.
Behaviour Change
We created tools for coaches and managers to guide discussions around how learners were using their acquired skills on the job and enable documentation of strengths and gaps.
Respecting How People Learn
Learning is layered and thus needs to be assessed at multiple levels. We included both formative assessment which aids learning by providing feedback and enabling self-calibration, summative assessment which evaluates learning at the end of training, and ongoing observational assessment which evaluates application and promotes long-term retention.
Facilitating Application
In this case, the client didn’t just need data that measured their desired outcomes, they needed to be able to do something very specific with that data. To make this possible, we had to pay special attention to whether or not the learning experiences we were including in the program would produce the actionable data they needed, when they needed it.
Outcomes That Matter
If we had just focused on the initial desired outcome (sales volume) that was shared with us, we would have missed so many opportunities to build additional value into the program for both the business and the learners. By digging deeper, we ended up with a program that also gave both learners and their managers insight into strengths and gaps and supported ongoing performance conversations.
So What Happened?
Instead of a polished program that produced little insight, we built one that empowered both learners and leaders. While long-term metrics are still coming in, early feedback from managers and learners has been overwhelmingly positive.
Actionable Insights
- Learning can be measured—but only if you build for it. Design with visibility in mind from day one.
- When measurement feels useful instead of judgmental, it becomes a tool for growth rather than compliance