Building a quality
program from zero
Designed and launched a scalable, data-driven quality framework for a team. From blank page to A/B-tested scoring system, peer review infrastructure, and a resident mentorship program. Three phases, full documentation, and a rubric now being used to onboard 10 people.
This program started with a single uncomfortable truth: we were giving new hires a lot of freedom without any real visibility into whether they were doing what we needed. Metrics told us how much work was getting done, not whether it was good. We had a loose buddy system, but it wasn't working for anyone. Mentors weren't recognized for their time, residents didn't have structure, and nobody was learning from the experience. I wanted to fix all three things at once: build a formal quality program that gave new hires clear standards, gave mentors a reason to invest, and gave leadership real data on what was actually happening inside the work.
What I brought to this
What I learned
I wanted to deliver quickly and the first version of the quality rubric didn't yield the results I wanted. Choosing to see that as a signal and move quickly rather than focusing on a perceived failure helped me move on. What I learned from the experience was what made the next version better. Learning to read early failure as information rather than evidence that I'd gotten it wrong was one of the harder and more important shifts I made on this project. And it's a lesson I continue to learn.
I consulted with people who had more experience than me. Their advice was reasonable. But something told me it wasn't right for this team. Instead of dismissing that feeling or simply deferring to the room, I set off to understand why I felt differently. I spent significant time researching the psychology of feedback, studying how rating scales affect human behavior, and thinking carefully about the team I was building this for. The research confirmed what I had suspected. The recommendation I made went against what other teams advised, but it was the right thing and I knew it.
Rather than just setting out to create a process, I decided to first answer a harder question: is the goal to protect the customer or protect the product? One rubric was designed for each. The binary checklist ensured the absolute basics were covered. The BARS rubric was built to give people actionable feedback and improve performance over time. Understanding that distinction changed how I presented the options to leadership. I didn't want this to feel like another administrative task. I wanted it to tie into what the company actually cared about.
How it came together
Three distinct phases, each building on what the last one taught. The goal throughout: move from subjective coaching to a transparent, scored, data-driven quality infrastructure.
Built a rubric for peer mentors reviewing new specialists across investigation, communication, documentation, and strategy. Scored 1 to 3 with written notes per ticket.
Learning: qualitative notes don't produce actionable data at scaleConsulted adjacent quality teams. Identified that existing systems were built for different contexts and that this framework would need to weight technical depth and documentation differently.
Learning: borrowed frameworks don't always transfer — context mattersDesigned two competing systems — a BARS (Behaviorally Anchored Rating Scale) and a Binary checklist — tested side by side across four specialists, including two recent hires who had been struggling to hit metrics. The goal wasn't just to pick a framework. It was to understand what was actually happening inside the work.
Learning: the data revealed onboarding gaps that metrics alone had never surfacedThe result
Leadership was presented with both options, with full transparency on what each one was designed to do. The BARS rubric was selected. It is currently being used to onboard 10 people across the team.
One specialist's excellent rating dropped from 55% under binary to 20% under BARS. Not because their work changed, but because the binary system was rewarding completion rather than quality. BARS revealed who was meeting the standard versus who was exceeding it, and more importantly, it showed exactly where support was needed and why. That's what a coaching program is supposed to do.