Building a quality
program from zero
Designed and launched a scalable, data-driven quality framework for a team — from blank page to A/B-tested scoring system, structured peer review cadence, and a mentorship program. Delivered across three phases with full documentation and calibration infrastructure.
What I brought to this
What I learned
I have high standards and a strong pull toward getting things right before releasing them. This project pushed me to test sooner than felt comfortable — and the data that came back from the first rubric was the only thing that made the second one better. I learned to treat iteration as a feature, not a failure.
The initial qualitative rubric wasn't producing usable data. Rather than spending more time refining the wrong system, I benchmarked against adjacent teams and made the call to move to A/B testing two philosophically different frameworks. Pivoting faster when the signal is clear is something I now do more deliberately.
I consulted peers and adjacent teams, but ultimately the decision about which scoring framework to pursue was grounded in the behavioral psychology behind rating scales — not just in what other teams preferred. Knowing when to trust the research over consensus is a skill I developed through this project.
How it came together
Three distinct phases, each building on what the last one taught. The goal throughout: move from subjective coaching to a transparent, scored, data-driven quality infrastructure.
Built a rubric for peer mentors reviewing new specialists across investigation, communication, documentation, and strategy — scored 1–3 with written notes per ticket.
Learning: qualitative notes don't produce actionable data at scaleConsulted adjacent quality teams. Identified that existing systems were built for different contexts — and that this framework would need to weight technical depth and documentation differently.
Learning: borrowed frameworks don't always transfer — context mattersDesigned two competing systems — a BARS (Behaviorally Anchored Rating Scale) and a Binary checklist — tested side-by-side across two cohorts, measuring scoring alignment, grader sentiment, and coaching value.
Finding: BARS surfaces performance gaps the binary system missesWhat the data showed
Both rubrics were evaluated for scoring consistency, grader experience, and coaching value. The BARS system was more demanding to apply — and more revealing.
One specialist's excellent rating dropped from 55% under binary to 20% under BARS — not because their work changed, but because the binary system was rewarding completion rather than quality. BARS revealed who was meeting the standard versus who was exceeding it. That distinction is exactly what a coaching program needs.