Quality On The Brain: FDA Maturity Pilot Aims To Shift Industry's Compliance Mentality To A 'Quality Mindset'
Executive Summary
An upcoming voluntary pilot program from US FDA will be the perfect vehicle to help drive a compliance-to-quality culture change at medical device firms, developers of the program say. To be stood up early next year, the pilot will see manufacturers use third-party assessors to measure the maturity of their quality systems and manufacturing processes. In a survey, early users of the Capability Maturity Model Integration (CMMI) appraisal process said it helped them identify ways to increase product quality.
Becky Fitzgerald has quality on her mind, and she wishes that more medical device manufacturers did, too.
In the industry, "there's a bit of a gap, if you will, between the compliance perspective" – device-makers wanting to make sure they meet US FDA regulations – "and the quality mindset," Fitzgerald said. "There is a great need to shift the way work is performed at device firms from a compliance mindset to a quality mindset."
The longtime Michigan-based consultant believes an upcoming voluntary pilot program from FDA in conjunction with the Medical Device Innovation Consortium (MDIC) will be the perfect vehicle to help drive a compliance-to-quality culture change at firms. To be stood up in early 2018, the pilot will see manufacturers use third-party Capability Maturity Model Integration (CMMI) assessors to measure the maturity of their quality systems and manufacturing processes.
How a company views quality "boils down to behavior. It's a cultural thing. It's the way people do their work. And if they're doing their work effectively and efficiently, that results in the best possible outcome. A strong quality system equals strong output. So, the better the quality system, the better the outcome," said Fitzgerald, principal and cofounder of Two Harbors Consulting.
The soon-to-be CMMI pilot, she says, "represents a shift from neat baseline compliance requirements to build better quality systems and better quality devices."
After all, Fitzgerald would know. She was the CMMI lead appraiser who helped design and conduct proof-of-concept activities and pre-pilot maturity appraisals for the joint FDA/MDIC venture.
Device-makers that enroll in the maturity model pilot "will realize performance benefits. They will realize better efficiencies and effectiveness in the way they do their work," consultant and CMMI expert Becky Fitzgerald says.
CMMI is a framework that allows appraisers to score where manufacturers fall on a five-tiered maturity scale, with "1" denoting beginner efforts by a firm, and "5" signifying that a company is self-correcting, effectively manages quality and is continuously improving. Developed in the 1990s by Carnegie Mellon University's Software Engineering Institute in Pittsburgh, and now administered by the CMMI Institute, CMMI is based on the Capability Maturity Model.
"Maturity" in the context of manufacturing means that companies have adequately developed practices and processes to ensure that quality is pervasive throughout their organization.
Under the pilot, results of a manufacturer's CMMI assessment will be shared with FDA. The agency will then use the information to help shape its regulatory, compliance and enforcement decisions.
And device-makers will benefit, too, Fitzgerald said. Manufacturers "will realize performance benefits. They will realize better efficiencies and effectiveness in the way they do their work," she said. The CMMI "framework has been used for years [in other industries] with evident outcomes. So, there is an expectation that firms will receive benefits by taking a look at the way they do their work from something other than a purely compliance perspective."
Companies that play in the pilot will also "look at how rigorous and disciplined their approach is to quality," Fitzgerald said. "The more rigorous they are and the more disciplined they are in their approach, then the better, more effective and more efficient their outcomes will be."
Fitzgerald's comments to Medtech Insight come as FDA gears up for an Oct. 10 public workshop, during which the agency will lay bare a maturity model appraisal framework and a plan for implementing the pilot. (Also see "US FDA Maturity Model Pilot Program Gets October Meeting Date" - Medtech Insight, 24 Jul, 2017.)
FDA will formally announce the pilot in December and begin recruiting manufacturer volunteers soon after. Sean Boyd, deputy director of the Office of Compliance within the agency's device center, said at a Case for Quality workshop in May that FDA's goal is to collect data from firms via CMMI appraisals throughout 2018 and to have a fully realized program operational by 2020. (Also see "Gifts For Industry: From Waived Inspections To Pre-Market Leeway, US FDA Woos Firms For Maturity Pilot" - Medtech Insight, 25 May, 2017.)
"We don’t know for certain how many manufacturers are going to participate or who’s going to play. But because we’ve been able to bring together FDA and industry, and experts from CMMI together to work on this in a very thorough and rigorous and thoughtful way before the pilot program is even announced, we’re building the credibility of the program before we even launch it," MDIC Program Director Stephanie Christopher said.
The pilot "is a way for all of us to come together to incentivize quality and to make sure that we get high-quality products into the hands of patients who need them, and to create a system where, rather than focusing simply on compliance, we focus on quality," Christopher told Medtech Insight. "And I think that’s a message that resonates with the medical device industry members that are participating."
But Fitzgerald acknowledged that moving beyond simple compliance with FDA rules to having a more robust quality system and quality mindset won't happen overnight for manufacturers.
"All of that takes time," she said. "But industry, through the proof-of-concept and early pilot activities – through that very first dip-your-toe-in set of activities – already realizes the benefits from participation. The CMMI program will be one of those things where momentum for it builds by evident outcome."
CMMI's Five Maturity Levels
- Initial (processes are unpredictable and not controlled)
- Managed (processes are reactive)
- Defined (processes are proactive)
- Quantitatively Managed (processes are measured and controlled)
- Optimizing (continuous process improvement)
Survey Turns Up Mostly Positive Results
MDIC conducted three separate pilots in late 2016 with one small, one medium and one large device firm to gauge whether CMMI was the best maturity model for the device industry to use.
Aside from their size, the three companies were chosen based on the variety of products in their portfolios. "We wanted to make sure that the devices they made were diverse, including devices with software components, hardware components – you name it. We really looked for even the materials and components to be diverse amongst those three firms," said Vizma Carver, founder and CEO of Carver Global Health Group in Ashburn, Va., and co-leader of those early-on pilots and proof-of-concept activities with Fitzgerald.
A survey of 26 employees from the three firms, conducted late last year, support Fitzgerald's notion that the device-makers saw organizational benefits following their CMMI appraisals.
A whopping 96% of survey respondents – people who were involved in the pilots at their firms in some way, shape or form – said the appraisal process identified "areas or processes that could improve how work is performed to increase product quality." (See Figure 1, Q1.)
Figure 1
CMMI Survey: Pre-Pilot Feedback
Source: Survey conducted by MDIC of 26 employees at three device firms enrolled in separate pre-pilot CMMI pilots
Further, all of the survey-takers said "CMMI appraisal interview areas" didn't conflict with "regulatory compliance assessment areas," and that the appraisals accurately identified how leadership at their firms viewed the concept of quality. (See Q3 & Q4.)
Overall, 35% said the maturity appraisal was a very valuable experience, while 65% deemed it to be somewhat useful. (See Q5.)
Carver told Medtech Insight that she isn't concerned that more participants didn't find the experience to be of high value. She believes – as time goes on, and more appraisals are conducted and changes are seen – that those participants will eventually see more benefit in the process.
"Some will look at the CMMI model for the first time and say, 'Oh, they're talking about requirements; they're talking about risk management – well, that's a no-brainer. There wasn't anything 'a-ha' about that,'" she said, surmising that's why more survey-takers didn't grade the process higher.
But, Carver said, the CMMI process can be a slow-burn that causes big changes to happen over time. "That's why those numbers should eventually shift from 'medium' to 'high' in future surveys. And that will be an indication of whether the program is working."
She pointed out that, in the end, the maturity pilot is all about patient safety.
"The purpose of the [pilot] is for organizations that embrace quality to be recognized for doing that," Carver said. "And, most importantly, it ultimately becomes a benefit to the patient. I want readers of Medtech Insight to know that there has been no meeting [about the pilot] that I have been a part of where the patient and how this benefits the patient hasn't been the central part of the conversation. This is all about us honing in on the patients and everyone saying, 'What can we do to do it better?'"
FDA Inspections & Quality
The MDIC survey found that less than half of respondents – 48% – believe that FDA inspections pinpoint areas or processes that could be improved to increase the quality of their products. (See Figure 1, Q2.)
"That question was asked because we wanted to identify activities that help the industry move toward a continuous process improvement," consultant Vizma Carver said.
"We wanted to know: Do FDA inspections help with quality, or is the nature of the audit one that actually puts the organizations on defense, and thus really doesn't help?" she said. "Inspections are more about a presentation. But CMMI appraisals aren't about presentations – rather, they're about a facilitated dialogue amongst people who are actually doing the work so they can identify the areas that have attributes for success – but also discover areas where they can improve.
"So, that's the difference. That's what we're trying to identify: When firms have an FDA investigator come in, do they get that CMMI-type of experience of working toward better quality? Most people who took the survey don't think so."
10 Hours
It took an average of 10 hours per employee to participate in the CMMI appraisal, MDIC's survey of the three piloted firms shows. (See Figure 1, Q6.)
Because that's an average, consultant Carver said she would expect some employees to spend perhaps only a few hours participating in an appraisal, while others could spend up to two full days.
"For example, your project manager is going to sit through most of your appraisal sessions. So, that individual, who is kind of scheduling everything, etcetera, may have more time invested in it," Carver said. "Meanwhile, your contracts manager might attend one or two brief sessions – so that person will use a smaller number of hours."
Don't Call FDA's Pilot Incentives 'Gifts'
FDA is offering incentives to manufacturers that participate in the pilot program. The agency says it will put off regularly scheduled facility audits and waive preapproval inspections, and will allow firms more leeway for 30-day notices and pre-market submissions.
Further, FDA will engage with troubled companies enrolled in the pilot rather than quickly dashing off a warning letter or using some other type of enforcement. Other incentives are also being considered.
In a May story on the pilot, Medtech Insight labeled the incentives as "gifts for industry," but consultant Fitzgerald doesn't necessarily see it that way.
Manufacturers that volunteer to open their doors to FDA under the pilot are more transparent, she said, which means the agency has more confidence to give those firms access to such generous benefits.
"If you’re willing to show how your quality system is operating and how your company is operating, then FDA is saying, 'Look, that’s more information for us, and that will help us identify where we need to focus our attention and our efforts. In return, we can support you in going faster through the [pre-market or enforcement] process because we know more about you, frankly,'" Fitzgerald said.
"So, keep in mind that it's not 'gifting' – those firms will work hard for those incentives from FDA, which they'll receive because of their transparency and the maturity of their operations."
From the editors of The Gray Sheet