FDA Staffers, Device Firm Employees Are Training As Appraisal Team Members For CMMI Maturity Model Pilot

US FDA's ongoing Voluntary Medical Device Manufacturing and Product Quality Pilot Program, which assesses device-makers' manufacturing maturity and capability, has become so popular that both the agency and manufacturers are sending their own employees to CMMI Institute to be trained as Appraisal Team Members, or ATMs, to assist in device-related Capability Maturity Model Integration (CMMI) appraisals.

CMMI Institute is busy training and lining up applicants to be part of teams to assess the capability and maturity of medical device firms under a burgeoning US FDA pilot that looks to elevate product, manufacturing and process quality.

In fact, the agency's Voluntary Medical Device Manufacturing and Product Quality Pilot Program has become so popular that both FDA and device-makers are sending their own employees to CMMI to be trained as Appraisal Team Members, or ATMs, to assist in appraisals at device firms.

Pittsburgh-based CMMI Institute is the overseer of the Capability Maturity Model Integration (CMMI) framework, a process improvement model that is accompanied by training and an appraisal method. (See main story, "Assessing A Device-Maker's Manufacturing Capability Is Serious Business," below.)

FDA's pilot, which uses an industry-tailored version of the CMMI model, calls for an appraisal team to assess a firm over five days, typically Monday through Friday. Results of a company's CMMI appraisal are shared with the firm itself, and a summary report is sent to the agency. FDA will use the information to help shape its regulatory, compliance and enforcement decisions.

There are usually two to four people (or sometimes more) on an appraisal team, including CMMI lead appraisers and ATMs, depending on the scope of work to be done.

Because most CMMI lead appraisers aren't medical device experts, FDA insisted when it set up its pilot that at least one person on an appraisal team have knowledge of the device industry. Therefore, when assessing a device firm under the pilot, a lead appraiser with limited exposure to industry will partner with either another appraiser who has device experience (which is rare), or a designated ATM.

ATM training covers topics for those going on their first appraisal.

To be (a non-FDA, non-manufacturer) ATM in the pilot, CMMI usually selects candidates who "have a background in CMMI and have participated in at least three [non-FDA pilot] appraisals, with a preference for those with either Evaluation Appraisal experience or medical device experience," Kimberly Kaplan, program operations manager for CMMI Institute, told Medtech Insight.

Device firms are assessed using what CMMI Institute calls an Evaluation Appraisal method, which has been tailored for the FDA pilot to meet stakeholder needs for consistency and rigor.

After a short training on the pilot program by CMMI, ATMs are observed by lead appraisers during a device-related appraisal.

"An observation allows us to determine how well an individual applied the knowledge they learned during the appraisal," Kaplan explained. During the observation process, prospective ATMs are "responsible for leading exactly one interview session and writing the results for exactly one practice area, of course with the consensus of the team." (For more on employee interview sessions and the 11 practice areas covered by the pilot, check out Part 3 of "Medtech's Next Top Maturity Model.")

Next, an observation report is generated and submitted to the institute. Those who pass are then eligible to fully participate on device appraisals as ATMs.

"We are doing observations for ATMs in this program because if they are going to be one of our CMMI appraisers during an appraisal, we need to make sure they have a good understanding of the model and the method, have the right interpretation, understand the different requirements of them, and are able to execute the services in a way that doesn't hurt the fidelity of the appraisal," Kaplan said.

Embedded ATMs At Device Firms

CMMI Institute is also training what it calls embedded ATMs. These are people who work at device firms who are trained in CMMI to serve as an ATM when their own company is appraised.

Qualifications to be an embedded ATM is less stringent than for regular ATMs: embeds only need to have three years' experience in the device arena in at least two distinct areas, such as auditing, compliance, regulation, quality, operations or engineering. And embeds don't need to have a background in, or knowledge of, CMMI before they're trained.

"Embedded ATMs must go through a [three-day] CMMI introductory course, and we add one additional training day to teach prospective embedded ATMs the nuances of the MDDAP approach," Kaplan said.

MDDAP – the Medical Device Discovery Appraisal Program – is the name that CMMI Institute has given to FDA's pilot.

Embedded ATM training covers topics for those going on their first CMMI appraisal, including note-taking, interview facilitation, discussions for consensus, and understanding and interpreting the maturity model, among other lessons.

"For organizations, having an embedded ATM on staff can mean not having to hire an additional CMMI appraiser," Kaplan said. "Those firms can use their own internal team members to save costs, and to have someone who is familiar with what happened during the appraisal at their company. Embedded ATMs can help drive successful continuous improvement activities in their organization."

Embedded ATMs must also undergo the same type of observation as regular ATMs before they can be certified by CMMI.

Device-Makers Loaning Out Embedded ATMs

Kaplan said CMMI allows companies to loan their embedded ATMs to other firms.

"That's at the discretion of whether two companies agree to doing that or want to do that. We would not put someone from one organization onto another organization's appraisal without the request of both organizations to do so," Kaplan stressed.

In fact, an embed-share is happening for the first time this week during a pilot appraisal. This is being done because a particular firm's embed needs to undergo a CMMI observation to become certified by the institute before the appraisal at their own company begins.

"Firm A is lending their ATM to Firm B," Kaplan said, not identifying the names of the companies. "Firm A gets to have its embedded ATM be observed prior to its appraisal, so that individual could potentially function as a full ATM by the time their organization's appraisal comes around."

But what does Firm B get out of the deal?

"At this point, Firm B doesn't really get much out of it," she said. "But theoretically, if that embedded ATM from Firm A was already observed, it would save Firm B the cost of an appraiser," she said.

FDA has sent nine employees through ATM training.

Kaplan pointed out that those two firms will do the same thing during an appraisal scheduled for October.

"They are particularly friendly," she said.

"And two other firms have thought about doing this as well – Firm C and Firm D. They're also friendly with one another. That would happen in December if they decide to move forward," Kaplan added.

To use Firm A's embed, Firm B drew up a non-disclosure agreement that was signed by both companies.

"An NDA is all they agreed needed to be done between them to move forward with the process," Kaplan said. "But Firm C and Firm D could do things differently if they so choose. We aren't here to mandate what they need to do between themselves to make that happen."

FDA Employees Also Receiving ATM Training

And even FDA is getting in on the act, sending selected employees to CMMI for ATM training.

"We’ve had nine FDA employees go through our training that’s required to become an Appraisal Team Member in this pilot. And of those FDA employees, I believe some of them were investigators," CMMI's Kaplan said.

"We wanted to get a variety of FDA people involved in understanding what it is we’re doing that’s different in this pilot," she said. "The feedback was very positive, and I believe many of them left thinking about how they could leverage [the maturity model] in their own work."

At a Sept. 14 FDA/MDIC Case for Quality open forum, Cisco Vicenty, a program manager in the Office of Compliance within FDA's Center for Devices and Radiological Health (CDRH), said it's important for agency employees to understand the model if FDA is going to continue the program in 2019 – and beyond.

"The more we get [our] people accustomed to the terminology, the language, the approach in methodology that we're using for CMMI, the better it is to integrate and make some of the internal changes we're trying to go through" at FDA, Vicenty said. "That's just from a standpoint of our own internal improvements – it's of value.

"But the additional piece of this is, we've been trying to figure out a way to enable this to be an additional dialogue or an additional resource moving forward," he continued. "As we have been evaluating the need and scalability of the program, we do need to make sure we have people with the right skillsets, and more of them across the board."

Vicenty's hope is that FDA's ATMs gain "experience by participating in some of the appraisals if the organization is comfortable," but he noted that it's "not a requirement at the moment."

"It does give additional exposure to our team members, and it lets them see directly the impact of the appraisal method. So, it does open up the dynamic and shift what [FDA's ATMs] can see or expect on this type of engagement internally," said Vicenty, who is also a leader for the Case for Quality, the umbrella initiative overseeing the pilot.

But he stressed: "These appraisals are not an enforcement tool or are intended to be that way, so that is not the expectation of the FDA participants who are on the [appraisal] team."

Manufacturers that allow an FDA ATM to be part of its appraisal could see a subsidy from the agency.

"We do [subsidize] a little bit of the cost of the appraisal based on the FDA engagement," Vicenty said. "For the smaller manufacturers, from a resource standpoint, it does provide an additional option," Vicenty added.

CMMI appraisals can be pricey – especially for small firms, which can expect to shell out $27,000 to $64,000 for an assessment. Large companies pay an appraisal fee of $34,000 to $83,000.

From the editors of The Gray Sheet