Device-Makers Like Baxter Are Lining Up To Let CMMI Evaluate Their Manufacturing Site Capabilities. Here's What To Expect If You're Appraised

Baxter Healthcare's Elizabeth Zybczynski was excited when she first heard that her company would join a US FDA pilot program to measure the capability and maturity of medical device manufacturing sites. Having worked in the aerospace and defense industries before moving to the health-care arena, Zybczynski knew that the tried-and-true Capability Maturity Model Integration (CMMI) approach could elevate quality in the device industry. CMMI Institute's Kimberly Kaplan and CMMI appraisers Becky Fitzgerald and George Zack explain the maturity model process that Baxter – and 13 other manufacturers – have gone through, from initial intake phone calls to the unveiling of final appraisal results.

[Editor's note: This is the second of a multi-part Medtech Insight feature series on the appraisal of manufacturing capability and maturity, and what it means for the medical device industry. Check out Part 1 at https://bit.ly/2jGPlL1, Part 3 at https://bit.ly/2K69U34, and Part 4 at https://bit.ly/2pnSFNR.]

When Baxter Healthcare's Elizabeth Zybczynski first heard about a new pilot program from US FDA to measure the capability and maturity of medical device manufacturing sites, she knew she'd want to be all in.

That's because the ongoing Voluntary Medical Device Manufacturing and Product Quality Pilot Program – under the auspices of the joint FDA/Medical Device Innovation Consortium (MDIC) Case for Quality – would be using a modified version of the tried-and-true Capability Maturity Model Integration (CMMI) – a framework for elevating product and process quality that she knew well.

"I've been in the medical device space and pharmaceutical space for about 15 years. But before that, I was in the aerospace and defense industry. And CMMI – well, it grew up in that industry," Zybczynski, Baxter's global director of production and process control, told Medtech Insight in a May 7 interview. "CMMI has been in those industries for nearly 30 years now, so I know the model and I know what it can do for those industries to elevate quality – and I knew it would work well in the medical device industry."

Device-makers that undergo CMMI appraisals "have more predictable results; they have lower defect rates; they have accelerated time to market; they have higher capabilities," Baxter's Elizabeth Zybczynski says.

CMMI – a process improvement model, accompanied by training and an appraisal method – is used in an array of other industries, including health care, automotive and information technology. It aims to improve an organization’s capability to deliver business objectives and distinguish trusted business partners.

Companies that undergo CMMI appraisals "have more predictable results; they have lower defect rates; they have accelerated time to market; they have higher capabilities," Zybczynski said. "So when the person inside Baxter who was the point person for this came to me and said, 'Hey, do you want to participate in this?' my answer was, 'Absolutely.' I was actually waiting for this to show up one day in the device industry."

Baxter was one of the first organizations to undergo an appraisal under FDA's pilot, which aims to elevate device and manufacturing quality at participating firms. The program uses an industry-tailored version of the CMMI model and method, developed jointly by FDA, industry and CMMI Institute.

One of the more important aspects of the CMMI model is that it attempts to drive a company's mindset from mere compliance with FDA rules to having a more robust quality system and quality mindset. (Also see "Quality On The Brain: FDA Maturity Pilot Aims To Shift Industry's Compliance Mentality To A 'Quality Mindset'" - Medtech Insight, 29 Sep, 2017.)

"Because we were one of the first firms that went through the pilot, we've offered to mentor other companies as they come into the program," said Zybczynski, who noted that Baxter enrolled two facilities in the pilot and is looking to sign up more. "And that has been the biggest obstacle – getting people at those companies to change their mindset" from one of compliance to one of quality.

The manufacturers that Baxter mentors "so want this CMMI appraisal to be an audit, and it's so not that," she said. Instead, "it's structured as a set of interview discussions and it's about being candid about how you do your work. It's not objective-evidence-based audit or inspection; rather, it's purely based on those discussions.

"From the CMMI appraisals that we've done at Baxter, and from what I've heard from my industry peers, we all agree that [the appraisals] have been amazingly accurate, even without that objective evidence," Zybczynski added.

[Editor's note: Baxter Healthcare – along with fellow device-makers Boston Scientific, Edwards Lifesciences, and others – shares its full CMMI appraisal experience in Part 3 of "Medtech's Next Top Maturity Model."]

How To Prep For A CMMI Appraisal: Don't Really

Manufacturers that undergo FDA inspections or other types of regulatory audits are used to extreme prepping beforehand – everything from roleplaying to rummaging through paper and electronic files to find relevant documents so they're ready for investigator review.

But rounding up paperwork "is very much in the wheelhouse of the compliance mindset that we're trying very hard to step away from. We do not recommend documentation to be prepared prior to the appraisal," Kimberly Kaplan, program operations manager for CMMI Institute, said in a March interview.

CMMI appraisers "don’t just knock on the door and start wandering around. There's some planned discussions that occur," consultant and CMMI appraiser Becky Fitzgerald says.

Instead, "one of the first steps for firms is to schedule an hour-long call for me, and [then] the lead appraiser, with the stakeholders from their organization – whether that's the person who will be identified as their executive sponsor or the person who will be identified as their site coordinator to help with logistics, or any other key members they believe should be aware of the appraisal activity prior to us being onsite," she said. "So, that way, we can set up some clear expectations about what's going to happen next, and all of the different steps in this process."

Additional phone calls will likely occur before the appraisal begins. "During these calls, we’ll be asking some general questions to kind of understand how your organization operates, what are the processes, what is it you’re manufacturing, and what those devices go through to get from point A to point B," Kaplan said.

At this time, the firm also identifies employees for assessment interviews with appraisers, and schedules those sit-down talks. (For more on assessment interviews, see "Employees Interviewed By Appraisers; Responses Validated; Results Given," below.)

"We generally want these calls to occur within 30 to 60 days of enrollment into the program," Kaplan said. "With some organizations, the earlier this is done, the better, so there's more time for planning and we can have a smoother onsite experience."

Becky Fitzgerald, a CMMI appraiser, and principal and cofounder of Two Harbors Consulting, confirmed in the same interview that appraisers "don’t just knock on the door and start wandering around. There's some planned discussions that occur."

After that initial intake call with CMMI, Kaplan said firms might consider having a conversation with staff about the importance of being open, honest and forthcoming with appraisers during the assessment, which she describes as "a conversational-style approach."

"We really just want to understand your organization and where there are areas for improvement," she said. "So, the more open and honest you can be with the appraisal team, the better experience it will be for everyone and the better the results – the more accurate the results – we provide will be."

CMMI Institute offers on its website intensive training sessions wherein people can learn about CMMI and how it could apply to their organizations. But Kaplan doesn't recommend taking the courses as a way to prepare for what will happen during an assessment.

At a Case for Quality forum last November, Kaplan said "even just a quick training won’t bring you up to the same level as your lead appraiser. They will be the expert. They will understand it better. That training can sometimes have the adverse effect of making someone feel overwhelmed. After all, the more you know [about CMMI], the more you realize you don’t know. Or, it can have the effect of [organizations] now thinking they’re an expert in [CMMI] and know it as well as their lead appraiser."

The First Appraisal Day Arrives. Now What?

Maturity appraisals usually take place onsite over five days, probably starting on a Monday. When an assessment begins, two CMMI appraisers, bringing in a combination of CMMI expertise and medical device knowledge, will arrive at a company's doorstep. (There could be up to four people on an appraisal team, however, depending on the scope of work.)

That two appraisers are dispatched to assignments for the FDA/MDIC pilot is somewhat unique to CMMI's Evaluation Appraisal method, chosen by the agency and industry to assess the medical device maturity model. Typically, when an Evaluation Appraisal is conducted in other industries, only a lead appraiser is called in.

But "to ensure that [appraisals] are done consistently and with high quality, we’re putting two people on these [appraisal] teams so they can keep each other accountable," Kaplan said.

"One of these people is intended to be the lead appraiser who is steeped in CMMI knowledge with years of experience, and the other person is intended to be someone who’s got a solid history and knowledge of medical devices, as well as regulatory background," she explained. Sometimes, the same person can have both.

During a facility tour, CMMI appraisers might talk to company employees.

But, Fitzgerald said, finding a CMMI lead appraiser who also has extensive medical device knowledge is like "finding a unicorn."

Also, using two appraisers helps with the overall management of the process. "During a discussion where you have someone actively listening for input from participants and then following up with questions to pull on the threads for where things go – for how work is performed – the other person can capture that information," Fitzgerald said. "It's difficult to be both the facilitator and the person capturing the information, so by having two parties, you have two sets of ears, two brains, and an additional resource for information capture."

Early on the first appraisal day, the CMMI appraisal team holds a kickoff meeting with everyone at the company that will take part in the assessment. Kaplan pointed out that some device-makers choose to make this meeting mandatory for its entire organization – even for employees who won't be involved in appraisal activities – to provide further transparency, alignment, and ownership.

The kickoff meeting "is usually an hour where we set clear expectations about why it is we're there, what we're hoping to accomplish, how that aligns with their business objectives, what they can expect for the rest of the week, and some basic guidelines," Kaplan said.

Next, the appraisal team tours the manufacturer's facility, which lasts about 30 minutes to one hour. This tour gives the team a quick view into how work is being done at the plant, and to give context to interviews the appraisal team has with company employees over the course of the week.

George Zack – also a CMMI appraiser, as well as principal and cofounder of Two Harbors Consulting, along with Fitzgerald – told Medtech Insight that appraisers might talk to employees during a facility tour.

"We were recently at a facility where there was a clean room, and it really didn't make sense for us to garb up for the clean room and interrupt the production of that particular set of activities. We also knew that we had interviews set up with those people for later in the week," he said.

"But in other cases, it would not be intrusive and it would make sense to ask [employees], 'Hey, can you tell me what you're doing here?' And if it's not going to be intrusive to their production, I've had people tell me, 'This is how this is being performed. This is what I do. My name's Joe,' and so on and so forth," Zack said.

"Or, they'll say, 'I can't talk to you right now but I can follow up with you at this particular point,'" he added. "So, the tour is focused on initially getting an assessment of the organization as to what the layout is, getting a visual on it and listening to it, but there are in some cases opportunities to turn that into an in-the-moment interview with actual line workers."

Employees Interviewed By Appraisers; Responses Validated; Results Given

Then, over the next three days, the CMMI appraisal team conducts interviews with workers that were pre-selected by the firm and lead appraiser during the pre-assessment intake calls. The interviews last for roughly one to two hours each.

"The questions tend to be grouped by functional area, and we'll have the people who are doing the work that are most likely to be familiar with the kinds of things that we're asking together in the same room, so they're hearing each other and they're also providing input," Fitzgerald explained.

There are no wrong answers and no "gotcha" questions when employees are interviewed by CMMI appraisers.

The appraisers will ask the employees to talk about their work – what they do and how do they do it. They'll also be asked whether they have the resources, skills, and tools to properly do their work, among other topics.

"And then we'll ask open-ended questions. We'll say, 'Talk to us about how executive management knows about the work that you do.' And they'll discuss the sorts of things that get communicated up, and how they get communicated up" to top executives, Fitzgerald said.

There are no wrong answers and no "gotcha" questions. Instead, "the conversations are about getting individuals to talk about the work they do every day, and to think about their work from the perspective of persistence and habitual. So, what makes that work robust and repeatable? What makes that work habitual?" she said.

And, when an employee talks about their work at a facility, it will often present topic threads for appraisers to tug.

"We have the maturity model available to us during these interviews so we can understand where we want to follow up within the practice areas that are within scope of the appraisal," Zack said.

"So, while there's an open-ended nature to it, it's not just an open-ended conversation. It's vetted specifically against a framework," he said. For example, "we're not asking cybersecurity questions of a contract sterilizer. Rather, there's something that makes sense underneath all of it."

On about the fourth day of the assessment – typically a Thursday – information gathered by the appraisal team is validated. The team does that by again talking to the people they interviewed to make sure the appraisers and the interviewees are on the same page, and that there are no misunderstandings.

When assessment results are handed out during a validation session, "there will be reactions. People aren't just sitting there, clapping, saying, 'Yes, these are our results,'" consultant and CMMI appraiser George Zack says.

At this time, "we're still in information-gathering mode, but we do offer results based on what we have heard in interviews up until that point," Zack said.

During validation sessions, "we want to know: Did we get this right? Did we miss something?" he said. "Even if it's an area in which we're saying, 'You seem to be performing strong in this particular practice area; we don't have any significant results to communicate to you' – did we miss something?

"Of course, it's rare that a firm says to us, 'No, there's all these additional results you should consider,'" Zack added. "But there will be reactions. People aren't just sitting there, clapping, saying, 'Yes, these are our results.'"

Validation sessions are also "a very intentional transference of ownership of the appraisal results," Fitzgerald explained. The assessment "starts off with the way that the work is being performed, being described by the people who are doing the work. We reflect that back to them [in validation sessions] in a categorized way after bouncing it off of the practice areas and the maturity model. And then they take ownership and say, 'Yeah, that's right. That describes us,' or they say, 'I don't think you heard me right,' or, 'Let's talk about that. I'm not sure I understand that.'"

By the end of the validation sessions, employees "feel like they've been heard, so there is a momentum shift in the organization by the people who are doing the work because their voices have been heard, their input has been captured, and they were able to directly influence its conveyance to the larger organization," she said.

Finally, the CMMI appraisal team holds a results presentation for the firm on the last day of the assessment.

This results presentation is "the next step, essentially, in taking what we did in that validation and doing the final readout on that. It really opens up that conversation again to the organization to be accountable for their particular results," consultant and CMMI appraiser Zack said.

"We pride ourselves on being able to do this effectively because it removes that typical audit contention, which is, 'These are your findings, and goodbye,'" he said. "Instead, we say, 'These are the results we came to, this is how your organization owns those results, and this is somebody from your organization speaking to those results.'"

After the close-out presentation, the CMMI Institute performs a quality review of the appraisal. CMMI's Kaplan said "it takes about 10 to 30 days to be able to review that data and make sure that it follows the plan and that it follows all of the specifications in our method definition document."

The method definition document describes a detailed methodology that a lead appraiser follows when conducting an assessment. It provides guidance on how to execute an appraisal and rules about what activities should be performed.

After The Assessment: Checkpoints, Follow-Up Appraisals

After a manufacturer undergoes its initial assessment, a firm must invite back CMMI annually for in-person appraisals. In between those yearly assessments, however, one- to two-hour over-the-phone checkpoint meetings are scheduled so CMMI can receive updates on how the firm is progressing on any issues pinpointed during the previous appraisal, among other discussion topics.

During the checkpoint process, "the organization itself is reporting back some of their own performance measures against their own quality system in terms of how they’ve expressed it historically. We’re also collecting some information as to how they’ve progressed against the CMMI appraisal model," Zack said at the November Case for Quality forum.

CMMI checkpoints encourage manufacturer engagement in the program.

Checkpoints also encourage manufacturer engagement in the program. "You can consider the appraisal essentially as that initial health check," Zack said. "You've now heard where you're healthy and where you have some things that you need to do to improve your particular health."

Fitzgerald stressed that checkpoints and annual appraisals aren't meant to scold companies that could be doing things better. Rather, the meetings are meant to help firms move ever closer to gold-star quality systems and products.

"During a checkpoint, the lead appraiser talks with the organization about what's changed over the last 90 days. And it is exactly that broad of a question," she said.

"The organization might say, 'Well, we started to do this thing. We've installed a new tool. Now we're doing this. We've made some changes. We've launched a new line. We bought a new company' – whatever the changes might have been," Fitzgerald said.

That information "is talked about and captured in a feedback report that goes to the CMMI Institute. It does not go to the FDA. That report goes to the institute and gets stored there with that organization's records. And a copy of it goes to the organization," she explained.

Fitzgerald said device-makers should think of checkpoints as ongoing talks with CMMI "to support the organization's ongoing efforts for incremental improvement."

"It's not a 'Hey, how come you didn't get all of these things done yet?' conversation; rather, it's a, 'Hey, how are things going, what's changed?' conversation," she said. "It helps the organization think about its improvement journey at a regular cadence. Some organizations already do that; some organizations don't. But it does help to build that reflective capability that helps organizations learn how to become continuously improving organizations."

[Editor's note: In the next installment of "Medtech's Next Top Maturity Model," device-makers big and small share their unique CMMI appraisal experiences under the FDA/MDIC pilot program.]

From the editors of The Gray Sheet