Boston Scientific, Edwards Lifesciences, Baxter Used CMMI To Measure Their Manufacturing Capability. Here's What They Said About The Experience

Quality, compliance and regulatory officials at the three large device firms – along with their peers at Steris and CVRx – open up about appraisals conducted at their facilities under a voluntary US FDA pilot program that uses Capability Maturity Model Integration (CMMI) to measure the capability and maturity of their manufacturing sites. Along with providing a step-by-step walk-through of the assessments, the officials explain how a CMMI appraisal is nothing like a typical regulatory audit. They also talk about the types of benefits they're seeing (hint: cost-savings, and better manufacturing capability and quality), and how they network via monthly conference calls linked to the pilot. Also: Baxter Healthcare explains why it made its own maturity model, and how it plans to eventually replace the homemade tool with CMMI at all its facilities.

[Editor's note: This is the third of a multi-part Medtech Insight feature series on the appraisal of manufacturing capability and maturity, and what it means for the medical device industry. Check out Part 1 at https://bit.ly/2jGPlL1, Part 2 at https://bit.ly/2K6TWRV, and Part 4 at https://bit.ly/2pnSFNR.]

Four years ago, Baxter Healthcare Corp. wanted to get a bird's-eye view of how well its quality system was functioning, so it crafted its own maturity model. Dubbed "Quality Quotient," the homegrown tool was used to gauge the maturity and effectiveness of the large device-maker's quality system in each of its manufacturing facilities.

In the context of manufacturing, "maturity" means a firm has adequately developed practices and processes to ensure that quality is pervasive throughout its organization, which in turn should lead to better and safer products, lower defect rates, accelerated time to market, and stronger manufacturing capabilities.

While Quality Quotient gave Baxter a high degree of assurance that the company's global quality system was humming along, the manufacturer eventually realized that its maturity model, despite being useful, had limitations.

"The Quality Quotient model is about the stuff that we consider – and I’m making air quotes – 'inside the quality system,'" Elizabeth Zybczynski, Baxter's global director for production and process control, told Medtech Insight in a May phone interview.

"Our whole industry runs on an idea of 'inside the quality system' and 'outside the quality system.' And that distinction, I think, is what causes our struggles as an industry," she said.

But what does it mean to say that the device industry has "struggles"? "It means our industry has higher defect rates than what you will see in, say, aerospace and defense," Zybczynski said. And that's a problem when "the challenge of our time is that [device manufacturers] make, I believe, the most important products imaginable."

One big factor that sets the aerospace and defense industries apart from devices is that they apply a more established maturity model: Capability Maturity Model Integration. Rather than reviewing only quality systems, CMMI assesses a company's manufacturing capabilities as a whole.

"CMMI has a very well-established record of bringing business benefit. Companies all over the world adopt CMMI completely outside of any compliance obligations," Zybczynski said.

FDA's CMMI pilot program "has the ability to change the face of the industry in a way that is really needed," Baxter Healthcare's Elizabeth Zybczynski says.

Administered by Pittsburgh-based CMMI Institute, CMMI is a process improvement model, accompanied by training and an appraisal method. It aims to improve an organization’s capability to deliver business objectives and distinguish trusted business partners. Aside from aerospace and defense, CMMI is used in an array of other industries, including health care, automotive and information technology.

"The CMMI model is holistic. It's about all the behaviors an organization applies when it does its work – not just the 'inside the quality system' piece," Zybczynski said. "Because CMMI is inherently a holistic approach, that, in itself, makes it superior" to Baxter's Quality Quotient model.

Zybczynski has license to make such comparisons because she worked in aerospace and defense before making the leap to the device industry roughly 15 years ago.

"CMMI has been in [aerospace and defense] for nearly 30 years now, so I know the model and I know what it can do for those industries to elevate quality – and I knew it would work well in the medical device industry," Zybczynski explained in Part 2 of "Medtech's Next Top Maturity Model."

CMMI "gives you a very detailed roadmap that you can measure progress against, because ultimately it’s the journey that matters, right? It’s the improvement process that matters, and so there’s no good or bad appraisal. It’s what you do with that information," she said. "It's a very detailed, discrete roadmap on how to improve your maturity."

Since 2014, Baxter has applied its Quality Quotient maturity model across all its facilities. But recently the firm set aside Quality Quotient at two of its plants so they could be appraised by CMMI under the ongoing Voluntary Medical Device Manufacturing and Product Quality Pilot Program from US FDA.

Launched in January under the umbrella of the joint FDA/Medical Device Innovation Consortium (MDIC) Case for Quality, the pilot aims to elevate device and manufacturing quality in an array of firms. The pilot uses an industry-tailored version of the CMMI model and method, developed jointly by FDA, industry and CMMI Institute.

As part of the pilot, results of a manufacturer's CMMI appraisal are shared with the firm itself, and a summary report is sent to FDA. The agency will use the information to help shape its regulatory, compliance and enforcement decisions.

FDA said in a recent Medical Device Safety Action Plan that it wants a permanent CMMI program stood up by 2019. (Also see "New Safety Framework Mixes Current Efforts, New Investments At US FDA " - Medtech Insight, 17 Apr, 2018.)

Baxter had already undergone two CMMI facility appraisals as of last month, and was looking to enroll more plants in the maturity model pilot, which runs through Dec. 28.

But Baxter wants to begin swapping out its homemade Quality Quotient with CMMI at all its facilities soon, regardless of whether they're part of the FDA pilot program or not.

"Once our sites are doing CMMI, then they're not doing Quality Quotient anymore. And we are looking to globally phase out Quality Quotient in favor of CMMI for the whole organization," Zybczynski said. "And that's really outside the pilot. It's a decision we made as a business, with our operational excellence group, which is not a part of quality. It's a part of operations. So again, [CMMI gives] this holistic view of the world, and it's not, 'Quality [Assurance] said to do this.'"

Zybczynski pointed out that the CMMI model caught things that hadn't been detected in the past by Quality Quotient.

"First of all, [CMMI appraises] everything that's outside the quality system, because the Quality Quotient model was never applied to that. So, right out of the gate you have a broader base of assessments," she said. "But [CMMI] also found very different things that are going to put us on a very different improvement path. It's a totally different perspective."

Manufacturers Reap Benefits From FDA

Officials from some of the bigger firms that have had facilities appraised by CMMI under FDA's maturity model pilot program say they're already reaping benefits.

In a June 20 email to Medtech Insight, CMMI Institute said 16 manufacturers have signed on to date, of which 28 facilities are to be appraised. Fourteen appraisals have already taken place. FDA's goal is for CMMI to conduct, at a minimum, 30 appraisals this year, and to enroll as many firms as possible in the pilot.

For Boston Scientific Corp., there was a "number of reasons" it had two of its facilities appraised by CMMI under the pilot, the device giant's quality assurance director said during a February webinar hosted by MDIC that served as an update to the pilot.

"First off, we really were drawn to this idea of using the maturity model and practice capabilities in lieu of just a compliance approach, to really understand specific areas that our site can work on to improve," Joe Friedrich said.

A CMMI appraisal "felt like it would be maybe a better way to help us identify areas of improvement that could have a true effect on patient outcomes, and potentially be good for our business," he added.

Friedrich said another big reason why Boston Scientific played in the pilot program was because the "regulatory benefits are significant."

"Yes, there are some economic benefits that come out of this," Edwards Lifesciences' Rob Becker says.

As an incentive to join the pilot, the agency is giving firms streamlined options for 30-day notices, site transfer changes and pre-market submissions, among other sweeteners. (Also see "Gifts For Industry: From Waived Inspections To Pre-Market Leeway, US FDA Woos Firms For Maturity Pilot" - Medtech Insight, 25 May, 2017.)

"The speed of getting [product] improvements implemented is incredibly important to us," Friedrich said. "The last thing we want to do is have an improvement that is ready to go live, but we have to wait to put that improvement into place, into action. So, the opportunity to expedite that is very attractive.

"And then, of course, the overhead associated with streamlined submissions and faster turnaround – that is also a significant business benefit to us," he added.

Friedrich did not say how much money Boston Scientific would save thanks to FDA's pilot-related regulatory ease-up, but other device-makers have been more loose-lipped. Medtronic PLC, which had a facility appraised as part of the pilot, has claimed that just one of its facilities could save $1.7m by not having to submit 30-day notices to the agency. (Also see "FDA Looks For Diverse CMMI 'Maturity' Pilot Enrollees; Device-Makers Expect Big Savings" - Medtech Insight, 16 Oct, 2017.)

And Edwards Lifesciences Corp. – another pilot participant – has said it expects to save $120,000 in manufacturing submissions costs in 2017 alone, among other savings.

"From our standpoint, being able to look at the 30-day process a little differently and manage that a little bit differently is a tremendous benefit. We're looking forward to that," Rob Becker, quality director for Edwards Lifesciences, said at a Case for Quality open forum last November.

"Yes, there are some economic benefits that come out of this, but really, when we look through the benefits for participation, we feel very, very strongly [that CMMI] will improve our compliance, [and] we think we will have higher quality, and we think the benefits for participation in the program do a tremendous amount to facilitate both of those goals. And at the same time, we do think there are some economic advantages," Becker added.

Of course, any savings that may come from FDA's incentives must be weighed against the cost for a CMMI appraisal, which can range anywhere from $30,000 to $88,000.

The price is tiered for small and large manufacturers, and is otherwise mostly dependent on the scope of the assessment, which is different for different companies. And after a device-maker undergoes its initial assessment, a firm must invite back CMMI annually for in-person appraisals.

It wasn't necessarily cost-savings, however, that motivated Baxter to take part in the CMMI pilot. After all, of the company's two sites appraised by CMMI, "one has no PMA devices, and the other might have only a couple," Zybczynski said.

"Because of the nature of the [product] portfolios at those sites, [FDA's incentives] are not going to be a huge benefit for the initial two sites," she said. "However, from talking to my industry peers, for certain sites of theirs, that is a humongous driver. So, there’s a humongous business benefit to that. It’s just not one that we will necessarily realize in great quantity at [our] two sites."

"We’re not looking for any sort of … quick financial gains," Baxter's Patrick Caines says.

Baxter will benefit from an additional carrot from the agency: Firms that enroll in the pilot won't face regularly scheduled facility inspections, and pre-approval audits are waived. FDA, however, reserves the right to conduct an inspection if it is for-cause.

"Sure, it's nice to have surveillance inspections waived," Zybczynski said, but she made clear that Baxter's "primary motivation" is operational excellence.

She noted, though, that an inspection freeze is an "important incentive" for other companies that might not yet be sold on the benefits of allowing CMMI into their facilities for an appraisal.

Edwards Lifesciences' Becker said the firm spent $74,000 on its appraisal. But, he added, it typically costs the firm roughly $140,000 to prepare for and undergo an FDA inspection, which it won't have to endure as long as it stays enrolled in the agency's CMMI pilot and upcoming program.

So, Becker said, the appraisal more than paid for itself – although the agency does not come back annually to inspect, unlike CMMI, which requires a fresh appraisal each year.

Edwards Was Dabbling In Maturity

Like Baxter, Edwards Lifesciences was already dabbling in maturity work before enrolling in the FDA pilot program.

"One of the things that Edwards is doing internally is, we've made something we call our 'Operations Maturity Index,' [which] looks at a lot of the ways we do things and benchmarks them against Toyota – some of those standards for organizational maturity," Becker said.

So-called "lean manufacturing" has its origins in efficiencies achieved at the Toyota Motor Company. Its Total Quality Management approach has been adopted by many private and public organizations, including some in health care.

To be part of FDA's pilot, "we didn't make any organizational changes," Becker said. "We have an individual on our site who is already immersed in [the Operations Maturity Index], so we rolled [the FDA maturity pilot] into their responsibilities, which was a very natural fit and created a very natural owner for it."

The Biggest Benefits: Capability, Quality

Because the device industry has not historically been exposed to the CMMI maturity model, some of Zybczynski's peers at other firms that are also enrolled in FDA's pilot have "struggled" at times to "understand its inherent value," she said.

That value, Baxter's global director for production and process control claims, is overall better capability and quality – better capability in manufacturing, and better quality in products.

But lately, "what I’ve seen, as Baxter and other companies that are in the pilot have gone through the appraisals, [is that] they’re very quickly starting to see that inherent value," Zybczynski said. "I really do believe that this [CMMI pilot and eventual program] has the ability to change the face of the industry in a way that is really needed."

In fact, it was the CMMI model's ability to help engrain quality into a company's culture that first caught Baxter's eye.

A CMMI appraisal "supports our initiative around changing our quality culture at Baxter, shifting away from a compliance-only mindset, to one where quality is a competitive advantage," said Patrick Caines, Baxter's senior director of quality systems and compliance, who joined Zybczynski in the May phone interview with Medtech Insight.

"We’ve spent a lot of time over the last year, especially, talking about how we can accelerate this change in our quality culture and what institutions or structures we can put in place that would support it," Caines said.

"We see the benefits [of CMMI] more in terms of long term, where the shift in quality culture will help us get products to market faster with better, long-term quality," he added.

"So, we’re not looking for any sort of – beyond the inspection relief for sites that do well in these appraisals – quick financial gains," Caines said. "This is more a long-term play in terms of, as we shift this quality culture, eventually we’ll see benefits, better performance of our products, faster time to market and better overall performance of the business."

CMMI Appraisals Have Nothing In Common With Inspections Or Audits

Edwards Lifesciences' Becker wants device-makers that are on the fence about whether to join FDA's pilot program to know that a CMMI appraisal is not the same as undergoing an agency inspection.

The firm's appraisal "was about as opposite from a QSIT audit, I think, as you can get. It was very, very different," the quality director said.

QSIT is FDA's Quality System Inspection Technique. It's designed to make sure that investigators look at the most important compliance issues and ask pertinent questions linked to four major quality system subsystems: management controls, corrective and preventive action (CAPA), design controls, and production and process controls.

QSIT was devised so investigators could inspect within a three- to five-day window, which almost never happens for FDA inspections conducted within the US. (Also see "US FDA Inspections & FDARA: Will New Law Light The Way For Investigators? Maybe, Experts Say" - Medtech Insight, 5 Sep, 2017.)

One of the more notable differences between an appraisal and an inspection, Becker said, is that it takes much longer for an audit to be completed.

"We have big facilities [with] a lot going on, so our QSIT audits tend to take roughly two weeks," Becker said. Meanwhile, maturity appraisals usually take place onsite over a brisk five days, typically starting on a Monday.

The traditional adversarial inspectional "war room" mindset should be set to the side by firms.

When Edwards undergoes an FDA inspection, "We have a dedicated back room. We have information requests. We're collecting documents. We're processing records. We're making sure people who go to the front room are properly prepared for what they're going to talk about. We've got a lot of vested interest in that process," Becker said.

Many firms use so-called back rooms during an inspection, where company employees pull records and fulfill investigator requests. Front rooms are used by investigators as a work area to conduct business. (Also see "From 'Back' To 'Front': FDA, Industry Experts Advise Device Manufacturers On Best Inspection 'War Room' Practices – And Don't Forget The Swedish Fish" - Medtech Insight, 9 Dec, 2015.)

But device-makers don't need such things for a CMMI appraisal. Of course, appraisers will require space to work, but the traditional adversarial inspectional "war room" mindset should be set to the side by firms.

"If I look at it just from an economic standpoint, [the appraisal] was absolutely favorable, but for us, the more important thing was [that] it was planned upfront; it had a defined duration. We had preplanned communication so we could organize resources and [the appraisers] could plan their days accordingly," Becker said.

When the appraisal occurred, "we had minimal business interruption. It was very minimally disruptive, and I can't stress that enough," he said. "Typically, … when you have a QSIT audit or that type of audit onsite, lots of activity stops. Because of the resources, because of the complexity in all those aspects, most things stop because you just have too many essential people tied up in the activity."

Becker noted that roughly 240 person hours were spent on the company's appraisal; the firm typically uses about 1,500 person hours for an FDA inspection that can stretch on for weeks.

Boston Scientific's Friedrich pointed out that a CMMI appraisal is about continual improvement, unlike a regulatory audit, which centers on whether a company is compliant with the law.

"In an appraisal we highlight improvements made over time, as well as future plans as it pertains to the discussion. But in an FDA inspection we don’t discuss improvement opportunities or future plans," Friedrich said.

And "in an appraisal we explain other systems and tools used to do our work, for example, our product life-cycle process – this framework we call 'business essentials' that is a conglomeration of practices, improvements, continuous improvement boards, et cetera," he added. "Whereas in an FDA inspection, we don’t deviate from what is documented in the quality system, policies, procedures, et cetera."

Friedrich said Boston Scientific spent about 340 person hours on its five-day appraisal.

But "compare that to [an] FDA inspection, and it’s almost a thousand hours more … when you [factor in] the back room and front room and scribe, preparation, execution, and so forth," he said. "So, there's a huge difference."

Walk-Through: CMMI Appraisal Process

There are several steps to a CMMI appraisal – before, during and after.

PRE-APPRAISAL INTAKE CALLS. Roughly 30 to 60 days after enrolling in the FDA pilot, CMMI officials – including the lead appraiser assigned to the assessment – will hold a series of phone calls with the manufacturer being appraised.

Baxter's Zybczynski estimated her firm fielded three or four intake calls for each of its two appraised facilities.

The calls between Baxter and CMMI, which were 15 to 30 minutes in length, "fell into three categories," Zybczynski said, the first being an "understanding of the content of the model" – a device-specific framework that covers these 11 practice areas:

• Estimating
• Planning
• Monitor and Control
• Configuration Management
• Managing Performance and Measurement
• Requirements Development and Maintenance
• Process Quality Assurance
• Governance
• Implementation Infrastructure
• Product Integration
• Technical Solution

"I think everybody in the program – ourselves included – wanted to map the practice areas to the quality system," although that's not a necessary activity, Zybczynski said.

"That’s everyone’s first instinct: 'Well, how do I map this to my quality system so I can bring a document?'" she said. "So, one of the things we spent a decent amount of time on is understanding how the model works and getting to the point where we stopped wanting to do that."

While CMMI Institute encourages manufacturers to understand the maturity model on a relatively high level, it also cautions firms from going too far into the weeds, noting that a deep dive into the maturity model can feel overwhelming to a layperson.

The second intake call category, Zybczynski said, revolved around Baxter's facility, including talk about products made there and the plant's various functions.

"The appraisers are looking to … take a sampling of your work, and so they need to understand what the whole picture of your work looks like," she said. "Do you have design activity? Do you have multiple product lines that run differently, or do they run the same?"

Product sampling is a big part of those early-on conversations with CMMI, Boston Scientific's Friedrich confirmed.

"For the appraisal team, the product sampling was a means to show that they had appropriate site coverage," he said. "The appraisers wanted to make sure they had a good representation of different product lines that we had at our sites, and of the different types of processes and the different types of device classifications. So, we ultimately landed on a set of products that were designated as good representatives of everything that occurs at our site."

Product sampling also helped Boston Scientific determine which workers should be interviewed by CMMI appraisers, and it "provided the frame of reference to those participants when they were answering questions by the appraisers," Friedrich said.

The third type of call, Zybczynski said, centered around logistics, including a talk about where appraisers would work (they designated a large conference room) and where they would be housed during the five-day assessment (they chose a local hotel). During these calls, Baxter also identified employees for assessment interviews with appraisers, and scheduled those sit-down talks.

"The way we executed that is, [CMMI] sent me a series of Outlook invites for the discussion sessions. I think they were all 90 minutes long, and then I just forwarded them" to the appropriate people, Zybczynski explained.

Meanwhile, Edwards Lifesciences held three hour-long intake calls with CMMI prior to its assessment.

"It took me a little time to actually understand and sort of deprogram myself from what we think would be typical, and what we'd typically do for a notified body audit" or FDA inspection, Becker said. Instead, the calls were "really more about, 'What processes do you have at the site?'

"Processes?" he asked in bewilderment. "That type of questioning was very, very different. And so, it was like, 'Well, we do this, we do this, we do this, we do this.'"

Becker said he thought to himself: "Wow, this is a much different discussion, a much different perspective on what we do as a site and how we're approaching things."

KICKOFF MEETINGS & FACILITY TOURS. When an assessment begins, two CMMI appraisers, bringing in a combination of CMMI expertise and medical device knowledge, will arrive at a company's doorstep.

There could be up to four people on an appraisal team, however, if the scope of work demands it. That's what happened at Edwards Lifesciences, which saw "two appraisal teams of two appraisers each," Becker said. That was done to make sure the appraisal was finished within a five-day window – a potentially difficult task for only two appraisers given the large size of the firm's facility.

During interviews, CMMI appraisers "are talking to the people that are actually doing the day-to-day operations, and it’s so engaging for them," Steris' Kathie Bardwell says.

Early on the first day, the appraisal team holds a kickoff meeting with everyone at the company that will take part in the assessment.

Edwards held "two all-hands meetings," Becker said. "It was really, really important for us as a site to make the activity very visible site-wide, so we had essentially all the nonproduction personnel at the site. We did an all-hands kickoff where [the lead appraiser made] a nice introduction: here's what we're doing, here's why we're doing it, here's what the week's going to look like; we're doing this a pilot program with FDA and here's why we're participating.'"

Next, the appraisal team tours the manufacturer's facility, which lasts about 30 minutes to one hour. This tour gives the team a quick view into how work is being done at the plant, and gives context to the interviews the appraisal team will have with company employees over the course of the week.

"After the initial introduction, [we gave] a quick facility tour before we got into the interviews, just so the appraisal team [could] see how work was happening in various areas," Becker said. "It's not super detailed, but I think it was a helpful connection."

EMPLOYEE INTERVIEWS. Then, over the next three days, the CMMI appraisal team conducts interviews with workers who were pre-selected by the firm and lead appraiser during the intake calls. The interviews last for roughly one to two hours each, during which time employees talk about their job and how they work, from soup to nuts.

At Edwards, the firm scheduled employee interviews with CMMI a full week ahead of their appraisal.

"For us, it was fairly simple," Becker said. For example, "For the 10:30 a.m. practice area interviews on Monday, we knew we [selected a particular] set of individuals. We forwarded them a meeting [invite], scheduled them in a conference room, and that was pretty much it. That was obviously pretty straightforward to manage."

Kathie Bardwell, VP and chief compliance officer at sterilization and decontamination technology firm Steris Corp., agreed with Becker's assessment.

During interviews, CMMI appraisers "are talking to the people that are actually doing the day-to-day operations, and it’s so engaging for them," Bardwell confirmed at a Case for Quality open forum at Cincinnati's Xavier University in May.

Steris has had two facilities appraised under the FDA pilot.

"In each case, [workers] felt really good that they could talk about what they do for a living, explore things that maybe they know they don’t do really well, or things they have a problem with, and collectively talk about what they might do to improve it," Bardwell said.

She pointed out that it was vitally important that Steris not coach its employees on what to say during their talks with appraisers.

"We said to [interviewed employees]: 'Look. This is an opportunity for us to grow within our organization. We’re committing a lot of resources and dollars to this program. We want you to be transparent. We want you to be engaging and we want you to be open-minded,'" Bardwell said.

"The validation sessions were great because they gave people an opportunity to participate. They got a chance to understand a little bit more and learn a little bit," Edwards' Becker says.

Also at the May forum, Baxter's Caines noted that "the questioning and the conversations were around assessing the maturity of our system – was there that linkage and connectedness in terms of the inputs from the various elements of the quality systems, and how do they feed into other systems to give you a realistic, comprehensive view of how your product is performing."

Unlike Edwards Lifesciences, there were only two CMMI appraisers assigned to each of Baxter's facilities. During the Baxter appraisals, worker interviews were about 90 minutes long, and there were two or three conducted each day, over roughly three days.

"It was a very low burden on our sites, because [the interviews] were only a few hours a day for us," Baxter's Zybczynski said. Meanwhile, "the appraisers are working furiously in those in-between periods to capture everything, to make sure that they have a comprehensive appraisal."

She said the interviews mostly took place in a group setting. "We had up to seven people in one [interview]. And in one of them we may have only had two people."

Monthly Maturity Call

A voluntary conference call is held monthly between device-makers that play in the maturity model pilot and officials from FDA, MDIC and CMMI Institute.

During this time, manufacturers can discuss the appraisal process and ask questions of their peers – and the officials – in an effort to make the appraisal process as transparent and effortless as possible.

On those calls, "we have an opportunity to have input into the program, or talk about what’s working, or maybe something that we’re have a problem with," Steris' Bardwell said.

"And I can tell you, if I pick up the phone and call one of the folks involved from the FDA on the Case for Quality, I never wait a full day before the call comes back," she said. "It’s just been a really engaging experience."

VALIDATION ACTIVITIES. On about the fourth day of the assessment – typically a Thursday – information gathered by the appraisal team is validated. The team does that by going back to talk to the people it interviewed to make sure the appraisers and the interviewees are on the same page, and that there are no misunderstandings.

"Validation sessions are a presentation of [initial appraiser findings] back to the team that was interviewed," Edwards Lifesciences' Becker said.

"What that does is two things: It lets the team look at [the findings] and say, 'That doesn't seem right. We must not have talked about this or this,' or, 'Here's why we don't think that seems right.' But [validation] also, for areas that were identified as weaknesses, it gave the team a little more opportunity to explore that and say, 'What do you mean there?'" he said.

Further, "We didn't know exactly what [the appraisers] were looking for, so [the validation sessions] gave the team a chance to get some buy-in to understand, 'When you say that's a weakness, what do you mean?' Those sessions were very, very positive in terms of the total outcome," Becker said.

"The validation sessions were great because they gave people an opportunity to participate," he added. "They got a chance to understand a little bit more and learn a little bit."

Becker suggested that firms that go through an appraisal keep their validation sessions small in size.

"There are different ways of doing the validation sessions. One is do all the interviews and then have one big validation session. Ours was broken into smaller pieces, but even at that, combined a little bit," he said. "Feedback from our team is to keep those smaller, partly because the bigger they are, then all of a sudden you're not getting feedback, everybody doesn't participate, the quiet people kind of close down a little bit – those kinds of dynamics happen."

"By the time we got to the report-out, we weren’t surprised, and I can’t imagine a scenario where someone would be surprised," Baxter's Zybczynski says.

Validation sessions, Baxter's Zybczynski noted, "provide a roadmap of where your strengths and weaknesses are. And in the context of the model, we very quickly identified what was the highest priority for us to work on. [The appraisers] gave us very specific content, which gave us that roadmap for improvement, and it was very accurate."

PRESENTATION OF RESULTS. On the final day of the assessment, the appraisal team holds a results presentation for the device-maker.

"The way the model’s executed, there are no surprises. The interview sessions are very candid, then you have the validation sessions," Zybczynski said. "By the time we got to the report-out, we weren’t surprised, and I can’t imagine a scenario where someone would be surprised – other than pleasantly surprised."

Added Edwards Lifesciences' Becker: "The big difference between this [maturity model] process versus a QSIT audit, is we had weaknesses that were presented as part of the appraisal [results presentation]. If [those problems] were presented in [an FDA-483] format, immediately you're on a clock, you're thinking tactically – you're responding but you're not necessarily thinking.

"And what we like about this approach … is, we have weaknesses, and we generally agree with them, but we have a little bit of time to think about things," Becker continued. "We have a little bit of time to step back and say, 'All right, what does this really mean, and what do we actually want to do about it in a way that builds and sustains going forward?'"

CHECKPOINT MEETINGS. As noted earlier, a device-maker in the pilot program must undergo yearly in-person appraisals.

But in between those annual assessments, one- to two-hour over-the-phone checkpoint meetings are scheduled every 90 days so CMMI can receive updates on how the firm is progressing on any issues pinpointed during the previous appraisal, among other discussion topics.

"It’s not another full-blown appraisal," CVRx's Al Crouse says of checkpoint meetings. Rather, they're "a check-in to talk through the areas of improvement that were identified from the appraisal."

Al Crouse – whose small Minneapolis, Minn.-based, venture-backed firm CVRx Inc. had a facility appraised under the pilot – said the quarterly meetings aren't intended to be a reassessment.

"It’s not another full-blown appraisal," Crouse said. Instead, "it's just sort of a check-in to talk through the areas of improvement that were identified from the appraisal. What are the areas that we're working on? What are the accomplishments that are being made there, and how is that going for us?"

Since CVRx's appraisal, "We've spent a couple of hours going through those types of things, and talking through the areas that we are and aren’t working on," he said.

Crouse is senior quality director for CVRx, which makes the Barostim Neo device for the treatment of heart failure and high blood pressure. His comments came at FDAnews' 15^th Annual Medical Device Quality Congress in April.

A checkpoint meeting is the perfect time for device-makers to discuss how the model works – or doesn't work – for them.

For example, "We were told [by CMMI appraisers] that we don’t do a very good job of project management," Crouse said. "Well, what they were looking for in the model is that you identify down to each individual task within a development project what you’re doing, who’s working on it and when it’s going to be due.

"Well, we have seven people total on our development team. They all sit together. They talk every day. And spending time documenting what they’re doing just wastes time," he explained. "So, we told our appraiser that we aren’t going to work on that. I don’t expect that story to get better, and … we shouldn’t be spending time on that.

"It’s those types of things you can talk through with the appraiser so they get a better picture of where you’re at and what’s going on," Crouse said. "Ultimately, we’re working up toward that full second appraisal."

[Editor's note: In the next installment of "Medtech's Next Top Maturity Model," the path to becoming a CMMI lead appraiser is detailed, and two longtime appraisers tell how they earned their stripes.]

From the editors of The Gray Sheet