Assessing A Device-Maker's Manufacturing Capability Is Serious Business For These 2 Longtime CMMI Appraisers. Here’s Their Story

From training classes to onsite observations, two lead appraisers affiliated with Pittsburgh's CMMI Institute describe the high hurdles they jumped to be part of a flourishing US FDA pilot program that aims to elevate product, manufacturing and process quality at device firms. The pilot uses an industry-tailored version of the tried-and-true Capability Maturity Model Integration (CMMI) framework that's been around for decades. Beth Layman, an appraiser who has assessed more than a hundred companies in a variety of industries, says appraisers can't just be book-smart – rather, they must also possess a select set of personal skills, including flexibility and open-mindedness. Her fellow appraiser Thayne Hill, who "was there in the early days" of the Capability Maturity Model, says device-makers are often leery of appraisers because of past negative interactions with regulators, which can make his job challenging.

[Editor's note: This is the fourth of a multi-part Medtech Insight feature series on the appraisal of manufacturing capability and maturity, and what it means for the medical device industry. Check out Part 1 at https://bit.ly/2jGPlL1, Part 2 at https://bit.ly/2K6TWRV, and Part 3 at https://bit.ly/2K69U34.]

A few decades ago, as a civilian electronic engineer working in the US Air Force, Thayne Hill became part of a project that would turn out to be so game-changing for quality in an array of industries that it would help shape his career for roughly the next quarter century.

In 1991, the Software Engineering Institute at Carnegie Mellon University in Pittsburgh – set up and funded by the Air Force in the mid-1980s – first released its Capability Maturity Model. The CMM tool was used to assess the capability of US government contractors tied to military software projects. By using the CMM, the military could ensure that only vendors that showed they could provide the best quality products were offered contracts.

"The software CMM was built and funded out of the Air Force, and I just so happened to be working in the Air Force at that time," Hill told Medtech Insight.

"And the Hill Air Force Base in Utah – where I resided – was one of the prototype beds for the testing of this Capability Maturity Model," he said. "So, I became engrained into the Capability Maturity Model about 22 or 23 years ago."

"I was there in the early days," CMMI lead appraiser Thayne Hill says.

After the CMM prototype appraisal at the Utah base proved successful, the Air Force asked him to travel the country to spread the CMM model's philosophy to other bases.

"Because I was the Air Force representative and I was flying around everywhere and meeting everyone, I became known to the Software Engineering Institute. That's when I became a lead appraiser for the Capability Maturity Model, and I've held that position both as an appraiser and as an instructor since 1997," Hill said. "I was there in the early days."

A few years later came the evolution of the CMM: The very first version of the Capability Maturity Model Integration (CMMI) framework was introduced by the Software Engineering Institute in 2002, with updates in 2006 and 2010. Version 2.0 of CMMI was released earlier this year.

CMMI is a process improvement model, accompanied by training and an appraisal method. Used in an assortment of industries, including automotive, information technology, defense and aerospace, CMMI aims to improve an organization’s capability to deliver business objectives and distinguish trusted business partners.

After the CMMI model became popular globally in the 2000s, the CMMI Institute was founded in 2012 (and lifted out of Carnegie Mellon that same year). The institute provides training and certification to CMMI instructors and lead appraisers like Hill, who then make up a vast partner distribution network that provides services to organizations looking to evaluate and improve their capabilities.

"Over my many years as a lead appraiser, I've performed CMMI instruction and appraisal activities for many DOD [US Department of Defense] sites – Army and the Navy – including [aviation agency] the FAA," Hill said.

Hill has also appraised the capability and maturity of companies involved in telecommunication, insurance, construction, carpet and flooring, and aircraft manufacturing, just to name a few of the disparate industries that use CMMI. But lately he's been focused on the medical device arena as a lead appraiser for an ongoing US FDA pilot program that uses an industry-tailored version of the CMMI model.

Called the Voluntary Medical Device Manufacturing and Product Quality Pilot Program, the initiative aims to elevate product, manufacturing and process quality at device firms. (The CMMI Institute branded the pilot a bit differently, naming it the Medical Device Discovery Appraisal Program, or MDDAP.)

Launched in January 2018 and ending this coming December, the pilot falls under the umbrella of the joint FDA/Medical Device Innovation Consortium (MDIC) Case for Quality. Because of the initiative's success – it's very popular with device-makers – FDA will turn the pilot into a full-fledged program in 2019.

"The best practices of the CMMI model transcend all industries," Hill says.

As part of the pilot, results of a manufacturer's CMMI appraisal are shared with the firm itself, and a summary report is sent to FDA. The agency will use the information to help shape its regulatory, compliance and enforcement decisions.

In return, firms receive a bevy of benefits from the agency, including streamlined and accelerated options for 30-day notices, site transfer changes and pre-market submissions. Pilot enrollees also won't face regularly scheduled facility inspections, and pre-approval audits are waived. FDA, however, reserves the right to conduct an inspection if it is for-cause.

"My expertise has always been software and embedded software," Hill said. "So, if you're trying to draw the thread of how a structured DOD kind of a guy could get into the medical device industry, well, here's the thing: CMMI appraisers are placed in different environments all the time, so whether it be the FAA, or elsewhere, it doesn't really matter. That's because the best practices of the CMMI model transcend all industries.

"CMMI even transcends my chosen profession of software."

Nevertheless, when setting up its pilot, FDA insisted that at least one person on a CMMI appraisal team have knowledge of the medical device industry. Therefore, when Hill assesses a device firm under the pilot, he partners with either another appraiser who has device experience, or a designated Appraisal Team Member. A so-called ATM is typically someone from the device industry who has been trained by CMMI specifically for assisting with device-related appraisals. (For more on ATMs, see sidebar story, "FDA Staffers, Device Firm Employees Are Training As Appraisal Team Members For CMMI Maturity Model Pilot," below.)

"Sure, I have very little medical device experience in the traditional sense, but as a lead appraiser in this pilot, I'm always hooked up with somebody who is considered to be a medical device representative," Hill said. "So, the two of us are always together. What I lack in knowledge of the medical device industry, I’m learning very quickly. The other individual is always there to assist me."

CMMI also puts its lead appraisers through a MDDAP training course so they can learn more about the FDA pilot, the medtech industry and how best to assess a device firm. (See "But Wait: There's More To Learn For Devices," below.)

The institute "has developed very specific training materials covering a multitude of topics to take a regular lead appraiser who may or may not have experience in the medical device industry, and then convert them so they can be successful inside of a medical device site," Hill said.

130 Appraisals – And Counting

Of the roughly 130 CMM/CMMI appraisals Hill says he's completed over the course of his career, eight of them were for FDA's maturity model pilot.

In fact, Hill participated on the first assessment of a device firm under the pilot.

"I happened to be available for that very first device appraisal – that was my official entry into the program. I've remained with it since," Hill said.

In a Sept. 19 email to Medtech Insight, CMMI Institute said 17 device-makers have signed on to the pilot to date, of which 34 facilities are to be appraised. Seventeen appraisals have already taken place.

FDA's goal is for CMMI to conduct, at a minimum, 30 appraisals by the end of 2018, and the pilot is on track to meet (and possibly exceed) that number, the institute says.

A typical CMMI appraisal under FDA's pilot lasts a full week and begins on a Monday. The assessment includes a thorough review of a company's manufacturing processes and how it functions, as well as in-depth interviews with workers pre-selected by the firm and lead appraiser.

The interviews last for about one to two hours each over three days, during which time employees talk about their job and how they work, from soup to nuts. Then some validation activities occur, and a final report is presented to the firm by appraisers, usually on a Friday. (Check out Part 2 of "Medtech's Next Top Maturity Model" for a more detailed description of a CMMI appraisal, including activities that occur before and after an assessment.)

"It's been nine months since that first official appraisal I was on for the FDA pilot, and what's happened over that time is that everything has become much more efficient and effective. And what I mean by that is, on the first appraisal we didn't have all of the baseline templates, and all of the tools and mechanisms fully solidified. We were testing out various things," Hill said.

"But now we have SharePoint sites that hold all our templates, and the templates are constantly evolving as we try various things and improve things," he added. "We have a huge lessons-learned database that we're using, and the appraisers involved in the device pilot talk on a conference call about every two weeks. We share our experiences and learn from each other."

Kimberly Kaplan, program operations manager for CMMI Institute, acknowledged that FDA's pilot has evolved and stressed that appraisers are always kept up to date.

"Now that we're at a place where things are running more business-as-usual, we have a much more well-defined set of training that's required for appraisers," Kaplan said in an interview. "And we do have biweekly meetings with MDDAP appraisers to update them on the latest nuances or changes or adjustments to the tools they use."

During those calls, "we also review lessons learned from the most recent set of appraisals, so everybody's on the same page as we continue to iterate and evolve in the pilot," she said.

Appraiser Training: Not For The Faint Of Heart

The road to becoming a general CMMI lead appraiser includes some rather high hurdles for applicants to jump. First, they must complete a three-day intro class and a four-day advanced course centered on learning about the maturity model and how it's applied.

Applicants must also be sponsored by a CMMI partner – an organization that is "licensed to deliver CMMI Institute's leading-edge performance-improvement services throughout the global business community," the institute explains on its website.

In other words, a partner is sort of like a CMMI franchisee. The institute boasts more than 500 partners worldwide.

"Once we've confirmed that an applicant is indeed sponsored by one of our partners, they then can submit an application to become a lead appraiser," Kaplan said.

There are 380 lead appraisers qualified to the CMMI development model.

In that application, "you have to demonstrate that you have some teaching experience and/or skills, and be able to facilitate discussion and present information in a clear, concise manner," Kaplan said. "You also have to demonstrate that you have at least 10 years of experience in the view of CMMI that you're applying to be an appraiser for."

The institute offers four different "views" of the CMMI model: development, services, acquisition and people. The development-centered model was chosen for FDA's device pilot because it fit the industry best.

Globally there are 380 lead appraisers who are qualified to the CMMI development model.

Applicants who elect to follow the development view of the model must have 10 years of development experience in project management, and engineering experience in systems or software. "And at least two of those 10 years must include management of technical personnel," Kaplan noted.

Applicants must also hold a bachelor's degree (or have equivalent experience) in a related area and be fluent in English. Further, they must participate in at least two benchmark appraisals.

After submitting and passing a scenario-based written exercise, applicants are then qualified to advance to a five-day lead appraiser training course, after which they must pass a two-hour qualification exam made up of 60 multiple-choice questions.

"And after the exam, they still need to be observed," Kaplan said. "They have to lead an appraisal with an institute observer present who monitors what they're doing and evaluates their performance. The observer determines whether or not that applicant passed their observation."

Once the applicant clears the observation hurdle, they're officially certified as a CMMI lead appraiser.

But Wait: There's More To Learn For Devices

For a lead appraiser who wants to play in FDA's maturity pilot, he or she must possess particular credentials and undergo additional training.

"For the purpose of the pilot, there's an application process and a separate certification pathway to ensure additional quality requirements before an appraiser can be considered to work on behalf of the institute in this pilot," CMMI's Kaplan said.

She went on: "Right now the requirements are that the lead appraiser must have led at least 10 appraisals with no significant issues arising from the quality reviews. In other words, no quality issues over the last five years. And there is a preference for individuals with experience in the evaluation-style appraisal."

Device firms are assessed using what CMMI Institute calls an Evaluation Appraisal method, which has been tailored for the FDA pilot to meet stakeholder needs for consistency and rigor.

There are several CMMI appraisal methods to select from, but an Evaluation Appraisal "supports a higher degree of tailoring and fits the intent of the [device] program the best, so it was chosen by the FDA, industry and CMMI to use," lead appraiser Becky Fitzgerald explained in Part 1 of "Medtech's Next Top Maturity Model."

"It's vital that appraisers be quick studies," CMMI lead appraiser Beth Layman says.

Kaplan said an interview is scheduled with the applicant after the institute confirms that the appraiser indeed has the required experience to work in the pilot.

"During this interview we provide additional information about the pilot and present different scenarios to the appraiser to see how they respond, to see if they meet the soft skills that are needed for evaluation-style appraisals," she said.

If an applicant aces the interview, they're enrolled in a half-day MDDAP training course, during which they "learn about the history of the pilot, the intention of the pilot, and the nuances of how what we're doing in this pilot is different from a typical appraisal, because there are additional sets of requirements and other stakeholders [such as FDA and MDIC] involved," Kaplan said.

This training "is for folks who are experienced in appraisals, and goes into the specific requirements for scoping and planning an MDDAP appraisal, the hand-off with the institute, FDA-specific reporting requirements, use of the tools and templates in detail, ATM facilitation, et cetera," Kaplan added.

After completing the course, which can be taken digitally, the appraiser must then participate in a device-related appraisal as a team member and be observed.

"But it's a much lighter-weight observation, because they already know how to be a lead appraiser," Kaplan stressed. "This is an observation to see how well they understood and implemented the nuances that were defined in the training. As long as they pass that observation checklist, they will then be certified to be able to lead MDDAP appraisals, as well as participate as team members."

Do You Have What It Takes To Be A CMMI Appraiser?

Beth Layman, a CMMI lead appraiser for the past 15 years, told Medtech Insight that an appraiser must possess a select set of personal skills, including being flexible and open-minded.

Appraisers can't "look at the CMMI model in a rigid way," she said. "Because the model is not really prescriptive. The model is a set of requirements, and different companies implement them differently. So, you have to be flexible in your view of how the model can be implemented."

Over her career, Layman has performed more than a hundred appraisals in a wide array of industries, including automotive, defense, software, finance, aerospace and health care. She has conducted only one appraisal under FDA's pilot, but is gearing up to do more.

"It's also vital that appraisers be quick studies," she noted. "It's very important to be able to come in and absorb the context of the organization and pick up on its terminology and lingo, and retain information so you can make good judgements for the company about where it's at, capability-wise."

Layman, who is a process improvement consultant for her own Melbourne, Fla.-based firm Layman & Layman, said appraisers also need top-notch communication and people skills.

"Appraisers have to deal with a lot of people, talk to a lot of people and try to get people to share information with them," she explained. "That's why those types of skills are so important."

"I have never seen an industry that is so skittish when someone from the outside walks through the front door," Hill says.

CMMI lead appraiser Hill agreed that communication is key, but so too is listening to what people have to say.

"Appraisers have to have incredible listening skills because they have to read between the lines a little bit, because there are always guarded and protected positions, as people respond to discussions or interviews," he said. Further, appraisers need to approach medical device manufacturers with a softer touch than if they were assessing a company in a different industry, Hill said.

"To be part of the MDDAP program is a whole different animal for appraisers. Appraisers need to escalate their soft skills. That's because, in my opinion, I have never seen an industry that is so skittish when someone from the outside walks through the front door," he explained.

Hill believes device-makers can be leery of outsiders because of past interactions with regulators.

"The medical device industry and these operational sites that are producing devices seem to have been traumatized by some type of strong interactions between themselves and auditing that has occurred in the past," he said.

FDA has heavily stressed, however, that the maturity model pilot isn't about compliance. Rather, it's about driving process improvement. Still, some device-makers are going to be uncomfortable interacting with, and opening up to, CMMI appraisers.

"So, that level of skittishness requires a bit more soft skills to be able to engage those firms and develop a trust relationship, to the point where we can actually get them to open up and talk about perhaps what’s going well in their organization, and what’s not going so well in their organization," Hill said.

He added that there will always be some device company employees who won't let down their guard, but most loosen up and become more relaxed – eventually.

"That might not happen on the first day that I'm onsite during an appraisal. But usually sometime during the middle of the week, maybe on a Tuesday or even a Wednesday, the lightbulb comes on and they say, 'Hey, this guy isn't here to rake us over the coals. He genuinely cares and wants to provide us some objective feedback into our quality program, and perhaps we should listen,'" Hill said.

"Really, it's about the appraiser basically saying, 'Listen. I'm here to assist you, and that's all I care about. This isn't the FDA and this isn't auditing, and this isn't about compliance. This is just you and me, and we're going to talk about how we can make things better.'"

Device Appraisals A Passion For Hill, Layman

Currently FDA's pilot is using nine lead appraisers that have gone through MDDAP training and have been certified to participate in the pilot. One person has been certified as an Appraisal Team Member for MDDAP. Several others (appraisers and ATM candidates) have applied and are working their way through the approval process.

"Our goal at the beginning of the year was to have 20 appraisers in our pool by the end of the year to prepare to scale for 2019, with at last half of them being lead appraisers," CMMI's Kaplan said.

"The more people we have who are Appraisal Team Members, the more we can leverage our lead appraisers on additional appraisals, rather than having to combine [multiple] lead appraisers on the same appraisal."

Kaplan noted that if the pilot (and upcoming program) continues to be a success, CMMI will probably need 40 to 45 lead appraisers and ATMs in its MDDAP pool by the end of 2019.

"I smiled and said to myself, 'I've been waiting 20 years for this.' It's a breath of fresh air," Hill says of FDA's pilot.

Nevertheless, "there are probably a lot of appraisers that will approach the MDDAP program and then back away because they don't find it to be as lucrative as appraising companies in other industries," appraiser Hill said.

So why, then, is a seasoned CMMI lead appraiser like Hill throwing so much of his weight behind the device maturity pilot if he can make more money elsewhere?

"I’ll tell you exactly why," he said. "I spent a long time in the DOD, and the DOD has the same problem as FDA. They are about regulatory compliance. The bureaucracies of the government, even in the DOD space, is mind-boggling as to what you can and cannot do.

"For years I have fought the compliance-based regulation stuff in the DOD space. DOD has been using CMMI as a compliance framework for a very long time," Hill continued. "So, when I heard of this device program starting up, which doesn't have a checklist mentality and doesn't include compliance-based appraisals, and found out that appraisers would just go into a device firm and use the best-practice model as a guide to help figure out if there's something that can help those organizations – true, quality-based improvements – I jumped at the chance to be part of it.

"I smiled and said to myself, 'I've been waiting 20 years for this.' It's a breath of fresh air."

Lead appraiser Layman feels similarly.

"I’m very excited and motivated to be part of a program purposefully trying to get away from compliance, because sadly, we’ve moved too far that way with CMMI in the defense industry. And I’ve seen the downsides of that. So, I’m excited about what FDA is doing with this pilot," she said.

Layman isn't sure if MDDAP will one day drift toward a compliance mindset, but she's hoping it won't.

"Using a model that works for the medical device industry and gives medical device manufacturers an opportunity to focus on what is going to help them better meet their business objectives, rather than just focusing on maturity levels, is a big step in the right direction," Layman said.

"If the FDA can keep that at the forefront of this program – the concept that we’re trying to help device firms improve and satisfy their customers – that will be great," she said. "I think that will make the difference."

Still, "I am a little concerned about a future focus on maturity levels leading us down the path to just compliance for compliance sake – appraisals for the sake of achieving a maturity level, but not really something that’s going to improve companies," Layman noted.

"But I'm optimistic that won't happen. This pilot is different."

From the editors of The Gray Sheet