FDA Investigators Play Fast And Loose With Quality System Inspection Technique, Experts Say – But Is It Par For The Course?

US FDA’s Quality System Inspection Technique – or QSIT – isn’t being followed to the “T” by many agency investigators, say industry experts, including two who authored the auditing approach in 1999. Sources tell Medtech Insight that QSIT, designed to ensure that investigators look at the most important compliance issues and ask pertinent questions linked to four major quality system subsystems, is routinely bypassed by investigators bent on inspecting their own way. But it’s not unreasonable for investigators to deviate from QSIT as they follow leads wherever they may go, says a former director of FDA’s Investigations Branch who left the agency in June. Meanwhile, a key player in the development of the Quality System Inspection Technique, Tim Wells, explains why he’s drafting a “new” QSIT for industry use.

Two authors of US FDA’s Quality System Inspection Technique say too many agency investigators aren’t always following the auditing approach when conducting facility inspections, often catching device manufacturers by surprise as they manage onsite FDA audits.

The claims come as one of the authors – Tim Wells – says QSIT is outdated and desperately needs a facelift; he is in the process of creating his own updated version of the document for use by companies as they internally audit.

Yet other industry experts say it isn’t unreasonable for investigators to stray slightly from QSIT and they believe the aging technique is still relevant. And FDA, for its part, maintained in a Nov. 22 email to Medtech Insight that QSIT is indeed used during the vast majority of inspections.

But Wells isn’t so sure. “Yes, there are investigators that follow QSIT, but let me assure you that there are way more investigators, in greater numbers, that simply don’t follow the technique,” he said. Wells is a former longtime CDRH staffer who led the Quality System Inspections Reengineering Team that devised QSIT. He spoke with Medtech Insight during separate Sept. 16, Oct. 3 and Nov. 14 interviews.

The 108-page QSIT manual directs investigators to look for particular information on a range of quality-related documents and records, among other things. QSIT was penned in the late 1990s and piloted with industry between October 1998 and February 1999. The technique officially took effect on Oct. 1, 1999, and has not been changed since, despite FDA’s insistence that it routinely reviews such guidance documents for possible revision.

QSIT is designed to make sure that investigators look at the most important compliance issues and ask pertinent questions linked to four major quality system subsystems: management controls, corrective and preventive action (CAPA), design controls, and production and process controls.

No other commodity – not drugs, not dietary supplements, not even foods – requires such a prescriptive inspectional technique, making medical devices unique when it comes to agency auditing.

The predictability of QSIT allows firms to prepare for an audit appropriately. Investigator deviations from the technique mean less predictable audits and, quite possibly, unfavorable inspection outcomes for the companies.

QSIT is used for abbreviated FDA Level 1 inspections, which covers CAPA plus only one other quality system subsystem; the investigator can choose between production and process controls, or design controls. It is also used for traditionally less stringent Level 2 inspections, during which investigators review all four subsystems.

FDA emphasizes that only select elements of QSIT are used during rigorous compliance follow-up, for-cause and risk-based inspections – but those types of inspections are outliers.

“Quality system inspections should generally be conducted using the Quality System Inspection Technique, and each inspection report is reviewed by regulatory officials in the agency for quality control. The FDA also performs audits of the program to help ensure that proper procedures are followed,” the agency noted in its Nov. 22 email.

FDA offers the QSIT guide online and routinely instructs manufacturers to download the document so they’ll know exactly what an investigator will look for during an inspection. The predictability of QSIT allows firms to prepare for an audit appropriately. Investigator deviations from the technique mean less predictable audits and, quite possibly, unfavorable inspection outcomes for the companies.

Wells conceded that FDA likely isn’t instructing investigators to bypass QSIT during Level 1 or 2 inspections; rather, the problem lies with select investigators who choose to audit their own way.

“For example, I attended a foreign inspection; I was in the front room where the investigator works. The investigator had the QSIT manual in front of him. He had it on the table. He asked a couple of questions, but then you could just tell that his comfort zone was complaints, and he was in his element,” said Wells, who is currently president of consulting firm QualityHub.

Beginning an inspection by looking at a firm’s complaints is considered to be a “bottom up” method to auditing. But QSIT emphasizes a “top-down” approach; that is, an investigator will look at a firm’s systems for addressing quality first before reviewing any specific quality problems.

“You could just tell by that investigator’s words, his questions, his language – everything – that he wasn’t comfortable using QSIT. When he got into design controls, he was uncomfortable. He didn’t feel confident in management controls. He was not comfortable at all.”

QSIT was a collaboration between FDA’s Center for Devices and Radiological Health and the Office of Regulatory Affairs, which handles all of the agency’s field activities.

The technique is merely the third and bottom layer of FDA’s activities around facility inspections. QSIT is used in conjunction with the Investigations Operations Manual which, in turn, is used with the Compliance Program Guidance Manual.

The IOM is the primary source regarding FDA policy and procedures for investigators. Meanwhile, the CPGM directs investigators on how to conduct an inspection for a wide variety of products, including medical devices, drugs, biologics and vet medicine. FDA has been using a device-related compliance program since the 1960s and revises it every now and again. It was last revised in February 2011.

The CPGM “outlines the inspectional strategy and provides guidance for inspectional coverage.” It points out that “flexibility” is feasible when conducting inspections, FDA noted in its email.

But Wells says some investigators routinely take advantage of such flexibility by slipping into a comfort zone.

“I’ve been in the front room for inspections and I know they’re not following QSIT. And I’ve been in the back room, seeing the many different items that the investigator asks for. I notice that they are items that typically aren’t asked for during a QSIT inspection,” QSIT coauthor Denise Dion says.

“I can see why FDA would have investigators that don’t necessarily follow QSIT to the letter. That’s because they’re going to look at stuff they’re comfortable with,” Wells said. “For example, investigators with an engineering background love design controls and production controls. They are bored by CAPA and complaints, and even gloss over them. But there are other investigators that love CAPA and complaints, and won’t hardly touch design controls.

“QSIT served a purpose at a time in history when there needed to be guidance around inspecting to FDA’s Quality System Regulation, which included new items such as design control and management control,” he continued. “Part of it was teaching investigators that there was a new regulation, and QSIT helped investigators ask the proper questions.”

But the Quality System Regulation, released in 1996, “is not new anymore,” Wells said, which has the effect of pushing some investigators their own way.

“What’s happening is that investigators are gravitating to their comfort zones,” he said. “A lot of the investigators aren’t held accountable in terms of their style or their mannerisms of doing inspections. We knew right after QSIT was launched that certain investigators wouldn’t follow it. But there’s no policing that I’m aware of where FDA says, ‘Hey, you didn’t follow QSIT. You forgot to do that. How come you didn’t ask them that?’”

Pam Weagraff of consulting firm Quintiles IMS pointed out that it isn’t out of the ordinary – or necessarily wrong – for investigators to gravitate toward comfort zones.

“If an investigator has expertise, say, in cleanrooms, and he or she is looking at production or process controls, then they may delve a little deeper into cleanrooms because that’s their area of expertise,” Weagraff told Medtech Insight on Dec. 5. “Or, let’s say an FDA investigator has expertise in software, so by going through design control, he or she might dig deeper into software development practices. So historically, that’s not unusual.”

Weagraff is director of Quintiles’ Medical Device and Diagnostics Regulatory Group, within the firm’s Regulatory Quality and Compliance Group.

Between 1999 and 2003, Weagraff worked with device trade industry group AdvaMed and the Health Industry Manufacturers Association to write three “points to consider” guides to help manufacturers better understand FDA’s QSIT approach to corrective and preventive action, design controls and management controls, comprising 65 pages of material.

Others See Divergence From Technique – Including Second QSIT Author

There are others in industry who agree with Wells’ assessments, including Jon Mullen, VP of quality for device-maker Cynosure, who says some investigators simply don’t stick to QSIT when inspecting.

“I wouldn’t disagree with that statement, that investigators aren’t always following QSIT,” Mullen said in a Dec. 5 interview with Medtech Insight. “Not that they’re going rogue inspecting things they shouldn’t be, but it does seem like many investigators have their own processes – processes that aren’t necessarily based on the steps of QSIT.”

Cynosure is a maker of light-based aesthetic and medical treatment systems, such as Accolade for removing lesions and Cellulaze for eliminating cellulite.

At a device firm where Mullen previously worked, the investigator didn’t review CAPA during the QSIT inspection. “That didn’t make sense because CAPA is the heart of your quality system. Basically, all of your problems go in your CAPA system, so it’s a very valuable tool for an investigator to figure out where your soft spots are,” he said. “And because CAPA is at the heart of QSIT, a QSIT inspection without a CAPA review is, by definition, not a QSIT inspection.”

Mullen kept his mouth shut when the investigator failed to look at CAPA during the QSIT audit. “I would not offer up, ‘Investigator, hey, let’s go through my CAPA system. You didn’t ask about it.’ It’s crazy for them not to go there because if you have a defective CAPA system, you’re absolutely going to find where the problems with the company are. When you’re sitting in an FDA inspection, there’s one thing you want, and that is for the inspection to be over. While an investigator is there, there’s danger – danger that things are going to come up.”

Aside from Tim Wells, six other device center and ORA officials – and three advisors – made up the rest of the core QSIT team when the document was drafted in the late ’90s. Among them was Denise Dion, an 18-year veteran of FDA where she served as an investigator. Dion, who helped write QSIT as part of ORA, is currently VP of regulatory and quality services for consulting firm EduQuest in Hyattstown, Md. She spoke with Medtech Insight on Sept. 27.

Dion was adamant that she, too, sees FDA investigators simply not following QSIT and she says it’s a common occurrence. (See box, “Denise Dion's Take.")

"I’ve been in the front room for inspections and I know they’re not following QSIT," she said. "And I’ve been in the back room, seeing the many different items that the investigator asks for. I notice that they are items that typically aren’t asked for during a QSIT inspection."

Denise Dion's Take

Medtech Insight: How do you know that investigators are not following QSIT? Are you basing this on inspections of your customers, your clients?

Denise Dion: I’ve been in the front room for inspections and I know they’re not following QSIT. And I’ve been in the back room, seeing the many different items that the investigator asks for. I notice that they are items that typically aren’t asked for during a QSIT inspection.

MTI Does this happen during a lot of inspections?

Dion: Yes.

MTI: I want to make sure that you feel comfortable saying that. That’s a strong statement to make, I assume, that many investigators aren’t using QSIT. I just want to make sure that you feel like you’ve seen enough inspections that you believe there might be a trend of investigators not using QSIT. Am I correct?

Dion: Yes.

Dion pointed to the Compliance Program Guidance Manual. Under Part III, Sec. A1a, the guide states that quality system “inspections should generally be conducted using the Quality System Inspection Technique.” (Emphasis added.)

“Well, generally investigators are going to follow QSIT, but that doesn’t mean they have to follow QSIT,” Dion said.

But that’s nonsense, says a former director of FDA’s Investigations Branch. Ricki Chase, now compliance practice director at Lachman Consultant Services, left the agency in June.

“What if the [CPGM] said, ‘You will follow QSIT’? Well, if you will follow QSIT, that means that during an abbreviated inspection the investigator will do X and will do Y, and will not do anything else because they will follow QSIT,” Chase said in a Nov. 8 interview.

“So then what? Should the investigator ignore public health risks because he or she must follow QSIT? That doesn’t make any sense,” she added.

In her role at FDA, Chase was responsible for all operations of the Investigations Branch, including inspections, investigations, sample collections, consumer complaints, import operations and emergency-response programs.

“The investigator should have the skillset to make a decision about which complaints they’re going to look at or which processes they’re going to look at. That is supposed to be their professional judgement in the moment, on the job. They’re the one doing the audit – not somebody sitting at a desk somewhere who’s not there with their hands in the mess,” Chase said.

“You have to allow the investigator the ability, based on their training and their knowledge, their skillset and their experience, to make judgement calls,” and sometimes drift from QSIT.

QSIT, Chase says, often begs the question: “Why do medical device investigators need somebody to tell them how to specifically conduct an inspection?”

“Well, quite frankly, I don’t think that they do need it,” she said. “If you train your people on what the regulation is and they have good technical skills, and they understand FD&C law and they can make good, sound judgements based on fact, which they can document in a report – why do you need anybody to tell you to follow an inspection technique? And why can’t you allow them the discretion to make good decisions in the course of their duties? That’s the real question.”

“There are some investigators that don’t follow QSIT because they don’t know what in the world they’re doing. There are some investigators that aren’t following QSIT because they think they can do their own darn thing,” former FDAer Ricki Chase says.

Chase says “no one should make generalized statements that all investigators don’t follow QSIT.”

Of course, there are “individuals who willfully do not do what they’re trained to do or what they’re supposed to do – sure, those types are at FDA, but they’re not abundant. They just simply are not abundant,” she said.

But, indeed, “there are some investigators that don’t follow QSIT because they don’t know what in the world they’re doing. There are some investigators that aren’t following QSIT because they think they can do their own darn thing. And then there are some people out there who aren’t following QSIT for a given reason – it’s probably because there are other factors or extenuating circumstances which would be explained in the [Establishment Inspection Report] as to why they did what they did.”

Chase pointed out that when investigators’ work is reviewed by supervisors, it will be checked to make sure that QSIT was used as a roadmap.

Nevertheless, “If an FDA investigator is assigned to a Level 1 inspection, they should be staying on point and doing a Level 1 inspection, which means they should be doing CAPA plus either design controls, or production and process controls. And that’s what they should focus on,” she said.

“They shouldn’t expand beyond that without some very good reason,” Chase added. “Sometimes you see people who just can’t help themselves, and they can’t seem to stay on point and they start wandering off. And that’s usually a sign of immaturity more than anything. Not a sign of willful intent.”

Still, she said, having an investigator at your facility that helps find problems by slightly veering off the QSIT path isn’t necessarily a bad thing.

“The question is, do you want the investigator who follows QSIT and looks at, say, only 11 complaints, fills out the FDA-483 form and walks away because that’s what they’re supposed to do?” Chase said. “Or do you want the investigator to says, ‘Well, I know that’s what QSIT says I’m supposed to do, but I also know that based on my experience I think they have a much bigger problem than just these 11 complaints.’ I’d like to think that manufacturers would like to know about the problems.”

Fresh Investigators Follow QSIT To A ‘T’

Perhaps surprisingly, veteran investigators are more likely to deviate from QSIT during an inspection, former FDAer Chase said.

“What I tended to find during my time at FDA was that when you have senior investigators, particularly people holding the title of ‘specialist’ or ‘expert,’ they think the rules don’t apply to them. And they tend to just wander all over the map, and they say all kinds of things they shouldn’t say. And they do all kinds of things that they shouldn’t do. It pisses people off, and rightly so. And it happens. And when it happens, and the district management is made aware of it, the individual is counseled.

“There are rogue investigators out there. I’ve worked with them. I’ve managed them. I’ve directed them. They exist,” Chase says.

“If they have to be continuously counseled, then it becomes not only a performance issue, but also a conduct issue. And there are steps in place to address that,” Chase continued. “I won’t comment on whether I think those steps are effective, but there are steps in place that are intended to address those types of behaviors.

“But does it happen? Yeah, it happens. There are rogue investigators out there. I’ve worked with them. I’ve managed them. I’ve directed them. They exist.”

In general, investigators that most closely follow QSIT are those who are new to the agency.

“The new investigators are so terrified that they’re going to make a mistake that they tend to be very slow and very methodical, and very by-the-book,” Chase said. “They follow the QSIT manual line-by-line-by-line because they don’t want to make a mistake. So, your newer investigators – while they may be less experienced on the technological side – they’re very, very by-the-book.”

Do Firms Even Comprehend QSIT?

Sure, an industry quality-systems expert would likely notice if QSIT was not followed by an FDA investigator. But would a run-of-the-mill staffer at a device company even know if an auditor strayed from QSIT? Chase said she isn’t sure.

“There are a lot of manufacturers that don’t know how to follow QSIT – in fact, there are a lot of them out there that don’t,” she said. “And they don’t know what QSIT really means, and they don’t know how investigators are trained to interpret what QSIT means.”

After all, “there are plenty of manufacturers out there that don’t even have an SOP for how to handle an audit or how to handle an inspection. There are a lot of manufacturers out there that don’t do mock audits,” Chase said.

“Are those companies familiar with QSIT? Yeah, they probably read it. Just like they’re familiar with the [Investigations Operations Manual]. They know it exists,” she said. “But I tell companies this all the time: If you really want to know what’s going on, read the IOM, read the regulations, read the QSIT manual, read the Compliance Program Guidance Manual, read the Regulatory Procedures Manual – read everything you can find.”

However, there are few firms that have the money to hire someone to gather and review that information.

“It’s usually your larger firms that have the resources to do that,” Chase said. “You have to remember that the vast majority of medical device companies are medium-to-small firms. They are not big firms like Hospira and J&J. The majority are small- to medium-sized firms that don’t have the resources. What they do is they hold their breath. They hope that when an investigator comes in and does an inspection they’re going to be OK.”

MDSAP: QSIT ‘On Steroids’

But QSIT’s effect on industry could be waning now that the Medical Device Single Audit Program is up, running, and gaining more manufacturer attention by the day.

MDSAP, created by the International Medical Device Regulators Forum, allows firms to undergo one audit by an accredited third party to satisfy quality regulations for the US, Canada, Brazil, Japan and Australia. Like QSIT, MDSAP is a process model.

“FDA is accepting the MDSAP audit reports as a substitute for a routine inspection. But they’re not accepting MDSAP for for-cause or compliance follow-ups. They also will not accept MDSAP for pre-market approval applications,” consultant Weagraff said.

“The movement away from the pure QSIT program could be because those types of inspections are for-cause, or compliance, or pre-market approval. And those types of inspections, simply due to the nature of them, would not necessarily strictly follow the QSIT program,” she noted.

“A lot of times investigators will call you up for a QSIT inspection and tell you the things they would like to see. But it’s not a standard list. It’s inspector-specific. ‘I want the quality manual. I want this SOP. That SOP’ – you never know. It’s all over the map,” Cynosure’s Jon Mullen says.

Weagraff said MDSAP is “QSIT on steroids,” because whereas FDA allows for certain quality system subsystems to be skipped under QSIT, MDSAP’s process is extremely thorough, charting an auditing map for investigators.

“MDSAP is looking at all four of the major subsystems. It’s looking at management controls. It’s looking at CAPA, design controls, production and process controls,” she said. “In this case, the steroids are that instead of having an option to pick design controls or production and process controls like under QSIT, auditors are in fact doing them both – and even more – under MDSAP.”

Mullen of device firm Cynosure said his firm underwent its first MDSAP audit in October, an experience he said was overwhelmingly positive. Cynosure’s next MDSAP audit will come in March at the company’s Hicksville, NY, facility. Mullen said he prefers an MDSAP audit to a QSIT inspection.

“QSIT can be non-predictive. Investigators can go all over the place. Who knows what the investigators are going to cover,” Mullen said. “Whereas with MDSAP, you know the amount of time they’re going to be there. And with QSIT you don’t. I’ve had FDA investigators stay three weeks, and it’s brutal.”

MDSAP “certainly covers everything in QSIT, but it’s also much more prescriptive,” he said. “They have a guide they follow and they stick to it. In fact, we asked the MDSAP auditor to deviate – we wanted a section covered earlier because of the availability of people – and the auditor said he couldn’t do that because MDSAP has to follow the sequence that is prescribed.

“MDSAP is more predictive than QSIT, even though QSIT is supposed to be predictive. We knew everything for MDSAP. We knew how to prepare everyone and get all the information ready for MDSAP,” Mullen continued. “A lot of times investigators will call you up for a QSIT inspection and tell you the things they would like to see. But it’s not a standard list. It’s inspector-specific. ‘I want the quality manual. I want this SOP. That SOP’ – you never know. It’s all over the map.”

QSIT – Designed To Save Time – Doesn’t Really Do That

QSIT directs investigators to cover 38 core points related to CAPA, management controls, design controls, and production and process controls during a facility inspection.

“Those 38 points were what we thought were the key areas of a quality system back in 1999,” QSIT coauthor Wells said. “Looking back, stupid things – such as, ‘Did the firm create a quality policy and is it understood?’ – could have been removed from QSIT.”

That’s because investigators “are wasting good auditing time asking questions that aren’t necessarily pertinent; for example, investigators don’t even know how to ask good questions when it comes to management controls,” Wells said.

“When I left FDA there were the four subsystems in QSIT,” he said. “But what the agency later proclaimed – after I left – was that investigators can do a [Level 1] ‘CAPA plus one’ inspection, which allows investigators to audit only two systems. That approach supposedly allows for a two- or three-day inspection.”

Wells said FDA created CAPA plus one because it was trying to cram more inspections into an investigator’s limited auditing time.

“Well, gosh, you can’t really find anything in two or three days. It takes half a day just to figure out where the bathrooms are, and to get a product overview and an understanding of the building, the layout plan, and all those things,” Wells said. “Now, I’m being rather facetious, but if the investigator is limited in time, then, dammit, they better focus the two days they are there on the most important things.”

That’s because “if you inspect in less time, you’re just not going to get a fruitful inspection. You don’t have time to dig. But investigators need that time to peel the onion back two or three layers. That’s when FDA finds all the juicy stuff.”

“QSIT was designed for FDA investigators to be quicker and more focused, but nowadays everybody is using it like it’s the Bible for doing audits. But it’s missing stuff. QSIT doesn’t even ask about supplier controls,” QSIT coauthor Tim Wells says.

Although QSIT was set up to allow investigators to quickly inspect, it runs up against real-world issues that unfortunately cause delay, former FDAer Chase said.

“QSIT was designed so that if you did an abbreviated inspection, the most it should take the investigator would be one day for CAPA and one day for whatever the other subsystem is that’s been chosen, and then maybe a day for tying up loose ends and closing out. But you must remember: That’s an ideal world,” she said.

“That’s when the firm is cooperating, when they have the records available, when they give you the records, and when you’re not finding major deviations,” Chase noted.

“But there are many reasons why that doesn’t work and why QSIT audits take longer than expected, such as the firm doesn’t cooperate, the firm doesn’t bring the investigator the adequate documents or the firm uses delay tactics,” she said. “Or, the investigator might uncover a public health issue that will take time to process.”

Ultimately, what Wells would like to see is an FDA that conducts fewer inspections, but schedules them to be twice as long.

“In other words, if they inspected for 10 days total, or if they had two auditors at the inspection, hell, they would get some good inspections done. But that’s not going to fly because they want to inspect these firms every two years, yet the number of firms is increasing – and the number of investigators isn’t,” he said. “It is a dilemma that needs to be solved.”

‘New’ QSIT Being Drafted – But Why?

Wells has been busy at work drafting a modern version of QSIT that allows for the most important parts of a quality system to be reviewed within a reasonable – yet appropriate – amount of time.

“QSIT was designed for FDA investigators to be quicker and more focused, but nowadays everybody is using it like it’s the Bible for doing audits,” he said. “But it’s missing stuff. QSIT doesn’t even ask about supplier controls. It was missing stuff from right after we launched it in 1999. Within two weeks, I started getting complaints that QSIT was missing certain things.

“So all I’m doing is trying to supplement it, I guess you could say, with stronger questions and more focus.”

The “new” QSIT covers four quality system-related areas: post-market surveillance, design controls, supplier and process controls, and management controls.

“Instead of calling it ‘CAPA,’ I call my first subsection ‘post-market surveillance,’ which includes both CAPA and complaints. This new title lets auditors know that they’re not just dealing with CAPA,” Wells said. There are eight core points, or “expectations and guidance,” in this section.

Old QSIT Vs. Proposed New QSIT: QS Subsection Correlations
Old		New
CAPA: 10 core points	Vs.	Post-Market Surveillance: 8 core points
Design Controls: 15 core points	Vs.	Design Controls: 8 core points
Production and Process Controls: 6 core points	Vs.	Supplier and Process Controls: 6 core points
Management Controls: 7 core points	Vs.	Management Controls: 4 core points

The updated technique includes a mix of bottom-up and top-down styles of auditing.

“In this new tool, it’s probably more than an 80 percent bottom-up style of auditing,” Wells said. “Industry might not like that because if you can get the investigators to focus on, say, procedures – which is a top-down approach – then they know everything will be OK because anyone and everyone can write a procedure. Nothing will be challenged.”

The new QSIT is in a two-column format. The left-hand column includes FDA expectations; the column on the right offers audit guidance to ensure that expectations have been met.

In a Nov. 9 email to Medtech Insight, Wells explained: “I created the expectation column because I feel both the Quality System Regulation and the QSIT guidance fall short in explaining what is actually expected. While folks may argue this is best practice, I would argue that the agency expects these items.”

He added: “The first column showing the expectation is a roadmap for industry. While it is a regurgitation of the regulations, in many instances it takes us further.”

Consider these three core points on post-market surveillance from Wells’ new QSIT document:

Post-Market Surveillance
Expectation	Audit Guidance
Procedures: Both CAPA and complaint-handling procedures address all of the requirements of the FDA Quality System Regulation and Medical Device Reporting (MDR) regulation.	Top Down	Review of procedures, forms, work instructions and examples.
Trending: Appropriate statistical methods are used for complaint and CAPA trending. Software utilized for trending is adequate, appropriate and validated as necessary.	Bottom Up	Review trending of both complaints and CAPAs. Challenge the statistical thresholds for opening CAPAs. Confirm complaint frequency and severity are tied back to design risk analysis work (such as Failure Mode Effects Analysis).
Complaint Investigations: Post-market surveillance activities ensure that complaint-handling investigations are done to the necessary depth and based on the reported risk. Complaints are handled in a timely manner in order to allow for MDR considerations.	Bottom Up	Request printouts that show data, including the complaint allegations. Select samples based on risk. Review complaints that resulted in MDRs and some that did not result in MDRs. Determine their rationale for not reporting if you believe the complaint should have been reported. Determine adequacy (depth, timeliness and quality) of work done.

Wells knows that FDA would probably never consider his new document as a QSIT replacement; rather, he hopes it will be used by manufacturers as they audit internally.

“Sadly, everybody relies on FDA to be their auditors. They take those audits to the bank and think Moses brought them down from the mountain, when in reality they need to clean up their own act, have their own audits, have their own processes,” he said. “A lot of companies are using QSIT as their internal audit guide. I’m simply saying, ‘Hey, you could do way better.’”

Despite there being fewer core points to cover under the new QSIT, audits “are not limited by four-and-a-half days. Firms could take as much time as they need to cover their quality system,” Wells added, noting that he would be finished soon with the updated guide.

Three QSIT Areas That Could Be Bridged Now

While industry awaits Wells’ new QSIT, there are gaps in the current version of the document that could be easily bridged without redoing the entire document.

During an interview, Wells pointed out three specific gaps that concern him:

Design Transfer. “I see this as a major gap in many companies. Often Industry does not understand that design control includes the need to ensure someone (internal to your company or external) has validated processes, qualified equipment, test methodologies and equipment, trained personnel, adequate work instructions, calibrated instruments, et cetera, so they can properly make the product.

“They can’t throw the new design over the wall for someone else to manufacture. Product development people should own the process until the above items are completed and the manufacturing site has proven to be capable. This may slow down the product launch but it is essential. Lack of adequate design transfer is responsible for a lot of recalls, as well as injuries.”

Management Controls. “A lot of people seem to think management controls is management review. It’s actually much more than that. They also need quality goals and objectives so they can continually drive the organization into the area of improving their products and their quality system.

“Measurable outcomes should be expected, monitored and reported out in management review via scorecards and metric reports. Also, resources are only briefly mentioned in QSIT. Yet lack of resources (for example, when a company leans down their quality organization) can lead to massive compliance gaps, such as complaints and CAPAs not being reviewed, and Medical Device Reports not being filed. I believe QSIT would benefit by adding more specific questions on quality goals and objectives, and also on resources.”

CAPA. “The second and third CAPA questions in QSIT are clear and well written in that sources of quality data should be analyzed as part of CAPA. But FDA doesn’t go far enough, in my opinion, to drive companies to actually open CAPAs.

“For example, if you have a dysfunctional training process, you then need a CAPA to address why you have a dysfunctional training process. CAPA is too often used only for product issues. I see in industry that CAPAs are generally underutilized. Opening CAPAs forces the issues to be addressed. Management should be challenging the organization to not just close CAPAs, but to open CAPAs when they have gaps in their quality system. QSIT could sprinkle the CAPA question throughout the audit guide to ask if CAPAs are being opened.”

From the editors of The Gray Sheet