BFFs With FDA? Repeat Visits From Investigators Can Lead To Poor Device Quality, Recalls, Study Suggests

When a US FDA investigator is overly familiar with a company and its people, it can lead to a weaker facility inspection that might not uncover quality system problems – which in turn could lead to troublesome devices and recalls. So says a study conducted by professors at the University of Wisconsin, Indiana University and the University of Minnesota, which found that a device-maker's recall hazard increases 21% when it's inspected a second time by the same investigator. Even more concerning: The recall hazard shoots up 57% after an investigator's third audit of a firm. Might rotating investigator assignments help? And how might the agency's new "program alignment" inspectional scheme affect how often an investigator visits a particular firm?

Device-makers repeatedly inspected by the same US FDA investigators run the risk of making poor-quality products and conducting costly recalls. That's the conclusion of a study conducted by business and management professors at the University of Wisconsin, Indiana University and the University of Minnesota.

A 17-page paper published in August in the journal Manufacturing & Service Operations Management details the study, titled "Do Plant Inspections Predict Future Quality? The Role of Investigator Experience." Authors Enno Siemsen, George Ball and Rachna Shah say the more familiar an investigator is with a firm, the more likely it is that the company will undergo a recall event.

Through their work, the researchers discovered that a favorable FDA inspection outcome is associated with a 31% decrease in future recalls – but only when an investigator audits a manufacturer for the first time. After that, the "relationship between inspection outcomes and future recall hazard diminishes quickly as site-specific experience increases," they wrote.

In fact, "there is a 21% increase in the recall hazard on the second visit by the same investigator, and a 57% increase on the third visit, regardless of the inspection outcome from the investigator," the authors wrote. "After an investigator has visited a site more than once, his or her inspection outcome no longer serves as a reliable predictor of future recalls."

In other words, an investigator's familiarity with a company can lead to a weaker facility inspection that might not uncover quality system problems, which in turn could lead to troublesome devices and recalls.

"What we’re basically saying is that if investigators stay with the plant for too long, then that’s bad for the plant, in a sense. They don’t get the quality signals anymore. They don’t get good feedback," said Siemsen, a Procter & Gamble Bascom professor at the University of Wisconsin-Madison's School of Business, and executive director of the university's Erdman Center for Operations and Technology Management.

"Investigators do develop ties with the people at the plant, and the moment that happens, they become less adversarial and don’t ask the right questions anymore," Professor Enno Siemsen says.

For their study, the team reviewed device-related inspections and recalls data provided by FDA's Center for Devices and Radiological Health for the years 2000 to 2006, linking audit results and recalls to specific firms by matching that data to investigator and plant identifiers.

That seven-year timeframe was selected because FDA didn't have recalls data coded within a database before 2000. "We had CGMP [current good manufacturing practice] audit data going back all the way to 1994, but the recall data just wasn't there," Siemsen told Medtech Insight.

And the study didn't look at information from 2007 and beyond because FDA didn't include investigator identifiers in its inspection data after that time.

"The CGMP audit data is available publicly starting in 2008 and onward, but not with the investigator identifiers, which are key to linking the [inspection and recall] datasets," Siemsen said. "We only had CGMP audit data with investigator identifiers through 2006 – and very early into 2007 – that was complete. So, 2000 to 2006 was the timeframe where we could connect both of FDA's [inspection and recall] databases, essentially."

Although the study's data window closed 11 years ago, Siemsen says the study's results hold up because the agency didn't make any major changes to the way it implements inspections after 2006 – at least, not until it began its "program alignment" inspection scheme in May. (See "Might 'Program Alignment' Help With Investigator Rotation?" below.)

While the results of the data analysis are clear, determining causality – why there's an increase in recall risk after a second inspection by an investigator – is more difficult to pinpoint.

"Is this happening because investigators get tunnel vision and don’t see things anymore? Or is this happening because investigators develop social ties with the people at the plant and become less adversarial? Or is this happening because of some form of regulatory capture, where investigators believe they'll find employment in these firms at a later time? It could be any of those factors," Siemsen said. "From this kind of [inspection and recall] data, we can’t tell what the precise causal mechanism is."

Nevertheless, he believes the most plausible explanation is that investigators and company officials form social bonds.

"The first time an investigator inspects a firm, everybody [the investigator] meets at the plant is new to him or her. The second time, the investigator is seeing probably 60% to 70% of the same people at the firm. The investigator already knows them. And maybe in the meantime, there have been some interactions, so they're not quite as unfamiliar anymore," Siemsen said.

"Investigators do develop ties with the people at the plant, and the moment that happens, they become less adversarial and don’t ask the right questions anymore. And they don’t provide the right feedback anymore, which means one of the key mechanisms of how they function is basically not there anymore," he said. "I think that’s probably what’s driving it."

"In general, the sense we got was that people are aware that if they see the same investigator over and over, it becomes easier and easier for them to get through these audits," Siemsen says.

An investigator's chumminess with manufacturers might also extend to their overlooking or downplaying of problems.

A repeat investigator "knows the people who work at the plant. He or she has seen them regularly. So, the investigator might set that boundary a little bit more to the side to where they're not raising the alarm if they see a problem. Or the investigator might have an informal talk with the firm without putting issues found during the inspection into a warning letter – or even tone down the problems in the Establishment Inspection Report," Siemsen said. "I do think that is fundamentally what is going on."

The study paper details how a quality VP at "one of the world's largest medical device companies" told the authors that he was "pleased" when the same investigator would return for subsequent inspections because "he knew the inspection would be 'easier' than when an unknown investigator arrived."

But that's a double-edged sword, because while an easy inspection might not lead to unfavorable FDA-483 observations, it probably won't help a troubled quality system, either.

"There’s a conflict here where on the one hand, you don’t want to risk getting a warning letter or an FDA-483, or anything like that, because that’s bad publicity, but on the other hand, you do want to know early if things are going wrong," Siemsen said.

"That’s, I think, a clear tension within companies where different people have different incentives," he said. "But in general, the sense we got was that people are aware that if they see the same investigator over and over, it becomes easier and easier for them to get through these audits. And 'easier' really means that the inspections are less thorough and not as difficult to complete."

That's why it's of the utmost importance for device-makers to lean heavily on internal audits, rather than relying solely on findings from agency investigators, to determine the health of their quality system.

"If manufacturers don't get that good feedback from the FDA, then they better have good internal quality auditing systems in place," Siemsen said.

How The Study Came To Be

It took Professor Siemsen and his study coauthors – George Ball, an assistant professor at Indiana University's Kelley School of Business, and Rachna Shah, an associate professor at the University of Minnesota's Carlson School of Management – about four years to fully analyze FDA's data and publish their scholarly paper. ("The journey of a paper to publication is long," Siemsen lamented.)

The ball got rolling on the study when Ball, a former operations manager at Guidant (now part of Boston Scientific) and a former director of manufacturing at AGA Medical, approached Siemsen with a thesis: Inspection outcomes can be linked to product recalls.

"So I said, 'Why not look at that?' George had the expertise, having worked in the device industry, and I had the data. I already had FDA's CGMP audit data for medical device manufacturing on a hard disk but I never did anything with it. So, when George came along, it seemed like a great opportunity," Siemsen said. "Plus, there was the advantage that George had been on the receiving side of CMGP audits, so as a plant manager, he knew the plant’s perspective very, very well.

"That was sort of the starting point, and that's how the study and paper developed."

Investigator Rotation Recommended

The study recommends that FDA rotate investigator assignments to ensure that there's a fresh pair of eyes at every inspection.

"FDA may want to consider a policy of investigator rotation and refrain from ever sending an investigator to the same plant more than once," the study paper says. "An alternative solution may involve investigator sequencing, in which the FDA could ensure that an investigator never visits the same plant two times sequentially, or in a back-to-back manner."

The paper points out that if FDA "strategically decided to not allow investigators to inspect a specific plant more than once, [then] investigators would inspect approximately 10 new plants each year."

There would be a price, though. The paper states that additional travel costs incurred by a rotating slate of investigators would cost the agency more than $750,000 a year.

But it would be worth every penny, Siemsen said. That cost "is minimal, I think, compared to the potential benefits of protecting the public health. One major recall avoided would probably cover that," he said.

"Now, when I say $750,000, that's a small price compared to the cost to society for a major recall, but it’s still a big chunk of money, of course, for the FDA. The FDA is already severely budget-constrained," Siemsen said.

Because of that, "if investigator rotation is not going to happen immediately, then I can understand. Budget concerns are a key factor."

Might 'Program Alignment' Help With Investigator Rotation?

That FDA often sends the same investigators to inspect the same firms is openly acknowledged by industry.

In fact, for several months running up to the agency's launch of its ambitious "program alignment" initiative in May, experts were sounding the alarm that manufacturers might not be visited by the same investigator that they've been used to seeing every few years.

"It will be somewhat of a new era for the industry in that folks who traditionally were inspecting your firms may not be returning, and you will be seeing some new faces – hopefully better qualified and trained," King & Spalding's Steve Niedelman warned firms in a February Medtech Insightpodcast.

Program alignment is the most sweeping change to FDA's inspectional approach in agency history. Under the scheme, inspections performed by FDA's Office of Regulatory Affairs (ORA) – the office that conducts all of the agency's field activities – are structured along commodity-specific product lines to make audits more predictable and consistent for investigators and manufacturers. (Also see "'Program Alignment' Snaps Together: What's Next For US FDA's Inspection Scheme" - Medtech Insight, 9 Nov, 2017.)

"If your firm is in a particular area and I only have, say, three investigators in that area linked to your program, I'm not going to parachute somebody in from the other side of the country to do the inspection," FDA's Ellen Morrison says.

While device-makers might see new FDA faces at their facility door when inspection time comes, that doesn't mean the agency plans to consistently rotate those new commodity-focused investigators, as the study suggests FDA should do.

At FDAnews' 12^th Annual FDA Inspections Summit in Bethesda, Md., in November, FDA's Ellen Morrison said investigator repeats are sometimes unavoidable.

"If your firm is in a particular area and I only have, say, three investigators in that area linked to your program, I'm not going to parachute somebody in from the other side of the country to do the inspection," said Morrison, who is ORA's associate commissioner for operations, and head of ORA's new Office of Medical Products and Tobacco Operations (OMPTO).

"Generally, we don't want to send the same person in every time [to the same firm], but we don't have a hard-and-fast rule here," she told Medtech Insight.

Morrison's OMPTO office was born out of program alignment restructuring. OMPTO is made up of four distinctly different commodity-linked offices, including the Office of Medical Devices and Radiological Health Operations (OMDRHO). The OMDRHO office is split into three divisions, in the northeast, central/southern, and western areas of the country, encompassing all 20 former district offices.

But OMDRHO's division boundaries were drawn "out of convenience" and isn't meant to limit where investigators inspect, consultant David Chesney pointed out at the Inspections Summit.

An investigator "who is on the West Coast could very well end up doing inspections on the East Coast, but as Ellen said, FDA isn't going to have investigators crossing in the air over Toledo just for the sake of stirring the pot," he said.

However, "if there is an urgent matter, say, in Los Angeles, and the best-skilled person to do that inspection happens to reside in Florida, then FDA may send that investigator to California," Chesney said, noting that the three OMDRHO divisions are separated by "artificial geographic boundaries."

Chesney, who is principal and general manager of DL Chesney Consulting, was director of FDA's now-defunct San Francisco District Office from 1991 to 1995. He also worked for the agency in a variety of other roles over 23 years before retiring.

Firms Should Rotate Supplier Auditors, Too

The study also recommends that device-makers rotate their own auditors that are tasked with auditing vendors.

During supplier audits, "complacency and reduced inspection performance may arise from the stale and routine nature of inspecting the same location repeatedly, from investigator fatigue or lack of interest, or from developing friendly relationships with auditees," the study paper states.

Professor Siemsen elaborated: "You send your people to your suppliers, and if you always send the same people, there’s a very good chance that after the second or the third visit, they’ll have a very similar point-of-capture like that of FDA investigators, where now your people are familiar with the suppliers and they develop social relationships, and so on.

"One purchasing executive told me, 'We rotate our auditors to different accounts every six months because otherwise the suppliers will capture them, to some degree,'" he added. "So, I think there’s a lesson here to be learned for supply chain management and, in particular, for quality inspections in the supply chain."

FDA's Reaction To Study Findings

Roughly two years ago, Siemsen's study coauthors, George Ball and Rachna Shah, presented the results of their study to CDRH officials.

Interestingly, those officials "believed that site-specific experience was beneficial and would make inspection outcomes more (rather than less) predictive of recalls," the authors wrote.

Nevertheless, "senior CDRH leadership has confirmed that as a result of this study, the FDA is examining the feasibility of investigator rotation and/or investigator sequencing for medical device plant inspections," they added.

"From the interactions we’ve had, the FDA was very receptive to the idea of rotating investigators, but they also said they would consider this as practical advice," Siemsen said. "But we have never received any confirmation whether the agency actually made any policy changes to the way they allocate investigators."

When pressed by Medtech Insight, the agency would not corroborate the authors' assertions and characterizations, and would not say whether investigator rotation is something that's being considered.

"We are aware of the study and will carefully consider its findings," an FDA spokeswoman wrote in an email. "In general, the FDA does not comment on specific studies, but evaluates them as part of the body of evidence to further our understanding about a particular issue and assist in our mission to protect public health."

From the editors of The Gray Sheet