Human Factors Experts Forge Ahead With ‘Tricky’ In-Person Studies In The Age Of Coronavirus

Sanofi usability expert Molly Story and other longtime human factors professionals detail some extraordinary measures they’re taking to keep study participants and moderators safe, from the simple (masking up) to the more extreme (using separated testing rooms). Also: the US FDA weighs in on remote testing.

Not long ago, in the time before COVID-19, human factors professionals at Sanofi went about their day-to-day routines with a minimum of concern, conducting in-person “summative” testing of products – a type of validation that ensures user needs are met – often multiple times in a given month.

“We have so many products in our pipeline. We have so much going on all the time,” Molly Story, the company’s global MED advisor, told Medtech Insight. “We are testing something every month. Not always validation, of course, but we are constantly doing human factors testing.”

But things are different now at Sanofi, a global manufacturer of pharmaceuticals and combination drug-device products.

During human factors testing, plexiglass shields separate participant from expert. Face masks are mandatory. Social distancing is always observed. Testing rooms are vigorously cleaned.

“It’s a tricky business, but my team has created a full policy on how to conduct these studies” as the COVID-19 pandemic rages on, said Story, who was Sanofi’s head of global usability engineering and risk management for more than six years before taking on her current role.

“We’re basing our policy on WHO [World Health Organization] guidelines, CDC [US Centers for Disease Control and Prevention] guidelines and local guidelines," she said. “As a result, we’ve completed three validation studies under COVID conditions. We did two of them ‘face to face’ with a lot of precautions in place, and we didn’t have any trouble.”

But there are limitations. Sanofi is testing only in areas where the local economies are at least “partially open,” Story said, and the company refuses to test at-risk populations. “Our whole policy is pretty complicated, but we’ve been able to work within it.”

Because human factors studies are traditionally close-contact activities (an obvious no-no during a deadly pandemic), experts in the field have been forced to whip up creative solutions to keep the flow of new products moving, all the while keeping everyone involved in the process as safe as possible.

“That physical separation between the moderator and the participant, to me, is really powerful.” – Molly Story

Given the types of products Sanofi makes, a majority of its human factors testing must be done in-person; after all, the company isn’t comfortable sending potentially dangerous items like injection devices and drugs to users to test remotely.

“We don’t want not-approved devices floating around in the world. There is no guarantee we will get them back, and even if we do, who knows what happens in between,” Story said. “So we’re not going to do any remote delivery device testing.”

Instead, Sanofi has set up its testing rooms so there is no contact between the firm’s human factors experts – or moderators – and test participants.

“The moderator is in the room behind a plexiglass shield, and they’re both wearing masks, and they’re separated. They even come into the room through different doors,” Story explained. “And we space out the time between sessions more than we have in the past, for two reasons: first, to make sure we clean everything in between testing, and second, so that no two participants are in the building at the same time.”

While the new COVID-19 measures aren’t ideal, they nevertheless produced what she calls a “surprising” result.

During Sanofi’s recent summative tests, “the need for social distancing and the need to make sure that everything the participant touched was clean meant there was no interaction, which actually made the participants more independent,” Story said.

“That was surprising to me. The participants seemed to take more responsibility for the entire experience, and I believe it was closer to reality,” she said. “That physical separation between the moderator and the participant, to me, is really powerful. It really did seem to make a difference in the way they operated.

“Having witnessed thousands of these studies over the course of my career, I have to say that these particular tests were indeed different.”

In-person human factors testing “looks a little different than it used to, but it is going really well.” – Andrea Dwyer

In the age of coronavirus, Story’s story resonates with other human factors professionals, many of whom are trying to find their own way through the pandemic by putting unique twists on testing to see what works and what doesn’t in the new reality.

Emergo by UL is “doing a ton of testing in-person” during the pandemic by using, of course, personal protective equipment like gloves and masks, says Andrea Dwyer, associate research director for human factors research and design at the Boston-based consulting firm.

But Emergo’s human factors team has gone a step further by placing moderators and participants in entirely separate rooms.

“We only go into the same room when we need to. But we’re really able to leverage technology a lot to be able to have conversations as if we’re in the same room together,” Dwyer said on 16 September at RAPS Convergence 2020, hosted by the Regulatory Affairs Professionals Society.

In-person testing “looks a little different than it used to, but it is going really well,” she said, noting that – like Sanofi – Emergo isn’t testing anyone who is  immunocompromised.

“Participants seem to be really appreciative of all the steps we’re taking” to keep them safe, Dwyer added.

An important advantage to having moderators and participants in separate rooms is that face masks aren’t necessarily an issue, should the moderator need to see a participant’s full facial expressions as they test products.

“To me, the biggest loss in the COVID sessions has been the masking. You can’t see facial expressions,” Sanofi’s Story lamented, pointing out that human factors experts “can read a lot in someone’s face.”

Plus, she said, “it’s much more difficult to develop a rapport between the moderator and the participant when they’re both wearing masks.”

Let’s Talk About Remote Testing

In some cases, remote human factors testing has become a useful crutch during the pandemic.

Story recounted a recent remote test Sanofi conducted of a clinician portal for a software app the firm developed.

The clinician “shared their screen so we could see exactly where their mouse was and what they were doing, and we could talk to them,” she said. “But we weren’t there with a camera, so we couldn’t see what they were doing physically. But we could at least see what they were doing on the screen.”

Story added: “One of the nice things about doing it this way was that it was more realistic. The clinicians were using their own computer in their own environment.”

Conducting the test remotely was also less disruptive to the clinician’s day. Before the pandemic, Sanofi would have clinicians travel to one of the company’s testing sites to try out such a software interface.

“We are open to considering remote testing for validation testing that is supported by an adequate rationale and conducted via a protocol that is aligned with the principles of our [human factors] guidance document.” – US FDA

Stephen Wilcox, principal consultant for Philadelphia-based Design Science, says he isn’t particularly keen on remote testing.

“Our approach, for the most part, has been to tighten up our procedures to be able to do in-person testing safely rather than to focus on remote testing,” he said in an interview. “We’re successfully recruiting people to come to our labs. We’ve been doing testing every day for weeks.”

Wilcox conceded that some remote testing could be effective with a good video or other communications component, but it’s still not ideal because, “for example, there are times when the person running the test has to hand things to the participant.”

He also noted that “more complex devices tend to require demonstrations, training and pointing things out, which is hard to do remotely.”

Experts that spoke with Medtech Insight agreed that while remote testing has limitations, it can be helpful when performing “formative” human factors tests.

Formative tests are lightweight studies typically conducted early on in a product’s design process. Such tests happen before the more formal summative, or validation, testing takes place. (We explained the difference between formative and summative testing in a 2009 article. It’s still relevant today.)

Longtime human factors expert Ed Israelski says he’s a proponent of remote testing, especially for formative tests.

“Remote testing gives you better coverage. You can test more people, and you can do more iterations in terms of formative studies,” said Israelski, a former Abbott human factors director who is now self-employed as a consultant.

“With remote testing, you can do perhaps four or five iterations of formative, whereas if you did it all by scheduling travel, then you get into issues of budgeting and scheduling, and all of that kind of stuff,” he said in an interview.

Israelski believes that, much like the explosive growth in telehealth, an increased use of remote human factors testing is here to stay thanks to the pandemic.

“Remote testing is much like telemedicine, where patients and some clinicians didn’t have as much experience with it before COVID-19,” he said. “Now they’re saying, ‘Well, you know, it’s not as bad as I thought. It’s not perfect, but it’s tolerable.’

“That might not be a bad outcome.”

What FDA Says About Remote Testing

When it comes to formative human factors testing, “sponsors may choose to conduct [it] in a remote environment, but if remote testing is not sufficiently representative, then formative testing results may not be useful,” the US Food and Drug Administration told Medtech Insight through a spokesperson.

And while the agency isn’t particularly excited about the use of remote testing for summative tests, it is nevertheless open to the concept.

“Human factors testing depends on many factors related to the device, its use and design,” the FDA said. “We are open to considering remote testing for validation testing that is supported by an adequate rationale and conducted via a protocol that is aligned with the principles of our guidance document” on human factors.

The FDA released its guidance, “Applying Human Factors and Usability Engineering to Medical Devices,” in 2016.

“For summative (validation) testing, remote observation (video monitoring) should be comparable to having a moderator observing the user’s behavior in-person,” the agency advised. “Also, we would expect sponsors to test the intended users in a comparative-use environment.”

The FDA went on: “Whether remote human factors testing can support a premarket application depends on the representativeness of the validation test setup and methodology. This is because the agency needs to ensure that the results of the human factors validation test can be generalized to actual use of the subject device in order to provide reasonable assurance that it is safe and effective for its intended use.”

As for carrying out studies in the era of COVID-19, the FDA issued a guidance on the topic in March, and updated it in July. “Conduct of Clinical Trials of Medical Products During COVID-19 Public Health Emergency” explains how sponsors can use tools such as remote-monitoring technologies and adjust trial protocols to allow studies to continue. (Also see "Emergency Clinical Trial Guidance Offers A Peak At The Future Of Virtual Trials: Expert" - Medtech Insight, 7 Aug, 2020.)

The agency says its guidance “demonstrates FDA’s recognition that the COVID-19 pandemic may impact the conduct of clinical trials because of quarantines, site closures, travel limitations, interruptions to the supply chain for the investigational product, et cetera.”

“It seems to me that FDA will have to start accepting more remote tests for summative, otherwise things will come to a grinding halt in terms of new products.” – Ed Israelski

While the FDA insists it’s “open to considering” remote testing for summative tests, Israelski believes the agency is actually “very skeptical” about the approach.

“I don’t know of any validation tests that the FDA’s approved yet that allow for remote testing,” he said. “They have a lot of reservations about remote validation, because they want validation testing to be much more systematic, with larger sample sizes, making sure that they’re conducted as a real validation of what the product would be used like when it’s out in the field, on its own.”

But in the context of COVID-19, “it seems to me that FDA will have to start accepting more remote tests for summative, otherwise things will come to a grinding halt in terms of new products – other than those that get emergency use authorizations,” Israelski said.

He concedes, however, that there’s an “increased risk” that remote summative testing could lead to some user interface problems.

“There might be user interfaces that wouldn’t be as good as if you had done more rigorous, in-person testing,” Israelski said. “But remote testing can give you a lot of very valuable, useful information. After all, if this pandemic goes on for another two years, I certainly wouldn’t want the medical field to stop, in terms of advancement. So I think it’s a risk we would have to accept.”

However, Sanofi’s Story says she “disagree[s] with Ed on this one.”

“FDA shouldn’t loosen the rules just because it’s inconvenient,” she said. “Again, with precautions, you can do this testing in areas that are open. That, to me, is the biggest lesson: go someplace where the transmission rate is low and recruit from populations where there’s not much COVID. It can be done. You just have to be a little bit more creative.”

Story went on: “There are ways of doing this, and I don’t think it’s FDA’s responsibility to soften the rules because it’s more difficult. If you’re motivated, if you have product that needs to get out the door, then find a way to do it.”