How Inefficient are Clinical Trials?

This is the second article in a series about clinical trials and opportunities for reform. You can find the first article in the series here.

This article is about the challenges and inefficiencies of clinical trials. But before I cover the problems with trials, I want to praise them: Modern clinical trials are a marvel. They have been responsible for expanding our scientific knowledge and unlocking many treatments that otherwise would be unavailable or impossible to evaluate. They’re also continuing to evolve and improve; many of the methodological elements that are considered essential to clinical trials today, such as pre-registration, rigor around hypothesis generation and testing, and data safety monitoring boards, have only recently become widespread. Moreover, their influence has not been limited to medicine. As the science of clinical trials was developed and evolved in the 1960’s, similar approaches became more widespread in other fields, such as development economics.

Today’s clinical trial system is capable of remarkable feats. Everyone is aware of how rapidly COVID vaccine trials were set up and executed. And we are asking more of clinical trials than ever, as a growing number of drugs target rare diseases with difficult-to-recruit patient populations, with challenging and complex designs.

Yet the challenges of our clinical trial system are evident in the statistics. Today’s clinical trials cost more than ever. While precise estimates of clinical trial costs are hard to come by, one widespread estimate is that trials cost about $40,000 per patient. In my last article in this series, I highlighted a quote from George Yancopoulos at Regeneron (which develops treatments primarily for rare, serious diseases), who noted that, in his experience, costs had risen from $10,000 to up to $500,000 per patient. Other analysts have noted a more modest, but ongoing increase in overall trial costs. The trials that do take place are getting smaller, which is consistent with a story of rapidly increasing per-patient costs. And all of this is consistent with a broader trend in research called Eroom’s law: a pattern of declining productivity in pharmaceutical research and development that has been taking place since the 1950’s.

Rising costs are driven by increased trial complexity. A growing chorus of pundits like Ezra Klein have taken to decrying “everything bagel” policies that try to prioritize too much at once. Clinical trials are plagued by a similar problem, which former FDA commissioner Robert Califf has referred to as “Christmas tree protocols”: trials that are overburdened and overstuffed with features. The concept of clinical trial complexity isn’t well-defined, but we can look at how complexity is increasing across the two major drivers of clinical trial costs: recruitment and data collection. Inclusion/exclusion criteria in clinical trials are extensive and often poorly justified, making recruitment challenging. And clinical trials are collecting vastly more data than they once did; research from the Tufts Center for the Study of Drug Development reveals that from 2010 to 2020, trials nearly doubled the number of endpoints they assessed and tripled the number of data points they collected.

Challenges on the ground

Zooming in further to look at how clinical trials are conducted makes the challenges clearer. For readers who are not familiar with how clinical trials work, I’d like to start by emphasizing how decentralized clinical trials are. Large clinical trials are run by a federation of trial “sites” – typically academic medical centers or other large health systems – each of whom are responsible for reading a clinical trial protocol written by the pharmaceutical company and figuring out how to implement it. Companies send armies of inspectors on site visits to make sure that the trials are being conducted in accordance with the protocol. Rather than managing this federation of sites themselves, most drug companies hire contract research organizations (CROs) to make sure the trial is running as planned.

A few years ago, a blogger pseudonymously named “milky eggs” provided a fascinating description of what it’s actually like to operate a clinical trial from the perspective of a biotech company. It’s hard to summarize, and I recommend you go read the whole thing yourself, but the author complains about how bureaucratized, meeting-intensive, and un-automated the industry is. The author concludes: “Everyone in this industry is acclimated to a very low level of productivity.” Is this level of inefficiency widespread? I am not certain - and I know there are examples of well-run, efficient trials out there. But this account is consistent with the statistics on trial efficiency, and makes sense based on what we know about how clinical trials are run.

Clinical trials have adopted some technology, but in many ways their process is still paper-based; they’ve simply replaced actual paper with its digital equivalent. This is evident in the names of the software used in the industry: Electronic Source, Electronic Trial Master Files, Electronic Data Capture, and Electronic Patient-Reported Outcomes. Industry is working on a digital-first approach to trial operations that may improve things, but work is still in the early stages, and the old paper-based processes are deeply entrenched.

There is now lots of enthusiasm about using data collected in routine health care in electronic health records to improve clinical trials, with several established technology companies, startups and academics working on this, including my former employer, MITRE and their colleagues at the Alliance for Clinical Trials in Oncology. But it is still considered a speculative and experimental concept. You might see data from an electronic health record get manually transcribed into a case report form, but you will rarely, if ever, see clinical trials actually pull data out of the electronic health record and into the drug manufacturer’s trial database. Paper is still the primary data entry tool in most clinical trials.

Data collection in clinical trials: a study in waste

Perhaps no practice better illustrates the waste and inefficiency in clinical trials than the process for study investigators to collect and verify the accuracy of patient data. Here’s how it works: First, the drug company writes a clinical trial protocol that specifies, at a high level, what data needs to be captured and when. For example a protocol might ask study sites to collect data on adverse events, liver/kidney function, and key endpoints at different timepoints. To collect the data from the sites, the pharmaceutical company develops electronic “case report forms” for sites to complete (If you’ve used SurveyMonkey or Qualtrics you have a good sense of how these forms are filled out).

You might expect clinicians running a clinical trial to fill out those case report forms when their patients come in for visits, but that is not in fact what happens. Instead, each site develops its own method for collecting the data (this is typically done on paper), then transcribes them into the case report form after the visit is over. Naturally, this transcription can introduce errors, so to address that, most trials do something called “100% source data verification”, in which every single entry on every case report form is compared against the records captured at the study site to make sure it was transcribed accurately. This step is reported to account for an average of 25% of the entire clinical trial budget.

It seems like source data verification ought to be simple enough to eliminate. Couldn’t you just capture the data electronically when the patient comes in and send it straight to the drug company, rather than transcribing it? This does happen sometimes, but it is surprisingly uncommon. On top of that, it’s not clear who is even asking pharmaceutical companies to do 100% source data verification. In fact, the FDA has actively discouraged it on multiple occasions, instead promoting a more selective form of data verification called “risk-based monitoring”, a practice also encouraged by global standards. Also, the benefits of 100% source data verification in terms of data accuracy are minimal; very few errors are caught using this method, and they are rarely consequential.

And yet the practice continues. FDA has held conferences trying to figure out why it’s still happening and how to promote alternatives, one of which I helped organize in 2019. IPF has also published an issue brief with some good ideas on how to promote alternatives to 100% source data verification. The fact that 100% source data verification remains common in trials, despite these efforts, can tell us a lot about the intractability of the challenges facing the clinical trials industry today.

Final thoughts

It is clear that clinical trials are inefficient. That inefficiency shows up in the statistics, and it is felt on the ground too: complex protocols with too many endpoints and too much data collection are slowing down recruitment and driving up costs. Trial sites are overburdened with data entry and verification along with cumbersome paper-based processes. The industry is plagued by burnout and turnover. And even when obvious sources of inefficiency are found, they don’t seem to get fixed.

The big question is: what is driving this? And what can be done about it? I look forward to sharing some ideas in future posts.

Adam