I recently wrote a 2,000 word note for funders that explains why I think market design ideas and trial design ideas should work closely together. I am adapting it lightly for our blog. I start by explaining what innovative designs and market shaping are, so some of you may want to skip these first two sections. At the end I also try to explain why I think this idea is particularly timely.

(1) Innovative trial designs

In a nutshell, innovative trial designs (see glossary in a box below) can reduce costs and speed up testing of interventions. They achieve higher efficiency by creating shared trial infrastructures, borrowing information across populations, and adapting testing to information that is learned during the experiment. I previously argued that trial innovation is particularly important in this period of AI advances: as drug discovery improves, having an efficient infrastructure for testing innovations in the real world will increasingly become the main bottleneck.

Historically, most methodological innovation in trials came from medicine. However, clinical trials remain overly conservative. Drug regulators are not adequately equipped to support innovation. Pharma companies are averse to it, especially for expensive late-stage trials. Existing research infrastructure and contract research organisations are set up for conducting traditional trials. Outside of clinical trials, e.g. in development, adoption of innovative designs is also limited.

To illustrate this slow rate of progress, the first FDA draft guidance for Bayesian clinical trials of drugs and biologics was only published this year. And a review paper by Griessbach et al found only 127 platform RCTs in total, with over 80% most of them in oncology and COVID. In contrast, every year the FDA receives over 2,000 investigational new drug applications and over 20,000 new trials are registered on ClinicalTrials.gov.

For medical interventions that need regulatory approval this creates an additional chicken-and-egg problem: regulators are not set-up for innovation because producers do not demand it and producers do not demand it because regulators are not set-up for innovation.

For that reason we should not expect the trial innovation to come from producers or regulators.

If you’re familiar with innovative designs, you may want to skip to the next section now

Glossary of innovative trial designs

There is a broader need for trial innovation, but here I talk about their statistical aspect, trial designs. “Innovative designs” (sometimes “complex innovative designs”) is a catch-all terms without a clear definition, but it’s good to distinguish the following:

• platform trials, which focus on creating infrastructure for many (sometimes dynamically added) treatments;

• adaptive trials, which pre-register dynamic decisions (e.g. dropping a treatment or changing dose based on results in an initial set of patients);

• Bayesian trials, which can borrow relevant information, e.g. across trials or patient groups;

• related to the platform idea are multi-stage, multi-arm trials, which are built for comparison of many promising interventions (and eliminating the less promising ones with time) and seamless trials, which use a single trial protocol for e.g. choosing the best dose of treatment (Phase 2) and then large-scale testing of its effects (Phase 3);

• basket trials (one therapy, many diseases) and umbrella trials (many therapies, one disease) are typically used for targeting treatments in cancer.

There can be a lot of overlap in these definitions and most trials meet several of these definitions.

Example: adaptive platform trial for patients hospitalised with Covid-19

RECOVERY, a large-scale platform trial that started in the UK, was responsible for rapidly evaluating many possible COVID treatments in 2020. It was run within the UK’s National Health Service. Its headline finding was that dexamethasone, a cheap generic steroid, could reduce a large fraction of mortality during that phase of the pandemic—as well as several other potentially beneficial treatments. This required testing 20 treatments with a flexible protocol that was adapted dozens of times.

I will give more examples later in this note.

(2) Market shaping

In science and global health, large philanthropic and public bodies plug gaps left by private R&D—both in terms of existing and new technologies.

For existing technologies, let us stay with the example of clinical trials. In that world producers, in pursuit of regulatory approval, may prefer for evidence to stay fragmented and put effort into carefully choosing trial design features (population, outcome, comparators) to ensure the most favourable results. At the same time, producers also underinvest in socially valuable evidence. For example, (1) adjusting dosage, (2) impact of combining several drugs, (3) repurposing (unless there is a lot of time left on the patent—see below). Public and philanthropic funding is often directed to resolve these issues.

Beyond generating such new evidence for existing technologies, these funders also look to support new technologies for which there is currently a market failure. They can do it through using market shaping with “push” or “pull” funding. In simple terms:

• push model (pay for inputs): funder pays upfront, with a traditional, no-risk research funding model: e.g. paying for a Phase 1 clinical trial for a highly promising universal flu vaccine;

• pull model (pay for outcome): funder promises to pay for an invention, e.g. committing to purchase a billion doses of a universal vaccine—provided it gets approved by the regulator.

The most important difference is that pull funding can work if the funder can clearly specify in advance what would count as success, but producers know more than the funder about how to get there. Pull initiatives include supporting development of pneumococcal vaccines, neonatal sepsis tests, or carbon removal technologies.

Pull and push can be used not just for new technologies, but also to solve the R&D gaps that I just mentioned in existing technologies. As an example, pull funding mechanisms have been proposed for repurposing. That is because in many cases repurposing targets are off-patent medicines, where the main barrier to repurposing is efficient research, as no single company can easily generate revenue to fund a large confirmatory trial.

(3) An example of how market shaping and trial design fit together

I argue that innovative designs are very relevant to funders interested in market shaping. At a basic level any funder wants to ensure that all evaluations of possible solutions are as fast, powerful, and inexpensive as possible. This is something that innovative designs can help with. However, there are also some cases where innovative designs have some unique advantages for market shaping.

To make this concrete, let’s imagine a funder who wants to support development of a broad spectrum antiviral. They do it by creating an advance market commitment: a promise to buy a large number of doses of an antiviral, provided it gets regulatory approval and meets a target product profile (TPP), e.g. some efficacy threshold.

Five firms that had ideas for how to develop an antiviral now decide to move into human subject testing, hoping to get their product approved, meet the TPP, and win the big AMC market. All five of them do well in early trials and are now interested in proving efficacy. How should these five antivirals be tested?

(Is five firms a high number? For many innovations we will only have one or a couple of firms that can move to this stage, but I will argue below that with time it may change.)

In one world there are five separate trials aimed at proving efficacy: five protocols, five placebo groups. Each company designs their trials in a way that they think will get them to regulatory approval and meet the TPP—and unlikely to generate any information beyond this, e.g. on dosing.

In another world (which I prefer), the AMC designer is also the trial designer. Firms are given an offer to participate in a platform trial. In exchange for playing under the same rules as competitors they get access to unified trial infrastructure: patients, trial approval, regulatory oversight.

This trial design can extend further. It can dynamically remove treatments that do not show promising results. It may start as a small scale evaluation and then seamlessly transition into a large-scale efficacy trial of the best candidate. It may also look at adjusting dosages or considering combinations of treatments.

(4) Advantages of jointly tackling trial and market design

What are the major advantages of my suggestion?

1. Because everything is done in the same trial, it is cheaper, faster, and statistically more efficient (less bias, more power).

2. A shared design allows the funder to make better head-to-head comparisons between these antivirals. In some cases it also avoids companies from competing for trial subjects.

3. It can de-risk a sensitive part of the R&D process for firms. For producers many uncertainties revolve around whether their intervention can feasibly be tested. Under this proposal the funder gets control of the information that is generated, but assumes (some of) the burden of creating the testing infrastructure. This solution can also be thought about as an intermediate solution between push and pull funding—especially if the funder also covers some/all costs of the trial.

It’s important to add that while above I tried to give a specific example, this idea is not about a particular trial design, particular type of innovation, or a set number of firms.

At the same time, complex trials like in my example are not always needed, or even feasible. Continuing the example, if each of the five antivirals was developed at a different time, it would not be advisable to have a single shared control group due to time lags. (Although there are ways of working around this issue that are worth discussing separately.) More generally, sometimes an innovative design is counterproductive—e.g. when it imposes considerable overheads—and a traditional RCT is the best choice.

However, even when there is no need for any innovative elements, it may be best to avoid leaving the trials entirely to firms, for the following two reasons. First, many nascent technologies come from academic research teams or, in the biomedical field, from small biotechs. Neither of them tend to have enough expertise to run good trials. Second, harmonising trials (outcomes, measurements, inclusion/exclusion criteria) will make them more comparable.

To sum up, while firms have private information and expertise that usually means it’s good for them to be drivers of innovation, they (1) they may be less efficient at testing their innovations (insufficient know-how or lack of coordination), (2) have incentives against producing most publicly useful information (e.g. comparison with other technologies, dosing).

(5) Why now is the right time to support joint design

The idea of using innovative trial designs by public funders to address public good questions is not new. At start I mentioned RECOVERY, which in essence was a platform trial for repurposing treatments in a pandemic. Another trial embedded within the UK’s NHS, STAMPEDE, is a multi-stage, multi-arm trial that has been evaluating treatments for prostate cancer. Both have led to changes in treatment standards in the UK. REFINE, a basket trial, has been looking at dropping intensity for many cancer immunotherapies. Publicly-funded innovative trials like these can evaluate both new and existing treatments, at scale and speed that would not be possible otherwise.

The new idea, one that I hope emerges in the coming years, is for market design innovation and trial design innovation to work together, or even be considered as a single problem. To put it more technically, embedded in each problem of how to optimally fund some innovation there is also an implicit optimal approach to testing if it really works. In this note I argued that this optimal approach is sometimes (but not always!) quite distant from the current practice of largely leaving the testing to producers.

I think the case for doing this has always been compelling, but this moment may be uniquely good in this historic moment. As I mentioned at the beginning of this note, I think the value of testing is going to increase, possibly very steeply, if AI is able to deliver many new candidate innovations. Therefore any improvements to evaluations will become increasingly valuable too.

If AI produces a lot of candidate innovations, our approach overall to testing may have to change wholesale, since we will increasingly be dealing with a large volume of candidates—all of which we would like to ideally compare head-to-head. That future may look increasingly like the case of checkpoint inhibitor therapies for cancer of 2010s. After the first of these drugs showed large benefits, many companies quickly developed new treatments—similar enough that they sometimes got called “me too” therapies. Each of these drugs came with hugely expensive trials, often fine-tuned to show narrow and marginal claims of advantage over competitors. They all looked great, but at the end of the day clinicians and payers did not know which one was needed in which situation (and dose, and combination).

The second point is that market shaping is a very natural fit for huge—and hugely ambitious—efforts that the third wave of philanthropic funding will want to fund. With very large funding we should be thinking not only about ambitious innovations, but also about unlocking a centralised systems of testing. With additional funding much larger and more complex trials of public goods will become much more feasible. The sooner we start working on how to design them, the sooner we can reap their benefits.

Thanks to MSA colleagues Leah Rosenzweig, Siddh Haria, and Sarrin Chethik, as well as Colin Aitken, Saloni Dattani, Adam Kroetsch, Arthur Baker, Michael Kremer, and Adam Howes for LOTS of feedback and some great conversations about this.

Most people didn’t believe it was possible to develop vaccines in the timeframe needed for them to be useful. Some looked at past vaccine timelines, which had averaged roughly 8 to 12 years, and thought this one would be similar. Others thought that, even though it was an emergency situation, it would still take at least a year and a half, or two years, or maybe even four.

I had a different conclusion. In a piece I wrote in the summer of 2020, I explained why I believed that vaccines would most likely arrive within a year of the beginning of the pandemic (placing a 58% probability on enough doses for 25 million Americans being approved and available between October 2020 and March 2021, with my central estimate landing around February 2021).

We now know how the timeline panned out and my forecast was, if anything, slightly too pessimistic, since vaccines first became available three months earlier, in December 2020. In this post, I want to go into more detail and take a look back at what happened. Why were Covid vaccine trials so fast?

Subscribe now

The scientific foundation

One of the most surprising things about Covid vaccines is how many of them there are: mRNA vaccines (Pfizer and Moderna), viral vector vaccines (Oxford-AstraZeneca and Johnson & Johnson), protein subunit vaccines (Novavax), inactivated whole virus vaccines (Sinovac and Sinopharm), and others. Although their protection dropped as the virus evolved into new strains, I take this bounty of options as a result of the coronavirus being relatively easy to develop vaccines for.

Part of the reason is the disease itself. With some infections, like HIV, no one clears the virus naturally or develops lasting immunity to it, so it’s hard to know what a vaccine should mimic. Covid-19 was very different: it was evident early on that most people recovered and developed antibodies that could neutralize the virus. This suggested it was possible to prime the immune system with a vaccine.

Another reason was prior research into coronaviruses. Work on SARS and MERS from earlier coronavirus outbreaks meant scientists already understood some of their features. They had identified the key antigen the immune system reacted against – the spike protein – and had animal models and laboratory assays ready to go. Some vaccines against animal coronaviruses had already been developed. Candidate vaccines for the earlier SARS virus had also been developed for humans (and shelved after the 2003 SARS epidemic was contained). When SARS-CoV-2 arrived, lots of this groundwork could be picked up.

There’s another feature that explains why some vaccines ended up highly effective: a stabilized spike protein. The spike protein gives the coronavirus its crown-like appearance, and is the main ingredient in most Covid vaccines. But interestingly, it changes shape during an infection. Before it fuses with a cell, it sits in a compact “pre-fusion” form (on the left, below); afterwards, it springs into an elongated “post-fusion” form (on the right). The former boosts immunity most because it’s what antibodies generally encounter, before the virus has entered cells. But when the spike protein is isolated for a vaccine, it tends to collapse into the later, post-fusion form.

Over the past decade, advances in structural biology, especially in cryo-electron microscopy, have helped scientists see the pre-fusion shape of coronaviruses directly. With that knowledge, they introduced a few mutations to stabilize the spike protein in its pre-fusion form, better for the immune system to recognize. This is a big reason why vaccines with the stabilized pre-fusion protein (including the mRNA vaccines and Novavax) generated much stronger neutralizing antibody responses and were likely much more effective than vaccines without the stabilized form (including the AstraZeneca and Sputnik V vaccines).

All of these features meant the chances of a vaccine succeeding were somewhat high. They’re also why they could be designed very fast – of course, with a lot of work on the part of scientists. It took two days to design the Moderna vaccine, which then spent 63 days in preclinical testing before entering clinical trials, versus a weekend and about 100 days for the Oxford/AstraZeneca vaccine. But the vaccines would still need to be tested in clinical trials and gain regulatory approval afterwards, and this process usually takes several years.

Parallel trials

The timeline for vaccine development typically goes like this. A candidate vaccine is first designed and tested in the lab and in animals, as ‘preclinical testing’, which takes around 1.5 years on average. Next, in ‘phase 1 trials’, it’s tested in a small number of participants for basic data on safety and the immune response, taking around 2.5 years. Phase 2 trials expand this to hundreds of participants to refine the dose and gather more data on immunogenicity and safety, and take around 3 years. Then phase 3 trials test efficacy and safety in hundreds or thousands of people, and take around 2.5 years. After all these stages, all the data is submitted to the regulator as a package, which reviews it, usually taking another year.

Historically, successful vaccines have taken around 10.7 years to make it through the whole pipeline. Each phase happens sequentially and functions as a ‘go / no-go’ system, where candidates are dropped if they flop at a stage.

Covid vaccine trials took place quite differently. Rather than running sequentially, the phases ran in parallel, as you can see in the diagram below. For example, phase 1 and 2 trials were combined, collecting the relevant data for each at once, or phase 2 and 3 were combined in the same way. Ultimately, the same amount of data was collected in a shorter timeframe, by overlapping the phases rather than doing them one by one.

This was a major reason it was possible to condense the whole timeline so much. But even then, the trials still ran much faster than usual. For example, Pfizer’s combined phase 2-3 trial then took about 4.5 months, but it usually takes an average around 2.5 years for phase 3 trials alone. What made that possible?

Subscribe now

Easier recruitment

One of the biggest delays in clinical trials is the process of recruiting enough participants to run them at all. Only half of trials meet their original target for recruitment, and a fifth revise their target downward. But the Covid vaccine trials recruited huge numbers, fast. The phase 3 trials for Moderna and Pfizer enrolled around 30,000 and 44,000 participants respectively, making them some of the largest clinical trials in history.

I think there were a couple of reasons this was easier than usual. One was that people were simply more willing to volunteer. It was a pandemic after all, and everyone’s attention was on it, plus, many people were stuck at home without much to do anyway. Interest in volunteering for trials was very high. Dr Jim Kublin, who helped run a clinical trial network for Covid, recalled: “My colleagues were enrolling a phase one study for the Moderna vaccine, and they had like 10,000 people sign up on their website. They needed 40 people to enroll.”

The infrastructure to recruit participants was also more streamlined than usual. The National Institute of Allergy and Infectious Diseases (NIAID) created a clinical trial network with a shared volunteer registry called COVID-19 Prevention Network (CoVPN), from four existing NIAID trial networks, to supplement recruitment into trials. Volunteers could sign up with a short screening questionnaire to the network as a whole and be routed to whichever trial fit. The registry helped support recruitment for the late-stage trials of Moderna, AstraZeneca, Janssen, and Novavax, and by autumn 2020, almost half a million Americans had signed up through it to take part in Covid vaccine trials. Meanwhile in the UK, a similar shared volunteer registry recruited participants through the NHS. Over half a million volunteers signed up in total, feeding into trials including those for Novavax and Oxford/AstraZeneca.

Covid was Everywhere All At Once

It also took less time to test new drugs and vaccines because the disease was spreading rapidly. It’s challenging to run trials fast when diseases are rare or slow to develop. Taking a simple example, imagine a trial where 100 people are given a placebo and 100 people given a vaccine. If only 1 or 2% of them (so 1 or 2 people) caught Covid during the trial as the base rate, it would be very hard to spot a reduction from that already-low number. Even if vaccines reduced infections by half, you’d hardly be able to tell in this example, not with much confidence anyway. Without a much larger sample size, a difference between the two groups could simply reflect noise.

This reflects the ‘statistical power’ of the trial, and that it’s easier to figure out whether a vaccine is protective if new cases are arising quickly (i.e. the disease has a high incidence rate) or if the sample size is very large. You can see this in the model I made below. It’s also why it’s much harder to develop new vaccines for diseases we’ve reduced massively: it would be a struggle to test new polio vaccines in rich countries that have eliminated the virus, for this reason.

Fortunately for trial statisticians1 – but unfortunately for everyone else – Covid was spreading quickly. During Pfizer’s trial, the monthly incidence rate (the % of the population infected for the first time, per month) was around 0.5%. By September 2020, using seroprevalence data, the CDC estimated that up to 23% of people had already been infected in the hardest-hit areas, and just below 10% across most of the US.2 In this situation, what matters most is how fast new cases are accumulating while a trial is running: during the autumn and winter of 2020, outbreaks were developing fast, and cases were racking up quickly in the placebo group, which made it easier to see a reduction in the vaccine group. Within a few months, Moderna’s trial recorded 185 Covid cases in the placebo group versus 11 in the vaccinated group, and Pfizer’s trial recorded 162 versus 8.

Subscribe now

Emergency Use Authorization

People often point to the ‘Emergency Use Authorization’ pathway as the main reason Covid vaccines were introduced so fast – that they weren’t approved the regular way, and faced lower standards of evidence than usual. Let’s dig into this a little.

In general, an emergency use authorization (EUA) is a conditional authorization that lets the FDA allow a product to be used during a declared public health emergency before it has gone through full approval, and which can be revised or revoked at any time. And the bar is intentionally low: the FDA only has to find that the product ‘may be effective’ and that its known and potential benefits outweigh its known risks, given that there is no alternative that’s approved.

The standard of a product that ‘may be effective’ is much weaker than the evidence of effectiveness required for full approval. One example was the emergency authorization of hydroxychloroquine as a treatment for Covid in March 2020, which relied on observational data rather than efficacy trials. A few months later, when the RECOVERY trial showed, in a randomized study, that it had no benefit, its authorization was pulled.

Does this mean that the standards were so relaxed for vaccines? Not really. The vaccines faced a stricter EUA, called ‘EUA plus’. In its June 2020 guidance, the FDA demanded large randomized controlled trials demonstrating an efficacy of at least 50% before a vaccine could be licensed. Then in October, it added more conditions an EUA would have to meet: a median of at least two months of safety follow-up after the final dose, and review by an independent advisory committee in a public meeting.

This meant that the standards for Covid vaccines were similar to a full approval, with two notable exceptions. 1) While full approval required a complete manufacturing dossier, with validation and facility inspections, the EUA accepted a less complete package, and deferred the validation and inspection until later. 2) While full approval required a longer safety and durability record, typically at least six months, the EUA accepted a two-month median follow up time after the final dose.

It’s worth noting that the vaccines did fulfill the requirements of a full approval later on, starting with Pfizer’s vaccine in August 2021.

Incidentally, the EUA’s shorter follow-up requirement was the subject of political controversy. When Trump complained that the vaccine was being delayed until after the election, it was because of the follow-up requirement, which wouldn’t be complete until later – Pfizer only reached the median two months of safety follow-up in the third week of November, over two weeks after the election, and filed its EUA request on November 20.

I don’t have strong views on the requirement, but it seems hard to strike a balance between collecting enough safety data to build public trust versus speed up the roll-out, and unlike all the other reasons for acceleration, this was the only way in which the safety threshold was laxer than a full approval, which was achieved later on. It seems like the crux is how important public trust would be in increasing vaccination rates and whether the requirement achieved that.3

Rolling regulatory review

Typically, when researchers finish their trials, they compile a complete dossier with lots of information about the trial, the product, the manufacturing process, and so on, and submit it to the regulator, which then reviews the complete package.

It usually takes a while to review the complete package. In the US, the FDA’s goal is to review the data and make a decision within 10 months of receiving the application, or 6 months if it qualifies for ‘priority review; in practice it takes an average of eleven months. Meanwhile in the EU, the European Medicines Agency’s goal is 210 days (roughly 7 months), but the clock only includes the days the agency is actively reviewing; it pauses whenever the regulator sends questions back to the company and waits for answers, so in reality, a review usually stretches well beyond a year.

The Covid vaccines faced a different process. In Europe, they faced a ‘rolling review’ process instead, where regulators assessed the data – on the manufacturing, preclinical research, trial results, etc. – as it arrived, instead of all together right at the end. So by the time the final data was in, most of the review had already been done, and it took much less time to reach a decision. Pfizer submitted its formal EU application on 30 November 2020 and received conditional authorization about three weeks later, on 21 December, because the rolling review had already evaluated the data as it came in.

In the US, the rolling review process applied to the full approval, rather than the initial authorization, since the vaccines went through the Emergency Use Authorization pathway instead, which began on 20th November for Pfizer and granted approval on 11th December 2020. The complete application, which did involve rolling review, was completed in May 2021 and received full approval on 23 August 2021, roughly three to four months, still much shorter than usual.

Subscribe now

Can we apply some of these processes to speed up other drugs and vaccines?

By this point, you’re probably wondering why the same processes couldn’t apply to many other drugs and vaccines. I’ve wondered the same thing, and I think it’s true that various parts of this process could apply elsewhere too – such as clinical trial networks to recruit participants, more fundamental research into pathogens and candidate vaccines, and better trial design. (The Covid experience is a reason I became interested in clinical trial reform more broadly.)

But some parts would be challenging to scale up more widely. Rolling regulatory review, for example, is more demanding on the part of regulators. In the usual pipeline, most drugs or vaccines fail before they reach the finish line and are submitted for review, which means regulators are saved the time of reviewing lots of applications that would have been withdrawn anyway. (You could, of course, expand rolling reviews by hiring and training more regulatory staff, and reduce the trade-off!)

Parallelized trials are also challenging. When trials are run sequentially – with phase one, phase two, and then phase three – the sequence functions as a ‘go / no-go’ system, where developers can drop candidates they think are unlikely to succeed at the next stage. It saves time to run trials in parallel, but it also raises financial risk: instead of spending money on the next phase only once the previous one has paid off, a vaccine developer would have to commit to spending it up front.4 A similar problem applies to building production capacity during the trials, since that capacity would be wasted if the vaccine doesn’t work.

This would have been especially financially risky during a pandemic, because of the degree of uncertainty at the outset. What if the virus had actually been contained by the time the trials were complete, as was the case during the 2003 SARS outbreak? Or what if the fundamentals of the virus were misunderstood and the vaccines were designed incorrectly? By the time such large trials were complete, a lot of money might have been sunk into trials that would ultimately fail. This is where Operation Warp Speed comes in: in my view, it made it possible for a lot of the processes above to function at a lower financial risk, and in a more coordinated way.

Operation Warp Speed and its analogs

Operation Warp Speed was launched in May 2020 by the Trump administration, as a partnership between the Department of Health and Human Services and the Department of Defense. Its goal was to produce and deliver 300 million doses of safe and effective Covid vaccines, with the first doses becoming available by January 2021.

To make it happen, Operation Warp Speed funded the development of a range of candidates and their clinical trials, including the phase 3 trials of Moderna, Janssen, AstraZeneca, and Novavax.5 The CoVPN trial network was developed as part of Operation Warp Speed. Operation Warp Speed also paid for companies to start manufacturing their vaccines at industrial scale while trials were still running, so that millions of doses would be ready to ship once vaccines were authorized.

Importantly, it also committed in advance to buying hundreds of millions of doses of a vaccine that succeeded, meaning that the companies wouldn’t face a massive loss if the outbreak ended up petering out before vaccines were ready. This idea, called an advance market commitment, was originally developed by economists Michael Kremer and Rachel Glennerster to successfully speed up new pneumococcal vaccines in the 2000s, before it was adapted for the pandemic. As then-secretary of Health and Human Services, Alex Azar, described it, the government would invest more than 10 billion dollars to “de-risk vaccine companies’ development efforts and manufacture product in advance.” By the end of the year, it had committed around 13 billion dollars.

As I described earlier, running trials in parallel and manufacturing before trials are complete makes things faster but is financially risky, because a company commits money up front that would be wasted if the candidate vaccine failed. Operation Warp Speed absorbed much of this risk. (One thing that Operation Warp Speed didn’t do was change the regulatory bar: the EUA and rolling review procedures were set by the FDA separately.)

Other countries also used analogs to Operation Warp Speed to accelerate vaccine development and secure doses for their populations. The UK, for example, ran a Vaccine Taskforce, set up in spring 2020 and led by Kate Bingham, which also backed a spread of candidates across different technologies, made advance purchase agreements, and funded manufacturing. The EU ran a Vaccines Strategy, launched in June 2020, which used advance purchase agreements as well: the EU Commission financed part of companies’ upfront costs in return for the right to buy doses. Since it negotiated for all 27 member states, it had a lot of bargaining power, but it started much later; its first major agreement was signed months after the UK and US. And lastly, COVAX created a pooled procurement mechanism for countries around the world, aimed to secure doses for lower-income countries that couldn’t make their own deals (though they fell below their target of later delivering vaccines, delivering half of their target of 2 billion doses by the end of 2021 as high income countries struck bilateral deals quicker).

Operation Warp Speed-type efforts probably work best for large problems that need coordination across many actors, where the goal is well-defined and commercial markets won’t deliver it at the speed required, because the private returns are too small relative to the social value. But it’s also possible to work faster across the board, by creating clinical trial networks and shared volunteer registries to speed up recruitment, innovative trial designs that collect data more efficiently, investment into early-stage research, more regulatory capacity to review evidence as it arrives, and many other ideas we’ve shared on this blog.

Subscribe now

Faster trials saved millions of lives

The speed of vaccine development during the pandemic was unprecedented, but it wasn’t unforeseeable. It came from a coordinated effort that involved dedicated research funding, clinical trial networks, parallel trial phases, rolling regulatory review, and parts of the review being deferred rather than skipped. What eased the process were decisions to de-risk the process financially, with upfront funding and advance commitments to buy the vaccines if they worked.

The result was extraordinary. Millions of deaths globally were averted by Covid vaccines in just the first year of vaccination alone.6 We take it for granted today, but it was not a given that trials would be completed within a year. Nor was it a given that the vaccines would have around 95% efficacy against the original strain of the coronavirus, which was much higher than most scientists anticipated: the median forecast was 50%. Speeding up the process saved millions of people.

When it comes to clinical trials, it’s easy to assume the process is so bureaucratic that we couldn’t move quickly even if we wanted to. The pandemic shows that is not the case. When the need was obvious, thousands of people around the world – including scientists, operators, policymakers, regulators, economists and government officials – worked differently to make sure vaccines became available faster.

It also reveals how slow the current system is. If clinical trials can be sped up so much without compromising on safety, we should find ways to reform them outside of pandemics as well. Every year of delays in testing is a year patients go without potentially lifesaving treatment, risking their lives. We shouldn’t be dragging our feet for the sake of following the status quo. Rather than treating pandemic trials as exceptional, we should ask whether that speed should have been the norm all along.

HHS announced “Operation TrialBlazer”: a package of reforms aimed at making the US clinical trial system faster, cheaper, and more globally competitive. The announced reforms are broad and cross-cutting but focused heavily on early-phase trials, an area where the US is rapidly losing ground to other countries that have made it easier to advance promising therapies.

Others have already started to weigh in. Ruxandra shared some thoughts yesterday, calling the new HHS initiative a “welcome move.” Alex Tabarrok was more skeptical. While he was supportive of some the reforms, he also argued that the government missed an opportunity to follow Australia’s approach to phase I trials. As Ruxandra has discussed on this blog, Australia allows most phase I trials to proceed without any prospective government review under the supervision of an ethics committee. That approach has made early-phase trials in Australia faster and cheaper—and it has helped attract trials away from the US.

I share Alex’s desire for further reform. But I think HHS deserves more credit here. The officials who developed this plan appeared to be doing two things at once: improving the current clinical trials system, and laying the groundwork for more transformative reforms. The most ambitious ideas will require Congress to step in, but given the constraints they faced, HHS has taken some meaningful steps towards reform through this initiative.

Reforming the current system

First, HHS is taking several steps to reform the current approach to trial oversight, including new FDA policies and new resources for researchers running early phase trials. Among the most important new initiatives is the FDA’s effort to provide greater clarity around its phase I drug manufacturing requirements.

While these sorts of reforms could be considered incremental, they are critically important. As Ruxandra noted in her overview, onerous manufacturing “requirements” are one of the biggest barriers to starting Phase I trials; and are probably a major reason we are losing trials to other countries like China and Australia. The reason these requirements pose such a great challenge for researchers—and the reason I put “requirements” in scare quotes above— is because there is little agreement on what FDA’s phase I manufacturing requirements actually are.

Researchers and drug companies believe that the FDA requirements are onerous. And they act on that belief: For example, they often go out of their way to document a full commercial manufacturing process, even for products that are still in early phase trials. Yet the FDA itself has suggested that its regulations are more flexible than that, and that early phase trials don’t actually require this level of documentation.

A stark example of this disconnect appeared just last week, when the Reagan-Udall Foundation issued a report proposing reforms to FDA’s oversight of early phase trials. The report, which was informed by industry experts, suggested that FDA’s “stability requirements for study material often far exceed study duration”. For example, a company might have to prove that its investigational drug is shelf-stable for six months, even if the trial only lasts for a few weeks. If that were truly required, that would be a clear example of an excessively onerous FDA requirement. But FDA’s own regulations appear to state otherwise: they only require companies to submit “information sufficient to assure the product’s stability during the planned clinical studies.”

This kind of confusion is common in drug development. Drug makers have a tendency to be excessively risk averse and “overcomply” with perceived FDA regulations. FDA, for its part, has a tendency to ask for more data than necessary. The incentives push companies towards excessive documentation. The new clarifications help lessen the regulatory burden by laying out—for both industry and for FDA reviewers—what the phase I manufacturing regulations actually require. Even if the clarifications seem minor, they could make a big difference in reducing unnecessary documentation in early phase trials.

Setting the stage for transformative change

Clarifications are helpful, but if we truly want to see the US compete with other countries like China and Australia, we will need to explore different regulatory models altogether. To that end, I was excited to see HHS announce a new pilot program that may help lay the groundwork for a more ambitious approach.

The pilot is designed to improve the speed and quality of IND submissions by partnering clinical trial sponsors with “qualified research institutions”. These institutions, which could be academic medical centers or contract research organizations, would provide expert consultation to sponsors developing their IND packages.

The pilot is designed to achieve two goals: First, by giving sponsors access to expert institutions, it should help them produce higher-quality IND submissions that are less likely to result in clinical holds. Second, the pilot could speed up IND review by allowing sponsors who work with these institutions to submit components of their IND application to FDA on a rolling basis. Along those lines, the pilot will also explore ways for trial initiation steps—including IND preparation, site contracting, and IRB review—to proceed in parallel. The hope is that by the time of the formal IND submission, FDA’s review will be substantially complete, ethics review and contracting will already be underway, and the study can begin much faster. This could shave months off of IND development times.

On its face, the pilot represents and exciting new approach to clinical trial reform. But it also may be laying the groundwork for something bigger: the eventual creation of an Australian-style clinical trial notification system in the US. Australia’s system depends on the existence of a set of independent committees called HRECs who possess the expertise to determine whether a trial may proceed. By creating a class of qualified research institutions, FDA is also building the institutional capacity that an Australian-style notification-based system would require if it were to be brought to the US.

To be clear, it’s not obvious that this is HHS’ intent. And even if many in the administration wanted to move towards an Australian-style model, they would have good political and legal reasons not to say so explicitly in this roadmap—it’s a move that would likely require Congress. But if I wanted to move the US towards an Australian-style clinical trial notification scheme, this is exactly how I’d go about doing it.

Modernizing Trial Infrastructure

Too often, trial reform efforts focus exclusively on FDA. But in addition to regulatory reforms, the US also needs to build national-level infrastructure that makes trials easier to run—things like standard approaches to contracting, centralized IRB review, and the ability to leverage electronic health records to run trials. Countries like China are making major investments in their national clinical trials infrastructure, and the US will need to do the same if it wishes to keep pace.

That’s why I was pleased to see that the roadmap calls on other HHS agencies to help build that infrastructure: NIH anticipates leveraging its position as a major funder of early-phase research to push for greater efficiency in research. ONC, which oversees health IT, will support technology standards for trial matching and digital trial protocols, streamlining trial operations and patient recruitment. HHS is also exploring regulatory reforms that could remove barriers to patient compensation. In my own writing, I’ve advocated for much greater involvement from NIH and ONC in building trial infrastructure, and the work outlined in this roadmap is a promising start.

The bigger reform agenda

Operation TrialBlazer is a strong start. But if we wish to truly reform clinical trials in the United States, there is much more to be done. I hope that we continue to explore an Australian-style notification model for early phase trials. I’d also like to see even greater investment in trials infrastructure, and a broader effort to improve the efficiency of our large late-phase trials, which are the largest driver of overall clinical trial costs. And while these administrative actions are helpful, many of the most ambitious reforms will require legislation from Congress.

Still, for those of us who have been advocating for clinical trial abundance, yesterday’s announcements from HHS were an important step. They are a sign that the administration is coordinated, is taking the challenge seriously, and is setting the stage for further reforms to come. That’s great news for clinical trial abundance.

In a call with journalists, Acting FDA Commissioner Kyle Diamantas and NIH Director Jay Bhattacharya outlined a series of reforms designed to hasten Phase I trials and encourage companies to conduct studies in the US. The FDA’s efforts are intended to reduce Phase I timelines by six to twelve months. They include an expedited investigational new drug pilot program, updated guidance on the use of advanced computer models for dose selection, and updated frameworks for clinical trial master protocols.

I am particularly excited about the focus on clarifying what is required for a first-in-human Phase I study with respect to chemistry and manufacturing controls (CMC). Excessive CMC requirements are some of the most onerous and unnecessary barriers to clinical trial starts. In conversations with leading oncologists working on cell therapy, they have repeatedly told me they could test as many as 10x more cell therapies with minimal increase in risk (and in the process treat some patients), were it not for the requirement to do a “full-GMP” process.

It is extremely encouraging to see these announcements from the administration and to see this topic being taken seriously, after decades during which clinical trial reform remained at the bottom of policy priorities. Biotech policy in the United States is moving in a direction I have long supported and argued for, particularly in the case of accelerating Phase I trials. The proximate cause for these announcements appears to be the growing recognition of competition from China: writing in Fox News today, HHS Secretary Robert F. Kennedy Jr. calls on the US to win the race for medical innovation, citing China’s surpassing of America in early-stage clinical trials. Kennedy has previously told Congress that China is “eating our lunch” on new drug approvals and clinical trial starts.

This news today coincides with another announcement, an announcement that may perhaps serve to illustrate why the administration is so concerned about China. For the first time in history, a CAR-T cell therapy has been approved to treat a solid tumour. Cell therapies have transformed blood cancers for almost a decade now, but solid cancers, which comprise around 90% of all tumours, have remained out of reach. That changed today: CARSgen Therapeutics announced that China’s National Medical Products Administration has approved satricabtagene autoleucel for the treatment of patients with a specific molecular subtype of advanced gastric and gastroesophageal adenocarcinoma who have failed at least two prior lines of therapy

What is notable about this development, aside from the scientific achievement, is that this therapy was developed by a Chinese company and approved by a Chinese regulator, with no appearance at all from the United States. I have written before about the rise of China in biotech, using the example of another CAR-T therapy, Carvytki. Carvytki was first tested in humans almost a decade ago by a biotech company from China called Legend Biotech. After impressive Phase I results, an American large pharma company licensed the asset from Legend and carried Carvytki to approval, beating the American-originated therapy Abecma in Phase 3 trial results.

The Carvykti case should have been an early warning. Already by 2016, the same year Legend began the clinical trial that would first reveal Carvykti’s potential, China had overtaken the United States in the number of cell-therapy clinical trials. The gap has been growing since. Indeed, a decade later, we see another first: this time, it is not only the early stages, but the whole pipeline, up to approval, that has moved to China. This progression shows something that many of us have been warning for a while: being able to do fast iterations in humans through efficient Phase I trials is key to biotech advancement.

As I wrote at the beginning of this month:

The most valuable thing early-stage trials enable is iteration. They allow tight feedback between the clinic and the lab. There are countless ways to engineer a better CAR-T cell, and many cannot be evaluated in the laboratory alone. No cell culture or animal model fully replicates the complexity of a human tumor, and AI is unlikely to close that gap anytime soon. We simply lack the training data to capture what tumors are actually like in vivo: their geometry, vascularization and biomechanical properties.

China’s ability to run these trials quickly and at scale gave it a structural advantage in that learning process, whereas the US is currently undermining itself through burdensome manufacturing requirements and regulatory bureaucracy that make early experimentation slower and more costly than it needs to be. The policy world has, belatedly, taken notice of this. A proposal in the President’s 2027 FDA budget would streamline early-stage trials. This a welcome recognition that regulatory friction on early experimentation is a competitive liability. But a budget proposal is not a policy and will remain aspirational unless Congress acts.

For now, American and European pharmaceutical companies largely retain the upper hand in the later stages of clinical development, as shown by the fact that Legend ultimately licensed Carvytki to a large American biopharmaceutical company to get it approved. But the pipeline that feeds those late-stage trials is increasingly Chinese. Such early-stage dominance turned into vertical integration of the entire chain in solar panels, batteries and electric vehicles, and LCD panels. The question is how long Western companies can sustain their advantage at the later stages, when the discoveries that make those stages possible are increasingly being made elsewhere.

The announcement of the first solid cancer therapy approved in China and China only prove the veracity of these words. And although I am optimistic due to today’s moves from the FDA and HHS, those of us excited about speeding up Phase I trials and therapeutic abundance should take these announcements as the first steps, not a done deal. Much has to happen, including legislative action on the part of Congress, in order to make more efficient early-stage trials a reality.

What I really want to bring home though is that the story of Grace Therapeutics is not unique by any means. Excessive manufacturing requirements are a pervasive problem, especially in the field of cell and gene therapy and they are killing countless of promising medicines before they ever reach patients. And they manifest at every stage: starting with Phase I, as explained here, all the way to post-approval.

In this post, I want to focus on a story that happened post-approval and illustrates how even when a drug basically cures cancer it can be very hard to commercialize and turn into a profitable venture for the company making it. The drug in question is Carvykti, a CAR-T cell therapy for multiple myeloma that I wrote about at length last week with my friend Amol Punjabi, for Works in Progress. If you haven’t read that piece, the short version is this: Carvykti is, by most clinical measures, one of the most remarkable cancer therapies ever developed. It is a one-time infusion that has produced durable, long-term remissions, and in some cases what looks like an outright cure, in patients for whom every other treatment had already failed. A third of patients in its pivotal trial remained disease-free five years later.

And yet, despite all of that, the company behind it nearly had its commercial launch derailed. This, I think, is a very good case study on why the cell and gene therapy field is in trouble and how much of it comes down to excessive manufacturing requirements. I should also warn at this point that much of the information below is not public. Instead, it was assembled from interviews with experts and paywalled industry reports and investor calls.

How the Carvytki launch almost got derailed by manufacturing requirements

To understand how that derailment happened, it helps to first appreciate just how difficult these therapies are to make. CAR-T therapies are among the most complex products ever commercialised. Almost all approved CAR-T therapies are autologous, meaning cells must be extracted from each individual patient, genetically engineered, and returned to that same person, making the manufacturing process exceedingly difficult. Johnson & Johnson (J&J), the large biopharmaceutical company that co-developed Carvytki, faced additional hurdles in securing sufficient quantities of materials to manufacture the cell therapies, compounding an already complex situation.

But perhaps the most surprising element hampering the therapy’s launch was a surprisingly high out-of-specification (OOS) manufacturing rate, estimated at 30 to 40% in the United States. Before any CAR-T product can be infused, it must pass a battery of quality-control tests. One of the most important is the potency assay, an in vitro experiment in which the patient’s re-engineered T-cells are placed in contact with myeloma cells, and their killing ability is measured. This test is meant to confirm that the therapy is genuinely therapeutic. A batch that falls short on any single measure is classified as out-of-specification, and usually cannot be administered.

These rates were far higher than anything observed during the clinical trials, for several reasons. In a real-world setting, patient heterogeneity is considerably greater than in the controlled environment of a trial, where eligibility criteria are designed to select for patients most likely to respond well and manufacture cleanly. At launch, authorised treatment centres were immediately overwhelmed with advanced, heavily pretreated patients and who in many cases would not have met the original trial’s inclusion criteria.

But the strangest contributing factor was one that originated not in the clinic or the manufacturing facility, but with the FDA. Somewhere between the trial phase and commercial approval, the FDA had tightened its potency assay thresholds. Legend Biotech’s leadership later calculated that if those stricter commercial standards had been applied retroactively to the original trial data, the out-of-specification rate would have risen from 2 percent to 18 percent. The therapy that had produced some of the most striking results in the history of myeloma treatment would, under the FDA’s new benchmarks, have looked like a manufacturing catastrophe.

Some industry observers were quick to point out the problems with this. The potency assay, first of all, is not a safety test. A batch that fails does not pose a risk to the patient, but falls short of an administratively defined threshold of activity. More importantly, the two parameters that most frequently triggered out-of-specification failures were traced not to genuine defects in the product but to minor technical artefacts in the assay itself, so noise in the measurement, as opposed to evidence of a non-functional therapy. Actual manufacturing failures, in which the product was biologically compromised or unusable, were exceedingly rare. This was, in the view of most people close to the field, a problem of regulatory stringency rather than one of therapeutic efficacy.

The human cost of this could have been significant. Imagine a 67-year-old with relapsed myeloma who has already been through five prior lines of treatment. While waiting for Carvykti, she is placed on bridging chemotherapy, a holding pattern designed to stop her disease from advancing while her cells are being processed. And, after weeks of deterioration and waiting, she is told that her batch has failed the potency assay by a few percentage points and sent home.

This was the fate faced by roughly a third of scheduled patients. To avoid such an outcome, rather than discard the failing batches, J&J took another approach. It routed the out of specification batches through expanded access, a regulatory pathway that allows patients to receive technically non-approvable products outside the formal trial structure. Companies cannot charge for therapies given through this route, meaning every out-of-specification batch was administered free of charge. In exchange, J&J collected rigorous outcomes data and compared results between patients who had received passing and failing product. What they found was there was no meaningful difference between the two groups.

Armed with that evidence, J&J returned to the FDA. When seeking approval to use Carvykti in earlier lines of treatment, the company combined the expanded access outcomes data with results from CARTITUDE-4, a randomised Phase III trial of 419 patients, and constructed a detailed analysis mapping manufacturing specifications directly to clinical outcomes. The conclusion was that thresholds had indeed been set too high. At this point, one might conclude that everything worked out, as patients received their therapy eventually through the expanded access program are measuring success too narrowly. But events of this kind are deeply damaging to a company’s profits, which reduces investor confidence and makes future development less attractive.

This would be less of a problem if it was isolated. But the Carvykti manufacturing saga is just one example of a failure that is, in fact, routine, especially in the field of cell and gene therapy. Manufacturing remains one of the largest hurdles in testing and bringing a cell or gene therapy to market. A 2022 retrospective study of 90 cell and gene therapy programmes entering Phase III found that 43 percent experienced major disruptions, and that 60 percent of those disruptions were traceable to CMC issues: potency assay failures, process changes, supply shortfalls. A single year of delay can cost a gene therapy developer between $100 million and $500 million in lost revenue and ongoing overhead.

Part of the problem is excessive regulatory conservatism, as the Carvykti potency thresholds illustrate. But another, less appreciated factor, is secrecy. Bespoke manufacturing processes confer genuine competitive advantage, and companies have little incentive to share hard-won lessons with rivals or with the contract manufacturing organisations that serve the broader industry.

Secrecy also applies to regulatory decisions and compounds the problem. Because the FDA rarely makes its earlier judgments on manufacturing specifications public, developers have little visibility into what the agency has accepted before. That gives specialist manufacturing vendors that drug companies depend on considerable leverage. Drug developers cannot easily benchmark whether a given requirement is truly necessary or whether a cheaper approach would satisfy regulators. As a result, vendors can charge for uncertainty. The result is an emergent ecosystem plagued by inconsistency and lack of transparency, where prices are difficult to challenge, and manufacturing costs remain higher than they need to be.

Compounding the problem is that assay standardisation is not especially attractive work for academia. Method harmonisation may be enormously valuable, but it rarely produces the kind of landmark paper or career-defining result that researchers are rewarded for.

Regulators themselves are aware of the problem that manufacturing poses to drug developers. In 2023, the FDA convened a high-level meeting with the Alliance for Regenerative Medicine and the American Society for Gene & Cell Therapy to focus specifically on potency assays. The white paper that followed recognised the structural dysfunction, but offered little in the way of a practical solution.

And although manufacturing sounds like a boring afterthought, the presence of such hurdles represents a major blocker for the field, as it makes an already scientifically risky field even more unattractive to pursue for investors. Carvykti is, by most clinical measures, a landmark therapy and has changed the outlook for patients with relapsed myeloma and built a credible case for moving cell therapy into earlier lines of treatment. Yet the returns to investors have not been commensurate, as can be seen through the share price of Legend Biotech, the biotech start-up that first originated this therapy.

Investors read clearly into these signals and respond accordingly. Venture funding for the cell and gene therapy sector fell from $8.2 billion across 122 deals in 2021 to $1.4 billion across 39 rounds in 2024, a collapse of 83 percent in three years. A BioCentury analysis found the trend continued to worsen into 2025. These reports quote concerns over manufacturing requirements and general high manufacturing costs as a key reason behind the general bearishness surrounding the sector.

Until manufacturing and regulatory risks become easier to underwrite, capital will keep drifting away from one of the most promising technologies of the last century. The result is a loss for everyone. Some of the most promising cancer therapies of the next decade may take longer to reach patients, if they are pursued at all.

Yet at the same time, I suspect that for the typical viewer it was hard to tell what that exciting revolutionary thing actually was. The headline made it seem simple: The FDA was piloting “Real-Time Clinical Trials… that will report endpoints and data signals to the agency in real time.” Today, FDA sees trial results only after the drug companies who run the trials, called sponsors, have collected the data, cleaned it, analyzed it, summarized it, and packaged it for review. This new approach could allow the agency to receive useful information earlier, reducing the amount of “dead time” between study phases.

But if you watched the FDA’s press conference, you might start to think that this announcement was about something else entirely. During the event, the pilot participants each shared some remarks. Most of them did not focus on real-time submission of data to FDA. Instead, they spoke about other things: a representative from Amgen talked about pragmatic trials. A researcher at UPenn talked about how the new pilot would ease the burden on their staff. Kent Thoelke, CEO of Paradigm Health, the technology developer for the pilot, spoke about the importance of bringing trials into community and rural settings. This is all great stuff. But what does it have to do with real-time submission of data to the FDA?

But there is a good reason that the pilot participants were speaking in such sweeping terms. The benefits of real-time trials could go far beyond just faster submission to the FDA. Real-time clinical trials represent a new, more efficient way of running trials that affects everybody involved—including study site researchers and patients. Beyond just reducing “dead time”, they hold the promise of making trials faster, less expensive, and more accessible to patients. In this post, I’ll explain why this program could change how trials are run, and what it will take to make it successful.

How trials got re-engineered to allow for real-time data

To understand why FDA’s pilot is potentially transformative, it’s helpful to understand how trials are run today, and how the real-time trials pilot changes that. As long-time readers of this blog already know, today’s trials are deeply inefficient and labor-intensive, largely owing to the cumbersome way in which we collect and clean trial data. Today, trial data is collected from a broad range of different “source” systems; EHRs, lab systems, and—far too often—on paper. Then, traditionally, those data are manually transcribed into an electronic data capture system (EDC): the system that is used to collate and transmit the trial data to the drug company. This manual transcription is time-consuming, error-prone, and can introduce lag between when data is first captured and when it’s submitted.

After the drug company receives the data from the EDC, it undergoes multiple layers of quality checks, which add more work and introduce even more delay. The drug company reviews the data for discrepancies and missing values, and if it finds an issue it will return a “query” to the study site. The site will then need to go back to the source documentation to resolve the issue. Then, there’s an even more labor-intensive step: A study monitor, hired by the sponsor, will fly out to the study site and review all of the data captured in the EDC, comparing it against the source data.

Meanwhile, the trial is collecting adverse events and endpoints. Each of these undergo extensive manual review. Adverse events are reviewed by the sponsor to determine whether they are serious, unexpected, and plausibly associated with the study drug. If so, they are shared with the FDA immediately. Endpoints are often reviewed by expert adjudicators, who confirm that the study endpoint was actually reached.

This is why real-time submission of data to the FDA is difficult. In the current model, each piece of data needs to be transcribed, validated, verified, and potentially adjudicated before it can be analyzed. The process can be long and drawn out. No sponsor or site would want to share “raw” data with FDA before this process was completed. This process is also a big part of the reason that we see so much dead time between phases. After the last patient visits a study site, there is still much work to be done: unaddressed queries, source data verifications, and adjudication. Only after that work is done for every data element will the sponsor sign off on a “database lock”, at which point the actual analysis of the trial data begins.

The data bottleneck begets a decision bottleneck: between study phases, drug companies must review their compiled data and decide what they want to do next. So after the database lock, the sponsor will analyze the data and make their critical go/no-go decision. They’ve probably already given some thought to how they will design their next phase, but after looking at the data they’ll confirm their design choices and share them with the FDA. They often will need to schedule a meeting with the FDA to share their proposed approach and make any modifications the FDA recommends (these interactions are especially common before phase III begins). Finally the drug company will start the long and laborious process of recruiting sites and patients for the next phase. Today, each of these steps is done sequentially, creating costly delays between phases.

How the RTCT pilot changes things

But what if we could do away with many of those steps? What if, instead of data being entered into disparate systems, the source data was entered electronically, then compiled and automatically imported into the sponsor’s electronic data capture system (or just sent directly to the sponsor)? Then, much of the manual work of data transcription, checking, and verification could be automated. And thanks to AI, we can even automate the downstream work: After the data is submitted to the sponsor and validated, AI could help classify the seriousness of adverse events and even adjudicate the endpoints. Data that previously took weeks to enter, clean, and validate can be available nearly instantly to analyze—and, of course, to share with the FDA.

While I can’t claim to know precisely what Paradigm Health’s software is doing in these pilots, I suspect it is automating many of these tasks. And those kinds of automations yield many benefits beyond just enabling real-time submission to the FDA. We heard about many of them at the press conference. First off, the approach integrates data from disparate sources, reducing manual transcription work and errors. Sites will spend far less time dealing with source data verification and time-consuming queries. This saves an enormous amount of time and effort. Source data verification alone is believed to consume 25-40% of the trial budget.

This explains why FDA’s press conference kept drifting into the topic of site burden, community access, and pragmatic trials. The same infrastructure that makes real-time clinical trials possible also makes it easier for sites that don’t traditionally do clinical trials—like community and rural sites—to involve their patients in research. With much of the work automated, participation is simpler, especially when the trials themselves are designed to make use of the data collected in routine care.

Finally, when all of the systems that collect and use trial data are integrated and automated, it opens the door for even more innovation. Rapid, automated data analysis could make it easier to run adaptive and Bayesian trials by providing quicker feedback to sponsors about when a key milestone or threshold was hit. And the infrastructure that makes rapid data analysis possible also could be extended to automate many other tedious trial execution tasks, like patient recruitment, scheduling, and follow-up.

Will This Scale?

To be clear, the idea of using AI and automations to streamline trial data collection isn’t new; companies have been working on this for years. But FDA’s involvement could have a huge impact in actually getting these approaches adopted in a very conservative and risk-averse industry.

Observers are already asking how we can scale up this approach and apply it to more trials. Sean Khozin, who originated the real-time trials idea during his tenure at FDA, posted about this in his Substack: “The hard work, the transformation, is the scale-up. It is the second sponsor, and the tenth, and the hundredth. It is the standards work that has to happen quietly behind the announcements, the EHR-to-EDC plumbing, the sponsor data systems that have to be re-architected, the reviewer workflows that have to be redesigned around streams instead of submissions.”

If you are a longtime reader of my blog, you already know why scaling innovations in trials is so difficult. The trial industry is deeply fragmented, poorly coordinated, and risk-averse. And regulatory uncertainty makes companies fearful of doing things differently. You might also know that the industry has a long and inglorious history of piloting better ways of doing things, only to go back to doing things the old way after the pilot has ended.

The limits of this particular pilot illustrate the challenges. For the pilot, they picked a relatively simple task: the trials appear to be using one software vendor in just two health systems (this might change, but I expect the final number of vendors and sites to remain small). For this approach to be truly transformative, though, it will need to be scaled up to a much larger set of trials, including the large, expensive late-phase trials. These trials are conducted across multiple sites, often in multiple countries, using software from many vendors and pulling data from multiple EHRs and hospital systems. I don’t want to understate the technical achievement that Paradigm Health pulled off in developing the system used in this pilot—it’s difficult and impressive work. But expanding this approach will require a lot more technical work, including, as Sean Khozin noted, standards and plumbing.

FDA will need to be actively involved in this scale-up. Part of the reason that we do things in the current, inefficient way is because sponsors believe that FDA expects it. The cumbersome system of audits, source-data verification, and adjudications were not an inevitable part of trials; they grew over time to meet the perceived demands of regulators. If we want to automate those tasks with AI, sponsors will need to work closely with regulators to “de-risk” this approach. Pilots alone won’t be sufficient; FDA will need to set expectations, change policies, and even change its culture to accommodate a new approach to running and reviewing trials.

What Should the FDA Do Next?

As excited as I am about the potential of this new approach to running trials, the initial rollout was not exactly confidence inspiring. The FDA didn’t really tell a coherent story about why real-time trials matter, how the agency intends to use the data, or what it will take to scale up this work. But there’s still plenty of time for the FDA to provide more clarity—and I suspect they’re busy working on addressing these questions.

With that in mind, I’d offer three recommendations for the FDA:

First, the FDA needs to define the operating model for its use of real-time trial data. If you just read the press releases and materials, you might expect FDA reviewers to be monitoring a stream of data like a manager might monitor their corporate dashboard. Perhaps FDA reviewers will wake up in the morning and watch the trial results roll in while they drink their coffee. This, to be clear, would be a terrible idea. The last thing that sponsors need is FDA staring over their shoulders as they run the trial, (over)reacting to safety signals as they come in or prematurely drawing conclusions from interim views of study endpoints.

A more prudent model might look something like the one used in FDA’s real-time oncology review program, where the availability of early readouts allows FDA to gain familiarity with the trial outcomes earlier and therefore issue decisions more quickly. For example, FDA might receive a draft readout of the trial results the very day of the last patient visit before the trial’s “database lock”. This would give FDA a better sense of what to expect when the sponsor comes in later with its more polished data package.

But that is just one model of many the FDA should consider. Along with the real-time trials announcement, FDA published a request for information seeking input on potential pilots of AI-enabled methods to optimize early-phase trials.1 This could be a great opportunity to explore how these newer operating models might work in practice and anchor them in established principles for trial design and statistical analysis.

Second, FDA needs to help build the standards and infrastructure to scale this approach beyond just one vendor and a few trial sites. Today, this work is only possible because one company, Paradigm, worked closely with sites to integrate their disparate data sources. That is useful for a proof of concept, but it is not a scalable model for the broader clinical trials enterprise. Large late-phase trials (and even many early phase trials) are conducted across many sites, often in multiple countries, using multiple EHRs, EDCs, labs, imaging vendors, clinical outcome assessment tools, and monitoring platforms. For real-time trials to work, they’ll need to operate on common standards that all of these disparate systems can use.

FDA could play a role in setting data standards for real-time trials. FDA should work with ONC and standards development organizations to establish standards for how clinical trial systems share and transmit clinical and operational data. These standards could both facilitate real-time submission of data and help build the infrastructure needed to make trials more efficient and automated. These standards should include the metadata needed to understand the data’s provenance, its role in the trial, and whether and how it was verified or adjudicated. This work could begin now, building on work already underway in this area by existing standards development organizations.

Third, FDA should set clear expectations for sponsors when it comes to data collection, verification, and validation. The pilot shows that we can use technology to make trials far more efficient by reducing or automating cumbersome practices like source data verification, queries, and manual adverse event review. But for this to be possible, FDA needs to clearly define the conditions under which it will accept automated or AI-assisted alternatives, and how it intends to inspect study sites that use these approaches without requiring them to maintain reams of paper documents and PDFs. Otherwise, companies will default to doing the traditional, “safe” approach to data collection, even when better approaches are possible.

FDA’s expectations-setting should go beyond ordinary guidance. As readers of this blog know, FDA has tried in the past to issue guidance offering alternatives to practices like 100% source data verification, only to find that the guidance was not specific enough to change industry practice. I’d suggest FDA get more specific, identifying standards, practices, and data submission formats that it would deem acceptable for purposes of real-time submission and review.

If the FDA can address these issues, real-time clinical trials could go beyond just pilots and become the foundation for a better, more efficient way of running trials. But if FDA does not clarify its operating model, help build standards, and set regulatory expectations, this could go the way of most other trial pilots, which have failed to make any lasting impact on how the industry operates.

But there’s also an important story about how the clinical trials industry got this way. Trials used to be far less bureaucratic, and many of the practices we take for granted in trials, like monitoring and source data verification, only became standardized in recent decades. The story of how trials transformed is a case study in how regulation can reshape an industry–and it can help inform today’s debates over how trials need to change.

The story I’m about to tell is most often recounted by a group I refer to as the “trialists”: the academic leaders who helped design the most important large-scale clinical trials of the 1980s and 1990s, and who continue to influence the debate over clinical trials today. To these trialists, today’s clinical trials industry exists in a kind of fallen state. In their account, the 1980s and 1990s represented an efflorescence in the science and practice of trials: we proved we could run trials at unprecedented pace and scale— and at remarkably low cost. Trials seemed poised to answer many of our most important clinical questions. Then came a decline: in the 1990s, the trials industry underwent a transformation that left it corporatized, bureaucratized, and diminished in ambition.

Here, I’ll recount the trialists’ story. I want to explore what that story gets right, what it misses, and what it tells us about the prospects for real reform.

Subscribe now

The Rise of Large Trials

Within the story of clinical trials, there are two narratives running side-by-side. One is a story of continued scientific progress. The modern clinical trial is more than just a form of scientific experiment: it is a kind of truth-finding technology, refined and improved over decades, that has helped us unlock life-saving new treatments and improve care. Since the creation of the modern trial in the 1950s, the clinical trial has undergone a series of reforms that have cemented its status as the “gold standard” for clinical evidence.

But alongside the story of scientific progress is one of operational stagnation. Even as the science of trials advanced, the business of trials – sometimes referred to as the “trials enterprise” – has struggled to produce the evidence we need to develop new treatments and improve care. Most significantly, we have lost much of our capacity to run simple trials, cheaply, at scale.

To understand how this happened, we need to retrace the story of the trials enterprise. (If you’re interested in learning more about this story, I recommend this piece by Alexander Janaroff, which was a key source for this post, as well as FDA’s own history.)

Today, the drug industry’s trials are a big business; most trials are sprawling enterprises run by large organizations across multiple sites. But before this modern system of clinical trial research was developed, drugs were largely tested in an ad-hoc way by independent clinician-investigators. In the first half of the 20th century, the quality of those investigations could be mixed. At their best, investigators made genuine efforts to control their experiments, taking care to make adequate and unbiased comparisons between patients who took the drug and those who did not. At their worst, they relied on what can best be described as “vibes”. Drug companies might ask clinicians if the drug appeared to work, and use testimonials to justify their product’s safety and effectiveness. We had relatively few standards for what good, unbiased clinical research ought to look like.

Over time, however, the standards for research tightened, particularly by the 1950s as the principles underlying modern trials were codified. That process was pushed along in the 1960s as the FDA started systematically reviewing drugs’ efficacy. But even well into the 1960s, as scientific rigor increased, the fundamental operating model of the trial hadn’t changed.

To test a drug, a company would typically send samples to clinician investigators, who would take responsibility for figuring out whether the drug worked. FDA’s own statutes and regulations still reference this model: they stress the importance of bringing on “qualified investigators” to study drugs. The idea that trials might not be run by individual investigators but rather by large organizations across multiple institutions was relatively new and did not fit well with FDA’s model of regulation (a problem that persists today, incidentally, despite some subsequent regulatory changes).

In contrast to the large, multi-center studies that researchers were running as early as the 1940s, most early investigator-led studies were small, limited to just one institution. That created a problem: a single investigator in a single academic institution could study only so many patients. By the 1960s, it had become clear that these small studies were inadequate to the task of studying the latest drugs. The drugs themselves had changed: many drugs developed in the postwar period, such as antibiotics, had dramatic and obvious effects that were noticeable in small studies. By the 1960s, newer classes of drugs, including cardiovascular drugs, arose. Researchers and regulators were interested in studying these drugs’ effects on “hard” endpoints - like death, heart attack, and stroke - rather than just measuring their effect on biomarkers. These hard endpoints were observed relatively infrequently (fortunately, heart disease patients usually live for quite a while before succumbing to their illness).

These new drugs required a different approach to running trials. Given how rare these hard outcomes were, researchers realized that their studies would need to be far larger if they wished to detect an impact on mortality in a reasonable timeframe. In the 1970s, modern methods of study size calculation, advanced by researchers like Richard Peto, emphasized that the efficacy of many drugs could only be determined if thousands of patients were studied.

This realization led to the creation of a new approach to running studies, the “large simple trial.” In a pioneering paper by Salim Yusuf, Rory Collins, and Richard Peto, the authors described what this kind of trial should look like. First off, it should be very large: powered to detect small changes in mortality. That would require them to be run at many sites, each adhering to the same protocol. The authors emphasized that the protocol should be simple, and not too burdensome. They wrote: “Many clinicians involved in the management of patients are already overworked, and in practice a really large-scale trial is likely to succeed only if it adds little or nothing to their existing workload.”

The authors also stressed that simplicity need not sacrifice rigor or accuracy. In fact, simple protocols with simple treatment regimens would help improve patient and researcher compliance. While proper randomization was absolutely vital, other aspects of the trial might even vary a bit between study sites without biasing the results. As long as randomization was done properly, some “noise” should not threaten the validity of the trial. The largest trials of this era reflected this philosophy. One of the first large simple trials, the first of the “International Studies of Infarct Survival” (ISIS-1), which started in 1981, only required researchers to submit a 1-page form with patient data and had no monitoring or endpoint adjudication.

While these large simple trials never made up more than a small portion of overall trial activity, they had a major impact on the field. For example, the second ISIS study randomized over 17,000 patients and showed that streptokinase and aspirin used together could significantly improve the survival of heart attack patients. Trials like the ISIS studies and GISSI (an Italian study of heart attack treatments) each enrolled tens of thousands of patients, and their findings transformed care for heart attack patients and saved many lives. More broadly, the principles espoused for large simple trials–the importance of rigorous randomization, sufficiently large study sizes, and simple protocols–became important general principles of good study design.

The large simple trials also transformed the trials industry. Their rise led to the creation of a new kind of organization, the “Academic Research Organization” (or ARO): university-based organizations that specialized in running large multicenter trials, also sometimes called “megatrials,” for both academia and industry (examples include the Duke Clinical Research Institute and Oxford’s Clinical Trials Unit). The rise of the megatrial helped build the reputation of a number of renowned trialists who remain influential in the field today. This group included Peto, Yusuf, and Collins, as well as Robert Califf (who would later become FDA commissioner), and Martin Landray (who would later become famous for leading the RECOVERY trial during COVID).

Subscribe now

Transformation and Industrialization

With the growth of megatrials, the trials industry reached a critical pivot point. As early as the 1980s, industry had begun to demand more trials, more control, and more speed than university researchers and their AROs could provide. That led to the growth of a new kind of organization: the private contract research organization (or CRO). Over time, CROs displaced AROs as the primary operators of multi-site clinical trials. They were able to keep pace with rising industry demand, and even expanded those trials’ reach, extending their operations globally.

The rise of the CRO coincided with–and was facilitated by–another important development: the establishment of global good clinical practice (GCP) guidelines and intensified regulatory oversight over trials. While FDA oversight over clinical trials began in 1962, the 1990s GCP guidelines ushered in a new era of regulation. Trials began to face closer regulatory scrutiny, and monitoring and documentation began to increase.

These two developments reinforced each other. CROs had the organizational capacity to adapt to the GCP guidelines and help drug companies meet them. In turn, the heightened oversight under the new GCP guidelines gave FDA greater confidence in the ability of CROs to manage trials. Prior to this, there was an expectation that an impartial, unbiased trial should be run by academics–not by the private sector.

By the 2000s, CROs were running the majority of research; the relative importance of academic medical centers and AROs had greatly declined. In 1992, CROs earned just $1 billion, but as of 2024, CRO revenues are estimated to have reached nearly $60 billion. Nowadays, they are truly gigantic organizations: the biggest CROs, such as IQVIA, ICON or Parexel, each employ tens of thousands of people and support many hundreds of studies every year - far larger than the AROs ever were. This transition changed the nature of the trials industry. Trials were industrialized, at least, in part (I’ll argue soon that this industrialization was incomplete), and were conducted at a larger scale than ever before. Trials were also bureaucratized: the rise of the CRO coincided with the rise of the extensive system of monitoring, auditing, and documentation that defines today’s trials.

Meanwhile, the large simple trials pioneered by the leading trialists had begun to decline, driven by several factors. Funding had fallen off: the pharmaceutical industry still funded some trials run by AROs, but academic medical centers cut back their own research budgets under increasing financial pressure. Meanwhile, NIH, which had played a crucial role in shepherding the rise of the large simple trial, cut back their support for trials too. The decline in larger-scale confirmatory trials was especially stark: NIH-funded phase 3 trials fell from 230 in 2005 to 62 in 2015.

There was also a shift in interest – driven in part by evolving science. By the 2010s, biomarker-based research and precision medicine would come to dominate the drug development landscape. Drug developers moved away from large-scale cardiovascular trials as they pursued products for narrower uses tested in smaller studies.

But a crucial change reinforced–and perhaps even drove–these other trends: increased bureaucracy. In an effort to comply with regulatory requirements, drug companies and CROs added additional procedure, complexity, and bureaucracy into their trials. As bureaucracy increased, costs rose. Even publicly-funded trials and academic-funded trials were affected, since both academic and industry-led trials were run by the same staff in the same sites. Rising costs made it difficult for academic medical centers and NIH to continue to fund large trials. Even the drug industry lost interest in large trials as they grew more expensive.

Subscribe now

The end of trials’ golden age

To the trialists, the decline of the large simple trial might have felt like the end of a golden age. The trials enterprise that had given birth to large simple trials was gone, replaced with something more bureaucratic, more expensive, and less capable of answering the questions that actually mattered to doctors and patients.

The trialists directed sharp criticism at the FDA and other drug regulators’ “Good Clinical Practice” (GCP) guideline. Published in 1996 by the International Consortium on Harmonization (ICH), a global consortium of regulators, the guideline was intended to serve as a global standard for how regulated clinical research should be conducted.

The trialists lodged many complaints against the guideline, both general and specific. On the specific side, they pointed to a number of seemingly wasteful practices encouraged by the guidelines: expensive on-site monitoring of study sites, verification of every transcribed data element, extensive and exhaustive reporting of even irrelevant adverse events, and time-consuming documentation to support inspections and audits.

More broadly, research leaders argued that the GCP guidelines–unlike the clinical guidelines issued by professional societies–were created with no scientific input, had no logical basis, and had no evidence backing their recommendations. Moreover, the guidelines’ entire emphasis was wrong: rather than promoting the good scientific practice that had been espoused by the designers of the large simple trials, they emphasized extensive documentation, auditability, and a nitpicking approach to data quality that had little bearing on the overall validity of the study.

Worse yet, the regulations were applied too broadly. The GCP’s scope was initially limited to regulated research–that is, research whose results had to be submitted to regulatory agencies like the FDA. Gradually, however, the ICH GCP’s scope expanded: in the EU, the ICH guidance was required to be applied to all research. In the US, the guidance’s legal status is “non-binding” but, in practice, it too is applied to all clinical research conducted in the country. The leading trialists and academic researchers decried this change; likening it to a “straitjacket” on their research.

If the guidelines got most of the blame, the CROs and the drug industry came in for a close second. They interpreted FDA’s requirements in a maximally risk-averse way. Then their interpretations, reinforced by a fear of FDA inspections, created a stifling environment that drove up costs and bureaucracy at the expense of research quality.

There was an even darker edge to this concern. As several leading trialists presented suggestions to streamline trials, they feared opposition: “Certain entities have benefited from the complexity of the current regulatory environment — not just contract research organizations and companies providing training in the ICH-GCP guidelines, but also regulatory groups in pharmaceutical companies and other institutions, which have seen their revenue and influence increase substantially — and they too may oppose streamlining.” This view was part of a broader concern; that a kind of compliance-industrial complex had emerged, in which regulators, the pharmaceutical industry, and the contract research organizations shared an interest in retaining an elaborate and risk-averse bureaucracy.

As the trialists lodged their complaints, the pillars of the large simple trials movement were toppling, one by one. The trialists advocated for trials that were simple, large, and rigorously randomized. The industry was moving in the opposite direction. Trials were growing more complex, and CROs and the drug industry managed that complexity with tighter procedures and more layers of bureaucracy. This made large trials prohibitively expensive, so industry shifted its focus to smaller trials for complex biomarker-driven indications, and advocated for regulatory reforms that could make the trials even smaller and shorter. The last pillar to fall was randomization itself, as the drug industry encouraged the FDA to accept observational studies instead of trials. To the advocates of large simple trials, this was anathema; industry could more easily imagine doing away with trials altogether than making them simpler and cheaper.

Subscribe now

The Response to the Backlash

The trialists pushed back hard against these trends the best way they know how; through the institutions of science. In the 2000s, a series of journal articles and commentaries came out with titles like: “Randomized clinical trials: Slow death by a thousand unnecessary policies?”; “The Good Clinical Practice guideline: a bronze standard for clinical research”; and the more measured (but equally critical) “Clinical trials bureaucracy: unintended consequences of well-intentioned policy.” Over time, they would launch coalitions, host convenings, and lead scientific roundtables aimed at reversing the tide of bureaucracy in trials.

While these sorts of articles and convenings may have limited impact on the broader public discourse, they matter a great deal to FDA and other regulators, who depend on the support of scientists and academics. So as the criticisms mounted, the FDA engaged with the critics. In 2007, they partnered with Duke University to found the Clinical Trials Transformation Initiative (CTTI), aimed at making trials better and more efficient. The initiative’s founding CEO Judith Kramer, articulated the problem: “The investment in time, money, and human resources for clinical trials has been rapidly increasing in recent years, without a commensurate increase in the number of new products entering the market place. There is a growing consensus that clinical trials are inefficient and too costly.”

In the years that followed, FDA, informed by the work of CTTI and the trialists themselves, issued new rules and guidance that tried to address the most egregious examples of bureaucracy and waste. For example, in 2010, FDA published a rule limiting how often drug companies needed to submit expensive and time-consuming “expedited safety reports” during their trials. In 2013, FDA issued guidance clarifying that sponsors did not need to manually verify that every data element collected in the trial was accurately transcribed – a practice called ‘100% source data verification’.

Yet in each case, the rules were undermined by cautious, risk-averse interpretations. After the finalization of the 2010 safety reporting rule, CTTI found that expedited reports submitted to FDA actually increased. Likewise, years after FDA recommended against routine use of 100% source data verification, the vast majority of trials still adopted the approach. Similar efforts at reform, including regulations aimed at simplifying informed consent forms and reducing the scope of IRB review also had limited impact.

Despite these setbacks, efforts at reform continued, culminating in the 2025 issuance of the revised good clinical practice guideline. This guideline was explicitly designed to respond to the criticisms leveled by trialists. It cautions against overly complex trial protocols, and pushes hard for a “risk-proportionate” approach to tasks like study monitoring. Not only is the reform comprehensive, but–unlike the FDA-driven reforms that preceded it—the ICH guideline is global, raising hopes for more widespread adoption. But even as the guidelines were being developed, trialists expressed concerns that over-cautious interpretations of the guidelines could limit their value.

Subscribe now

“Back to the Future”: Reviving Large Simple Trials

Amid the backdrop of regulatory reforms, another important story was taking shape: an ongoing effort to revive the large, simple trial; or at least, to bring back some of its key principles. No story more exemplifies the power – and limits – of this approach than that of the COVID-19 RECOVERY Trial.

At the center of this story was Martin Landray, one of the advocates of simple trials we mentioned earlier. For years, he had helped design and run large simple trials alongside his colleagues at Oxford’s Clinical Trial Service Unit (CTSU) – an organization that had pioneered the concept of large simple trials. As trials grew more complex and bureaucratic, he was one of the loudest voices in favor of reform, and led several of CTTI’s efforts to reform and simplify trials. He even launched his own alternative trial guideline, “The Guidance for Good Randomized Clinical Trials” as a scientifically-grounded complement (and perhaps alternative) to the documentation-heavy and bureaucratic GCP guideline.

With the spread of COVID in early 2020, Landray had a chance to prove that a streamlined trial could be used to identify effective treatments for COVID. Before the pandemic had reached the UK, he had already begun to design the trial that would later become RECOVERY. He borrowed heavily from the large simple trials playbook: a one-page case report form, simple criteria for eligibility, and a simple protocol that could readily be administered at scale. But he modernized the approach, using web-based forms to collect data, an adaptive platform design to compare multiple treatments, online training for study sites, and the use of additional data collected from NHS’ electronic health record systems. It cost just $500 per patient.

The streamlined design allowed the trial to quickly recruit patients and produce results. Within just a few months, RECOVERY had enlisted every acute hospital in the UK (along with sites in many other countries) and had identified life-saving treatments for COVID. Those treatments quickly became part of standard COVID care in hospitals, saving millions of lives.

Like the 2025 good clinical practice guidelines, the success of the RECOVERY trial felt like a turning point. Landray and his fellow trialists had spent decades arguing for a return to streamlined trials, and, in this moment, they seemed to be vindicated. If RECOVERY’s streamlined and broad-based approach could work for COVID, surely the model could be adopted for other diseases.

In fact, RECOVERY was part of a broader movement, pushing for new kinds of trials that could recapture the benefits of those older, large, simple trials. The trialists have outlined what those new trials might look like. Like the old trials, they would be streamlined to impose minimal additional burden on study sites. But unlike those old trials, these new trials could leverage the data in electronic health records to streamline trials even further and collect more useful data. Their reach could be broader too: “point-of-care” trials could be run in community settings outside of academic medical centers, and decentralized trials could be carried out remotely through telemedicine. These newer trials could build on the strengths of simple trials while creating something better and more efficient.

But these new kinds of trials face huge obstacles. The RECOVERY trial was run in a COVID state of exception. It’s not clear whether other trials of similar scope and ambition would be possible today, absent a major public health crisis. Nor, it seems, was such a trial possible here in the United States. Our own efforts to run large-scale trials during COVID were largely unsuccessful. Researchers continue to find ways to run streamlined trials, but they remain a very small part of the research enterprise.

Subscribe now

Why large simple trials still matter

Today, the problem of clinical trial inefficiency and bureaucracy feels more urgent than ever. There is newfound energy and enthusiasm around doing something about the problem. Amid that backdrop, many of the debates that preoccupied the clinical trialists are playing out again: we are asking ourselves how we can make trials more efficient, more streamlined, and less bureaucratic. That makes this a good time to assess what we can learn from the past two decades of efforts at trial reform.

It’s clear that, for all they have achieved, the trialists have not yet been able to address their most important goal: to make trials significantly simpler and less bureaucratic. While the regulations have improved, the trials’ costs and complexity have only risen. Even if we stemmed the tide of bureaucracy, there is probably no way to return to the trials enterprise we once had.

But if we can’t go back to the heyday of large simple trials, the good news is that we wouldn’t necessarily want to. We should acknowledge the accomplishments of our current industrialized bureaucratic trials system. The industry is vastly larger than it was in the 1980s “golden age”, and our capacity to learn from trials has never been greater. We’re running more trials across more sites and more countries, for a broader range of diseases. And despite the greater size and complexity of the trials, we retain confidence in their rigor and validity. Thanks to the rigor of our system, FDA’s leaders were able to make the argument that we only need one trial, instead of the previous two, to demonstrate a drug’s effectiveness.

The challenge, then, is preserving the benefits of today’s trial system while streamlining it and reining in its excesses. For that reason, we should keep learning from the era of large simple trials. The principles behind those large simple trials are still valid and deserve to be more widely known and followed. And more importantly, when we study the history of trials–and the work of pioneering trialists like Martin Landray–our minds open to new possibilities. We learn that there is nothing inevitable about the way our clinical trials system works today: it especially does not have to be so complex and bureaucratic. We’ve transformed the trials system before, and it can be transformed again.

Back in January FDA published draft guidance on use of Bayesian clinical trials. Two of us already covered it, Adam Kroetsch on this blog and yours truly on Andrew Gelman’s blog. Now Andrew, Erik van Zwet and I also put together a short commentary for JAMA:

FDA Draft Guidance for the Use of Bayesian Methods in Clinical Trials by Andrew Gelman, Erik van Zwet and Witold Więcek (2026) in JAMA

It’s only around 1,000 words, so you should read it. The overall assessment is similar to the piece I wrote in January: we think it’s a very positive step. In one sentence, the contribution of this guidance is to clearly define how relevant information can be used to create Bayesian priors for trials, which can make many of them more efficient. Now the question is whether trial sponsors are willing to do it more often.

But there is also the perspective of the regulator: are there any broader benefits to the FDA from being Bayesian? We think the answer is yes. Following these principles can lead to more consistent and transparent decisions. This may strike some people as counterintuitive. The most common critique of Bayesian methods in clinical trials is that they “sneak in subjectivity” under the guise of prior. So how could they lead to more consistency? As Adam nicely outlined in his piece, the status quo is that the FDA already relies very heavily on “clinical judgement” in its approvals, which does not sound dissimilar from having a Bayesian prior to me! We write:

Consistency and transparency in these decisions are important—uncertainty about the procedures would prevent drugmakers from conducting research and lack of transparency can expose decisions to political pressure and lobbying. However, a survey of 912 FDA applications [5] found that FDA decisions did not consistently cite prior reasoning and that some approvals, after additional review, reflected new interpretations of existing evidence rather than new evidence, raising concerns about consistency across review cycles.

This survey of 912 applications, Janiaud et al, is also a short and interesting read. It is co-authored by FDA staff and it broadly acknowledges that there is no consistency, “resulting in standalone, bespoke decisions”. And that is not automatically a bad thing! As we also say in the piece, flexibility is necessary in regulatory decisions. But there are notable examples where this becomes a problem, where the basis of these judgements becomes… questionable:

A recent high-profile example may be illustrative. In February 2026, the head of the FDA’s Center for Biologics Evaluation and Research issued a refusal-to-file letter for Moderna’s mRNA influenza vaccine, citing the failure to compare the vaccine against high-dose competitors in older adults [6]; however, this decision was reversed after a week.[7] The problem of comparisons among relevant subpopulations can be directly and transparently addressed using hierarchical bayesian methods. Influenza vaccines are among the best studied therapeutics in existence. Relevant information from other trials could be incorporated into the prior and data reanalyzed to conduct a risk-benefit analysis in older adults without conducting a new trial.

In other words, this new guidance specifically focuses on the topic of how to transparently incorporate information across other trials and incorporate benefit-risk into the definition of success. If the FDA reserves the right to move the goalposts somewhat, that is a valid concern! But why not use the principles from its own Bayesian guidance to reach this decision: based on all available data, how likely is it that the vaccine is not as effective in the elderly as its high-dose competitors? What sort of prior assumptions are required to reach this conclusion? The Bayesian principles in this guidance make this project feasible.

In sum, we should be scrutinising both the work of the drug maker and the decision maker. Yes, it is entirely the sponsor’s job to generate data that proves their therapeutic is safe and effective. But then it is the job of the regulator to prove to us that their approval decisions are rational—especially in the cases which lean heavily on judgement. Or, as we like to say in my line of work: show us the model!

ANNOUNCEMENT KLAXON

The Irish government is seeking volunteers for their Clinical Trials Advisory Council. If you have relevant expertise and are interested in shaping European regulatory frameworks, consider applying.

Our friends at 1Day Sooner, best known for their advocacy for challenge trials to speed up COVID vaccines, are hiring for a new role that could genuinely impactful in helping to accelerate clinical testing and getting breakthrough medicines get to patients faster.

They’re hiring a Policy Lead for their new Clinical Trial Abundance program. As you’ll know from this blog, Clinical Trial Abundance is the idea that we can accelerate medical innovation by making it easier to collect and generate experimental data, and do more with the evidence we already have. It’s something a number of us are interested in, and 1Day Sooner are now building out the policy infrastructure to move it forward.

This is a policy-focused role: involving regulatory strategy, drafting comment letters and legislative language, building coalitions, and working with decision-makers across FDA, HHS, NIH, and Congress. They’re looking for someone with 5+ years in health or science policy who can translate high-level goals into concrete wins.

It’s remote and full-time, with a preference for DC-based candidates (though they’ll consider applicants across the Western Hemisphere). Salary is $100–145k in the US.

If that’s you, apply here: https://www.1daysooner.org/jobs/

If it’s someone you know, please share this with them!

Share

Thank you :)

– Saloni

There is a vast gulf between how the pharmaceutical industry sees itself and how the public sees it. To the public, the pharmaceutical industry is venal and corrupt, putting profits above patients’ needs. But when I speak to those who work in the industry, they are deeply mission-driven: many enter the field with sincere hopes of discovering treatments and cures for the world’s most devastating diseases.

At the heart of this mission are the scientists who discover new drugs. They’re the reason the industry exists, and they represent how pharma wants to be seen. If you visit a drug company’s website, you’ll find stock photos of them: They’re the intrepid researchers, toiling away in their lab coats, on the cusp of their next breakthrough. If the industry has a ladder of prestige, these scientists sit at the top of it.

Also near the top of the prestige hierarchy are those who work on clinical trial design. They may not discover new drugs, but they make decisions that can make or break a drug development program. Their work requires deep scientific expertise, regulatory knowledge, and creativity.

Many rungs below sit the people who actually test the drugs and prove they work. These are the “boring things after the creative, exciting part is done,” including the mundane but necessary work of “clinical operations”: setting up trials, negotiating payments, recruiting patients, collecting data. Even among those who work in the field, this is often considered the “unsexy”, “boring” part of drug development.

The field of clinical operations lacks drama and gets little attention. The industry, its leaders, and its investors focus on big milestones: a “go / no-go decision”, a high-stakes FDA meeting, the start of a $100+ million dollar trial, the pivotal readout that determines a program’s fate. By contrast, trial operations are a day-to-day grind. There are few big moments to celebrate. I suspect many CEOs spend relatively little time thinking about it.

But we desperately need to improve trial operations if we want to make clinical trials more efficient.

Trial operations have gotten less productive

That’s because trial operations are what drives trial costs, and those costs are rising fast.

To understand the rise in costs, it’s useful to understand how drug industry trials are typically run. Drug companies pay for trials, but they aren’t the ones who carry them out. Rather, their main job is to develop the trial protocol, which lays out how the trial ought to be run: who is eligible participate in the trial, what procedures should be run, what data should be collected, and how that data should be assessed. The work of running the trial itself is usually outsourced: Contract research organizations (CROs) coordinate and oversee the trials, and research sites—including large academic medical centers and research clinics—carry out the protocol.

As research sites do the bulk of the hands-on work of running the trial, they also incur the largest portion of the trial’s cost. Site costs have risen rapidly in recent decades. One analysis showed that site costs for phase 3 trials rose from $11,630 per patient in 2000 to $23,711 in 2011—an increase of nearly 10% per year. While we lack more recent data on site costs, other data suggests they are continuing to rise: a more recent study which measured total trial costs (not just site costs) found that phase 3 trial costs had reached $55,567 per patient by 2017. Meanwhile, as costs have risen, trials have also gotten slower. They take longer to set up, longer to enroll patients, and longer to complete. Those costs and delays limit how many drugs ultimately make their way to patients.

One reason that trial costs are rising is that the trials have grown more complex. Drug companies are collecting more data, running trials across more sites, and measuring more endpoints. They are also placing tighter restrictions on who can enroll. Some of the increases in complexity are scientifically justified, but much of it looks like bloat. This complexity drives up costs and puts strain on the trial sites. Trials are being optimized for regulators and risk-averse managers—not for the sites that actually have to run them.

But complexity isn’t the whole story. If we take a closer look at site costs, we find that they have risen even after accounting for the increased complexity, and faster than inflation in the health and research sectors. In other words, the trials industry, which already lagged in its use of technology by the turn of the century, has probably gotten less productive over time.

This won’t surprise people who work in clinical operations. While drug companies and trial sites take great care to adhere to protocols, procedures, and regulations, there is very little about the process that is automated or streamlined. Brandon Li, a clinical trials entrepreneur, remarked that “clinical trials are $200m group projects held together by duct tape and project managers.” If you’ve ever worked for an over-stretched and understaffed organization, you can picture the failures: manual transcription across paper and computer systems, reliance on ad-hoc spreadsheets, lost logins and passwords, inboxes full of missed emails, stalled contract negotiations, and unpaid invoices. Somehow the work gets done, but it feels much harder and less efficient than it should.

If that picture sounds discouraging, I’d like to emphasize the good news here: these problems are fixable. Making trials dramatically more efficient doesn’t require us to invent new science or technology. It doesn’t require us to reform the FDA. We don’t even need AI. (All of these things would be helpful, though!) All we really need to do is fix the basic stuff: stop using paper, stop transcribing information manually between systems, stop wasting time on useless administrative tasks, and stop making the protocols so needlessly complex. In other words, we just need to do the kinds of operational modernization that other industries have been doing for ages.

Fixing trials is boring, hard work

If the fixes are so basic, why haven’t we done them already? I suspect it’s because the work is too boring–and not prestigious. In an industry that prizes scientific creativity and expertise, operations don’t get much attention. Fixing trial operations and designing trials that are easier to run requires sustained focus and coordination. It won’t make big headlines in the trade press.

A lack of attention makes reform harder. Consider the problem of risk aversion. I have interviewed industry experts and insiders - and nearly all of them mentioned that operations are run in a deeply risk-averse way. Managers at drug companies would rather stick to tried-and-true approaches to running trials than try to reform them and risk regulatory scrutiny. This risk aversion has downstream effects on the entire trials industry. Since drug companies are reluctant to embrace innovative cost-cutting measures, CROs face little incentive to cut costs. And without sponsor support, it’s hard for trial sites to improve their operations.

But behind many cases of risk aversion lies a deeper problem: a lack of leadership attention. Risk aversion isn’t unique to pharma. You will struggle to find a middle manager in any large organization who is willing to be bold unless they have visible backing from leadership. As long as industry leaders neglect clinical operations, managers will default to doing what they’ve always done and productivity will stagnate.

And trial operations need a lot of attention. It’s difficult to change such a fragmented, entrenched, and heavily regulated system. Anyone in the industry seeking to make even a small improvement in trial operations must navigate a tangled web of contracts, regulations, and SOPs. Each of their business partners—whether they are sites, sponsors, or CROs—will handle things differently. There are also larger-scale coordination problems: sponsors can’t run more efficient trials unless the sites agree to modernize their operations. Sites can’t modernize their operations unless all of their sponsors agree on what that modernization should look like.

That means that there are no quick fixes. We don’t lack the knowhow or technology to improve how trials run; we lack the ability to deploy improvements at scale. We can’t just send consultants in to make trials more efficient, and there’s no B2B SaaS tool that will transform the industry. Improving the efficiency of trial operations will take hard work and focused attention. Policymakers and the government will probably have to get involved too.

If we want to make trials more efficient, we need to pay more attention to the boring stuff. We need to measure it and prioritize it. That’s part of why I created the Clinical Trials Efficiency Project. When it comes to improving clinical trial efficiency, there’s a role for new trial designs, new science, new technology, and new ideas. But I also hope we can bring attention to the boring bottlenecks that make trials inefficient and have remained neglected for too long.

Adam

It’s easy to see that proposal as an ambitious reform, and in some ways it is. But the history of medicine shows us that clinical trials have already undergone a series of transformations that once seemed equally bold. Over time, the way we test treatments has become more rigorous and standardized. Improvements in transparency are a continuation of this trend.

In this post, I’ll give a brief history of clinical trial reforms and why transparency is the natural next step.

The rise of randomized controlled trials

One of the most consequential rigor-enhancing changes in clinical trials was the rise of the randomized controlled trial. In 1962, the United States passed the Kefauver-Harris Amendments, which required that new drugs demonstrate efficacy through adequate and well-controlled investigations before receiving approval. In practice, this pushed regulators and manufacturers toward randomized controlled trials (RCTs). This was seen as a dramatic shift.

At the time, clinicians had to rely on scattered uncontrolled studies, controlled studies without randomization, and case reports of claims of miracle treatments. They’d see some patients improving and others worsening, with no clear way to estimate the average effect or detect harms that only become visible at scale.

The requirement for controlled trials might have raised costs and overturned established practice. But it addressed a real problem: if you want to understand the effectiveness and safety of a drug using observational, uncontrolled data, you’d face many challenges:

• Regression to the mean: Many conditions bring people to the doctor when symptoms are at their worst, and some improvement would have happened anyway. Someone might seek treatment for severe back pain after a particularly bad week, only for the pain to ease as the flare-up subsides. The same pattern appears in mental health: a person may begin therapy during an especially intense depressive episode, then improve as the episode naturally wanes. If improvement follows treatment, it’s tempting to credit the intervention, even when part of the change reflects a return toward a person’s usual baseline.

• External events: Outcomes can also shift because the world around patients changes. For example, asthma admissions may spike during a period of high air pollution and fall when air quality improves, a heatwave can increase dehydration and kidney stress. Even public health campaigns, changes in food availability, or a stressful economic downturn can influence sleep, diet, and cardiovascular risk. If a treatment is introduced during such periods, its apparent effects may partly reflect broader changes.

• Selection into treatment: People who receive a treatment often differ in systematic ways from those who do not. Patients who opt for a new preventive medication might be more health conscious, adherent to medical advice, or able to afford regular care. Conversely, a specialist clinic may see the sickest patients, or certain drugs may only be prescribed at severe stages of a disease, making treatment look less effective than it truly is.

• Blinding and concealment address a different source of bias: the expectations and behaviours of the people involved in a trial. Even after randomization, participants or researchers may figure out who received the active treatment because the drug looks, tastes, or smells different from the placebo, or because it produces noticeable side effects. When this happens, it can partly reverse the benefits of randomization: if participants or researchers can identify who received the active treatment, differences between groups may reflect expectations and behavioural changes rather than the drug alone, meaning the estimated effect could be larger or smaller1 than the true effect of the treatment itself. Concealment is a first step, preventing researchers from knowing which treatment someone will be assigned to before allocation. Blinding goes further, keeping both participants and researchers unaware of group assignments throughout the trial.2

Without careful design, these differences can wrongly appear to be treatment effects. But adjusting for these factors is difficult, and the flexibility can also open the door to questionable research practices: analysing data in selective ways, choosing favourable subgroups, or stopping analysis when results look promising.

Randomization helps because it creates groups that, at the outset, have roughly equal propensities for the outcomes being studied. Darren Dahly has a clear explanation of this logic. And like my friend Julia Rohrer, I think of randomisation as the closest thing epidemiology has to magic.

In my view, there’s no question that some treatments did have good evidence for them even before the rise of randomized controlled trials. In particular, researchers could notice when treatments were followed by large, otherwise unexplainable changes – like the elimination of a disease following vaccination, or reversals of diseases that usually progressed quickly or fatally, as was sometimes the case after treatment with drugs like antibiotics and insulin.

But in most cases, it would have been hard to identify effective drugs, especially if their benefits were more modest, or if their benefits took a long time to become visible.

Subscribe now

It’s useful at this point to break down randomized controlled trials into their components: an experiment, with randomization, a controlled group, and often the practice of ‘blinding’ as well. Each of these components has had many precursors.

Take controlled groups as an example, which trace back – at least as far as I knew – to 1747, when James Lind compared treatments for scurvy aboard a Royal Navy ship and found citrus fruit dramatically effective. [After publishing this, Erin Braid pointed out that there were far earlier experiments with controlled groups, including in Ancient Greece; James Lind’s experiment is commonly named as the first one or one of the first ones.] I also found it interesting to learn that some 19th and 20th century controlled studies assigned patients to treatment in an alternate sequence, as they arrived to see a doctor, which is somewhat close to randomisation, but doesn’t involve blinding the researcher or participant to which treatment they received.

Blinding has an even more interesting history. It traces back to at least 1784, when King Louis XVI commissioned scientists including Benjamin Franklin and Antoine Lavoisier to investigate the claims of Franz Mesmer, who said he could cure illness through “animal magnetism” (later named ‘mesmerism’ after him). Franklin and Lavoisier designed an experiment in which they physically blindfolded subjects while performing the magnetic treatment; but crucially, they sometimes withheld the treatment without telling the subjects.

When they reported sensations regardless of whether they actually received the treatment, the experiment showed the effects were driven by their expectations rather than magnetism. It helped popularise the idea of blinding, in which patients are prevented from knowing which treatment they receive – although it was often the case that blinding was used to ‘debunk’ findings, rather than being open-minded to the results. Louis XVI’s interest, for example, was not purely scientific: mesmerism had become a fashionable and lucrative craze in Paris, and he likely worried about public disorder and the erosion of medical authority, giving the monarchy strong incentives to discredit it.

Combining these practices into randomised controlled trials, however, is relatively recent, at least historically speaking. The first widely recognised modern RCT was the UK Medical Research Council’s 1948 streptomycin trial for pulmonary tuberculosis.3 Patients were randomly assigned to receive the antibiotic streptomycin (a new breakthrough at the time) plus bed rest or bed rest alone. The trial showed clear improvements in survival and lung disease according to radiology scans in the treated group, but they also found early evidence of antibiotic resistance emerging.

As the twentieth century progressed, RCTs became more common. A very interesting example I’ve read of recently was the US National Institutes of Health’s Coronary Drug Project, conducted in the late 1960s and early 1970s, before statins were introduced.

In the decades leading up to it, researchers were experimenting with a surprisingly eclectic set of ways to lower cholesterol. If high cholesterol drove coronary heart disease, then lowering it ought to prevent heart attacks, so investigators tried hormones, vitamins, industrial resins, and metabolic drugs. Doctors claimed various drugs might be effective:

• Thyroid hormone (notably dextro-thyroxine)

• Oestrogens

• Niacin (vitamin B3)

• Bile-acid sequestrants such as cholestyramine

• Fibrates

The Coronary Drug Project trial allowed for a proper, large-scale testing of these drugs in a randomized controlled trial to see whether they actually reduced heart attacks and mortality. The results were surprising – niacin and cholestyramine showed meaningful benefits, but some other treatments, including oestrogen therapy and dextrothyroxine, actually slightly increased mortality – and changed clinical practice.

By the 1970s, randomized controlled trials were widely accepted as a gold standard for evaluating therapies. And after the 1962 reforms in the United States, along with similar regulatory shifts elsewhere, controlled evidence became the norm for drug approval.

Subscribe now

Preregistration

Randomization solved the design problem, but new challenges emerged in how results were analyzed. By the late twentieth century, the environment surrounding drug regulation had changed profoundly. To bring a medicine to market, companies were now expected to demonstrate both safety and effectiveness, usually through at least two well-controlled clinical trials, and typically using randomized controlled designs.

This was a major improvement over earlier eras, but it also raised the stakes dramatically. When development costs run into the hundreds of millions and a successful drug may earn billions annually, a failed trial can mean the loss of an enormous commercial opportunity.

The 1980s and 1990s saw a wave of breakthrough therapies reach patients. Statins transformed the prevention of cardiovascular disease. Angiotensin-converting enzyme (ACE) inhibitors improved survival in heart failure. Antiretroviral drugs began to turn HIV from a fatal diagnosis into a manageable condition.

But not every promising therapy proved effective, and even successful drugs often looked more impressive in early studies than they did later in routine practice. Understanding why requires a brief detour into how trial results are judged.

In most trials, success is judged using a statistical test that produces a p value, roughly the probability of seeing a result at least this extreme assuming the treatment has no real effect. By convention, results are called “statistically significant” if this probability falls below 0.05.

At the margins, this threshold creates a temptation. When a trial narrowly misses its endpoint, researchers may search for a significant result elsewhere: in a secondary outcome, a composite measure, or a subgroup analysis that, by chance, clears the bar. Given the enormous financial investment involved in developing a drug, there are strong incentives to tweak analyses at the margins, for instance, by changing which outcome is analyzed after seeing the data, a practice called outcome switching.

This flexibility undermines the validity of results. While randomisation improved how trials are designed, it did not fully address how results are analysed and reported; and flexibility after seeing the data creates opportunities for finding results that are favorable but unreliable.

To reduce the scope for this type of post hoc reinterpretation, regulators and journals began requiring trial protocols to be registered in advance. Starting in the late 1990s and expanding in the early 2000s, investigators were asked to document their study design and prespecified outcomes before data collection was complete. In the United States, these requirements broadened over time and were formalized through federal law and FDA regulations, with trial protocols and outcomes posted on ClinicalTrials.gov. Registration also became a prerequisite for publication in major journals, reinforcing the norm across the research community.

Analyses of trials funded by the National Heart, Lung, and Blood Institute found that before prespecification became standard, published trials were more likely to report positive results. After registration became mandatory, the pattern changed: many trials found no effect, and some detected harm. It became harder to redefine what success meant.

Even so, the value of prespecifying analyses is not about eliminating judgement but in improving transparency. It allows readers to see what researchers initially set out to measure, what changed, and why. There are countless examples from history of how both successes and failures in science have contributed to knowledge; science advances by testing predictions, learning from the results, and refining theories over time.

Subscribe now

Trial reporting

By the late twentieth century, clinical trials had become the gatekeeper for drug approval. Regulators increasingly required convincing evidence of both safety and effectiveness, usually from multiple well-controlled studies, before a medicine could reach the market. This shift improved the reliability of medical evidence. But even as trial design grew more rigorous, a problem remained: what happened to the results once a study ended.

The results of many clinical trials were never published. Studies showing clear benefits were far more likely to appear in medical journals, while trials finding no effect or suggesting harm often remained unpublished. The problem became increasingly visible in the 2000s, as systematic reviewers began documenting “missing” trials.

Analyses of antidepressant trials comparing journal publications with the full set of studies submitted to regulators found many studies went unpublished, and were disproportionately those with negative or null results (see here for a more recent analysis, which finds improvement). The published literature suggested overwhelming effectiveness, while the complete dataset showed more modest benefits. When meta-analyses relied only on published studies, average effect sizes were inflated, meaning clinicians and patients were often seeing an overly optimistic picture of drug effectiveness.

There are probably many reasons for this: Journals prefer novel and positive findings, and often require multiple submissions and long delays before acceptance, companies have little commercial incentive to highlight disappointing results, academic researchers face career pressures that reward publishing and success, and negative results seem less actionable. The path of least resistance is to simply move on, hopefully to something more promising.

But the implications are not merely academic. Incomplete reporting makes it harder to learn from failures and adjust drug development plans, to estimate the true average benefit of treatments, to detect rare harms, and to make informed decisions about patient care.

Since the 2000s, governments and regulators began introducing legal requirements for trial registration and results reporting. The 2007 FDA Amendments Act in the United States required many trials to post summary results in a public registry. In Europe, transparency rules evolved into the EU Clinical Trials Regulation and a centralized database designed to make trial information publicly accessible. These policies aimed to ensure that results entered the public record regardless of whether they were favourable.

Compliance did not improve overnight and enforcement was initially limited, but pressure emerged from multiple directions. The AllTrials campaign pushed for the registration and reporting of all trials, the Good Pharma Scorecard tracked company transparency commitments, and the FDAAA TrialsTracker made reporting performance publicly visible. In the United Kingdom, parliamentary scrutiny and reviews by the National Audit Office and the House of Commons Public Accounts Committee criticized failures to report trial results, increasing political and public pressure.

This combination of regulation, monitoring, and public accountability gradually shifted behaviour. Reporting rates rose substantially in the United States after reporting requirements took effect. Recent research by Cunningham et al. finds that academically funded trials initially had very low compliance rates, but rose rapidly. Industry funded trials began at higher compliance rates, and also increased. The researchers also found that this shifted the behaviour of other firms, which became more likely to wait for trial results before advancing similar drug candidates. As with earlier reforms, mandatory reporting was initially framed as a bureaucratic burden, but it’s increasingly considered basic infrastructure for trustworthy medicine.

Today, roughly 75–80% of clinical trials that meet the FDA requirement report their results within a year, with higher rates of reporting among industry-funded trials than academic trials.

Conclusion

This blog is about clinical trial abundance, but it’s worth stepping back to ask why we care about trials at all. Clinical trials are not an end in themselves; they exist to generate reliable information about whether treatments are safe and effective. In that sense, it’s not just clinical trials that we want an abundance of, but information from them. The reforms described above, randomization, preregistration, mandatory reporting, addressed ways that information was being distorted or lost. Data sharing is the natural continuation of that arc.

In my previous post, I argued that sharing individual patient data could unlock a range of benefits that go beyond what published results alone can offer: pooling data across trials allows meta-analyses to detect effects on rare outcomes and confirm or rule out rare harms; it helps make sense of inconsistent results by letting researchers explore whether differences across trials reflect chance, patient populations, or study design; it makes it easier to ask new questions of old data, including when drugs developed for one condition show unexpected benefits for another; it enables better head-to-head comparisons between treatments, and can help tailor recommendations to patients’ characteristics; and it reduces redundancy: when trial results are visible, researchers can learn from failures and avoid repeating them.

So why isn’t this already standard practice? For industry-funded trials, the commercial disincentive is straightforward: sharing data helps competitors at least as much as it helps you. A firm that shares its patient data is effectively subsidising rivals’ research programs. But if all firms shared, the field as a whole would benefit, and so would each firm within it, gaining access to a much larger pool of evidence than any single company could generate alone. The case for coordination, whether through regulation or shared norms, follows from that.

For academic trials, the commercial disincentive largely disappears, and most academic research is publicly funded, which creates its own obligation to make results available. But it’s still uncommon to share data, probably as a result of practical frictions, career incentives that reward further publications foremost, and the absence of strong mandates. The relative neglect of data sharing in academic trials is harder to justify.

The case for sharing individual patient data from trials is strong enough that I think it warrants the same kind of coordinated push that succeeded for trial registration and results reporting. The answer could be better incentives, data infrastructure, and in some cases mandates from funders and regulators to make sharing practical and beneficial rather than burdensome.

Scientists often say that the best research doesn’t just answer a question, it opens up new ones. That’s how I think about data sharing as well. Without individual patient data, we’re left mainly with headlines, with limited ways to verify or explore the data to answer those questions, or build upon accumulated knowledge.

I’d like to thank Matt Clancy, Dylan Matthews, Nisha Austin, Ruben Arslan, Jamie Cummins, and Adam Kroetsch for feedback that improved this post.

During a recent interview, Dwarkesh Patel and the CEO of Anthropic, Dario Amodei, discussed whether clinical trials will remain a meaningful bottleneck for drug development in the age of AI. Patel said that “most clinical trials fail because the drug does not work.” In response, Amodei speculated that as AI models get better at designing drugs, “clinical trials will be much faster … let’s say, they will take one year.”

This is a commonly voiced sentiment, but flawed. The truth is that the most significant barriers to progress today are rarely a lack of intelligence. London has a housing crisis even though the technology to design and construct homes has existed for centuries. The bottleneck in housing is not a lack of knowhow, but rather the weaponization of environmental regulations, planning, and NIMBYism. Much the same is true for clinical trials.

AI models can help design more elegant molecules, in the same way an architect can use AI to design more efficient floor plans, but neither intervention guarantees an efficient use of institutional machinery to make that design in the real world. Even the most promising drug candidates must be tested in human bodies which, in turn, need time to metabolize those drugs and develop side effects. Patients must be recruited and followed over time, and regulators must be satisfied. None of this is easily accelerated with AI.

Although I’m optimistic that AI will design better drug candidates, this alone cannot ensure “therapeutic abundance,” for a few reasons. First, because the history of drug development shows that even when strong preclinical models exist for a condition, like osteoporosis, the high costs needed to move a drug through trials deters investment — especially for chronic diseases requiring large cohorts. And second, because there is a feedback problem between drug development and clinical trials. In order for AI to generate high-quality drug candidates, it must first be trained on rich, human data; especially from early, small-n studies.

Clinical Variables

The Amodei interview conflates two distinct variables: the success rate of a trial (based on the quality of a drug), and the speed of that trial, understood as an operational process.

The first variable — the success rate of a trial — is the probability that a drug candidate will be both efficacious and safe in humans. The current success rate for a drug entering clinical trials is only about ten percent, meaning 90 percent of all drugs fail. Most AI efforts in biology aim to boost this success rate.

The second variable is the speed of data generation — the calendar time required to run an experiment after it has started. A clinical trial is just an experiment in human subjects, and the duration of that experiment is determined by both operational and biological constraints that are largely independent of how confident we are in the drug itself. Recruiting 1000 patients across 10 sites takes time; understanding and satisfying unclear regulatory requirements is onerous and often frustrating; and shipping temperature-sensitive vials to research hospitals across multiple states takes both time and money.

Amodei’s prediction that clinical trials could be done in a single year seems to assume that improving the first variable will also compress the second; but this is not so. Even if AI can help design more effective drugs, timelines will not compress until we solve the operational and regulatory bottlenecks of trials.

Admittedly, there is a tempting counter-argument: If AI does generate better drug candidates, then perhaps clinical trials will cease to be a meaningful bottleneck. If a drug is almost certainly going to work, then trials may become a “formality,” even if, in general, they remain unnecessarily costly and long.¹ This argument is also wrong, but understanding why requires being clear about what clinical trials are actually for.

Trials serve two distinct functions: validation — confirming whether a drug works and is safe — and learning, or generating biological data to refine our understanding of a disease, a compound, and the relationship between the two.

Validation is the primary goal of large-scale Phase III trials, which come later in the process and are typically designed to support regulatory approval. While data from these studies can deepen our understanding of drugs, their main goal is to figure out whether a treatment works under defined conditions. Learning, by contrast, is the dominant aim of early-stage trials. Conducted in smaller patient populations and often using exploratory designs, these studies are not limited to simple “yes or no” outcomes. Instead, they are experiments in the fullest scientific sense: they seek to uncover how a drug behaves in the human body, and how the disease itself responds. As argued in my earlier essay, Clinic-in-the-Loop, this makes these early stage trials active engines of discovery that close the feedback loop between hypothesis and human biology.

For large “validation” trials, is it plausible that their cost will simply cease to matter in a (theoretical) world where AI makes drugs with a high probability of success? I think the answer is no, for a couple reasons.

First, unless we increase the pace and volume of the early-stage “learning trials,” it is unlikely that we will ever approach such a level of certainty in drug discovery. Today, most AI systems in drug development are trained predominantly on in vitro data and animal models. While valuable, these sources only imperfectly capture the complexity of human biology. Without large amounts of high-quality data from actual humans, we should not expect AI to generate predictions that approach near-certainty about trial outcomes.

Second, even if improved modeling could compress early-stage development timelines, every successful drug must still demonstrate benefit on an endpoint; either a clinical endpoint or a surrogate endpoint.

For many diseases, however, the relevant endpoints take a very long time to observe. This is especially true for chronic conditions, which develop and progress over years or decades. The outcomes that matter most — such as disability, organ failure, or death — take a long time to measure in clinical trials. Aging represents the most extreme case. Demonstrating an effect on mortality or durable healthspan would require following large numbers of patients for decades. The resulting trial sizes and durations are enormous, making studies extraordinarily expensive. This scale has been a major deterrent to investment in therapies that target aging directly.

Lastly, the duration of a clinical trial does not merely determine how fast an individual therapy reaches patients. It also shapes which diseases attract serious investment and which do not. In a scenario where AI produces better drug candidates, yet trials remain slow, medicines will become unevenly deployed. In that scenario, capital and innovation will flow toward indications with clear, rapidly measurable endpoints — such as oncology — where trials can be completed relatively quickly. By contrast, fields like aging, where meaningful outcomes take years or decades to observe, will continue to lag unless there is genuine innovation in endpoint development.

Osteoporosis, a progressive bone disease that primarily affects post-menopausal women, illustrates these dynamics well. Firstly, it benefits from an unusually strong preclinical model in the ovariectomized rat (OvX model). Unlike many other chronic diseases, where animal models have poor predictive validity, the OvX model reliably recapitulates post-menopausal bone loss and predicts drug response. This rat model is so good, in fact, that Phase III trials for osteoporosis succeed 83.7 percent of the time, substantially higher than the cross-indication average of roughly 57.8 percent at the same stage.

Given the existence of a good pre-clinical model that allows us to select higher quality candidates and the scale of unmet need in osteoporosis, one might expect it to attract sustained and substantial investment. But instead, the opposite has occurred. Today, only two drug candidates remain in late-stage clinical development for osteoporosis.

The primary reason is that Phase III osteoporosis trials are exceptionally large, long, and expensive to run. The core challenge lies in the endpoint: fracture reduction. Fractures are relatively infrequent events, even in high-risk populations, and they happen unpredictably. To demonstrate that a new therapy meaningfully lowers fracture rates compared with standard of care, trials must wait for enough fracture events to accumulate to produce statistical confidence.

Because the event rate is low and influenced by many factors beyond bone strength — such as fall risk, age, and comorbidities — the signal-to-noise ratio is modest. As a result, Phase III osteoporosis trials typically enroll 10,000–16,000 participants and follow them for three to five years. The sheer scale and duration of these trials push costs to between $500 million and $1 billion. Thus, investment into osteoporosis drugs slowed not because the biology failed or drug candidates lacked promise, but because the cost of proving benefit became prohibitively high.

Osteoporosis is just one example where trial size and costs deter investment. But there is broader empirical evidence in this direction. A 2015 study examining oncology R&D found that hematological cancers — where the FDA accepts short-term surrogate endpoints in roughly 92 percent of approvals, allowing for shorter trials — attracted 112 percent more private R&D investment than solid tumors, where surrogate endpoints are used in only about half of cases. The authors traced this disparity to commercialization timelines. The shorter trials used for the former preserve more of a drug’s effective patent life, improving expected returns and drawing capital. Each one-year reduction in bringing a new therapy to market was estimated to increase R&D investment by between 7 and 23 percent.

If we want AI models to actually accelerate “therapeutic abundance,” then, we must first find ways to speed up these large validation trials. And to design better drugs in the first place, we must find ways to collect in-human data in early-stage “learning” trials much faster, too.

Regulatory Friction

The best way forward is to reduce operational and regulatory friction. AI tools can already help at the margins by automating submission drafting, improving site selection, matching patients more efficiently, and streamlining data workflows. But without deep regulatory reform, this is unlikely to shrink trial timelines or costs at scale.

One regulatory lever we could pull is to implement more high-quality surrogate endpoints. A clinical endpoint directly measures how a patient feels, functions, or survives — such as prevention of stroke or a reduction in fractures. A surrogate endpoint, by contrast, is a measurable biological marker or intermediate outcome that reliably predicts such clinical benefit. Instead of waiting years to observe clinical outcomes, trials that rely on surrogate endpoints can measure signals much earlier.

AI tools can contribute to the development of better surrogate endpoints, such as by identifying promising biomarkers, analyzing cross-trial datasets, and modeling causal relationships between intermediate signals and clinical outcomes. But here, too, technical capability is only part of the story. Institutional reform is likely to be the binding constraint. As my case study of the 12-year effort to qualify bone mineral density (BMD) as an endpoint for osteoporosis trials illustrates, the bottleneck was not scientific capability. Instead, the core barriers to faster progress were fragmented trial data scattered across sponsors, weak funding incentives for what is effectively a public good, and an unnecessarily lengthy and opaque regulatory pathway.²

For AI to generate high-quality candidates — the kind that might, one day, push success rates of drug candidates so high that trials become more of a formality — it also needs rich, dynamic data as input. But remember that such data can only come from trials in people (mice are nice, but most animal results simply do not translate.) This, in turn, creates a feedback loop: better AI models require better trial data, and better trial data requires running trials. The loop is only as fast as its slowest component, the trial itself.

A regulatory structure modeled after Australia’s Clinical Trial Notification (CTN) framework — administered by the Therapeutic Goods Administration — offers a concrete example of the kind of policy push that could speed up these types of trials. There, most early-phase trials proceed after approval by a Human Research Ethics Committee (HREC), with notification rather than pre-approval by the regulator. The regulator retains inspection powers and the authority to halt unsafe studies, but does not duplicate the scientific review already conducted by the clinician-scientists and toxicologists embedded in HRECs. The result is that clinical trial sites can begin giving drugs to patients much sooner (about two times faster than in the United States, according to informal interviews with industry leaders).

In the United States, by contrast, Phase I trials typically require submission of an Investigational New Drug (IND) application to the U.S. Food and Drug Administration before initiation. This dual review — by both an IRB and the federal regulator — creates redundancy that lengthens the feedback loop. A CTN-like model for Phase I trials could preserve safety oversight while shifting scientific and toxicological reviews to accredited, transparently governed IRBs with expanded expertise. The FDA would retain the power to inspect, impose clinical holds, and intervene in high-risk cases, such as for novel gene therapies. But for the majority of small-molecule first-in-human studies, the default could be notification rather than permission.

My criticisms are not meant to imply that AI is irrelevant to trials; that’s certainly not the case. But many of the bottlenecks that determine trial speed and cost are coordination, institutional and regulatory problems, and they cannot be solved by technology alone.

Australia’s system allows early-stage, investigator-led trials to start significantly faster—often twice as fast as in the U.S.—by bypassing the heavy-duty Investigational New Drug (IND) requirements intended for large-scale commercial trials. Australia has operated this way for 30 years with no difference in adverse safety events, yet the U.S. continues to regulate small, bespoke, tightly monitored studies as if they were massive Phase III trials.

It is, frankly, ridiculous that we have left this “Australian advantage” on the table for three decades while our own domestic pipelines have been throttled by administrative inertia. These small-scale, investigator-initiated trials are not just “preliminary” steps; they are the most information-dense experiments we can perform. In Australia, a researcher with a brilliant idea can move from bench to bedside in weeks.

U.S. companies themselves have increasingly started to move their early-stage trials to Australia where possible. So much so that in informal interviews with founders I have heard of the problem of the Australian trial system becoming “too clogged”.

The regulatory burden associated with launching clinical trials in the United States significantly harms American patients—especially those facing life-threatening illnesses such as cancer. Although the U.S. continues to lead the world in biomedical research and academic science, many terminal cancer patients struggle to access cutting-edge, experimental treatments. Instead, they are often forced to wait for full regulatory approval before these innovative therapies become available.

Implementing such changes would not only help current patients, but also future ones. It is perhaps the highest leverage thing we can do from a policy perspective to speed drug discovery. That is because it is precisely early-stage trials of this kind that enable something I call “Clinic-in-the-loop”: iterative learning in humans, the most important type of learning in biology.

We know this is crucial because some of the most revolutionary therapeutic breakthrough of the last decade (e.g. CAR-T cell therapy) was the hard-won product of two decades of small-scale, iterative “failed” early-stage trials.

To illustrate why these “information-dense” early-stage trials are the true engines of progress, I will draw heavily from an essay I wrote a couple of months ago for Asimov Press titled “Clinic-in-the-loop.”

Clinic-in-the-loop

A couple of months ago, I co-authored an essay with Jack Scannell, arguing that making trials more efficient and informative is essential to breaking Eroom’s Law. Critics of our essay, however, argued that making clinical trials more efficient risks treating biotechnology like a casino.

In their view, making it easier to run clinical trials would risk allowing more potentially harmful drugs to be tested in patients and, instead, biotechnologists should focus on making better drugs that are more likely to gain approval. These critics see Clinical Trial Abundance as accepting the status quo of drug development rather than challenging it.

But this is a misunderstanding.

In fact, Clinical Trial Abundance and better hypotheses for drugs are not merely compatible, but self-reinforcing. Faster testing in the clinic creates a feedback loop: ideas become trials, trials generate rich data (including both successes and failures), these data improve models, and better models inform the next generation of ideas. In this view, the clinic is not an endpoint of discovery but a central component of it.

To understand why clinical abundance is important, we must step outside the prevailing view of clinical testing as a mere “validation step” for scientific ideas. The familiar funnel metaphor of drug discovery, depicting a linear progression from basic science to regulatory approval, reinforces the flawed notion of clinical testing as a passive filter designed to screen pre-existing ideas. While this model is narrowly correct in a regulatory sense, it obscures the clinic’s role as an active engine of discovery.

The reality is that clinical trials rarely just deliver a “yes/no” verdict on a drug’s efficacy. Instead, the history of drug development shows that many successful therapies emerged only after initial versions failed in specific, informative ways. When a trial fails, it provides a unique physiological stress test that reveals exactly where a drug’s design fell short. By collecting data from “failed” trials, we can transform negative results into experimental corrections for the next iteration.

Consider CAR-T cell therapies. Once thought implausible or risky, CAR-T therapies now deliver long-term, treatment-free remissions in cancers where relapse had been almost certain.¹ In pediatric B-cell acute lymphoblastic leukemia (B-ALL) and aggressive B-cell lymphomas, for example, CAR-T has cured patients who, previously, had been given only months to live.

CAR-T therapy works by turning a patient’s own immune cells into living drugs. Doctors collect T cells from the blood, genetically reprogram them to recognize a protein on cancer cells, and reinfuse the modified T-cells into the patient. These engineered cells expand inside the body, move to tumor sites, and destroy malignant cells.

In 2017, the FDA approved Kymriah, the first CAR-T therapy, for children and young adults with relapsed or refractory B-ALL, a cancer of arrested development in which immature blood cells, specifically B cells, multiply out of control while failing to mature into viable immune cells. Relapsed B-ALL is the most severe form of the disease, because it means the cancer has returned after prior therapy. Even with aggressive care, only 10-20 percent of patients with relapsed or refractory B-ALL survived beyond five years.

Against this backdrop, Kymriah received accelerated approval from the FDA based on results from the Phase II ELIANA trial, a global, multicenter study sponsored by Novartis. In ELIANA, 82 percent of treated patients achieved complete remission, and subsequent follow-up analyses revealed that five-year survival rose to approximately 55-60 percent.

ELIANA was not a sudden breakthrough, though. It was, rather, the culmination of nearly two decades of clinical studies. During this period, CAR-T therapies evolved through repeated failure in the clinic, as careful studies of underwhelming results spurred new ideas to correct them. The ELIANA trial was led by investigators at the University of Pennsylvania, a group that had spent years studying CAR-T cells directly in patients well before regulatory approval.

In the mid-2000s, the earliest CAR-T therapies first entered human testing. And they emerged from a fundamental question: is it possible to engineer and redirect the T cell’s innate killing power against malignant cells?

Two well-established biological concepts made this seem plausible. First, T cells are extraordinarily cytotoxic.² However, their natural activation is governed by a “layered permission” system, meaning they cannot recognize targets directly, but must wait for other cells to process and present protein fragments in a precise molecular context. While this evolutionary safeguard keeps us from being attacked by our own immune system, it also provides cancer with many opportunities to evade detection by suppressing these signaling pathways.

To bypass these safeguards, researchers relied on a second insight: the ability of antibodies to bind directly and precisely to proteins on the surface of cells, called antigens. By equipping T cells with a synthetic Chimeric Antigen Receptor (CAR), bioengineers created a functional shortcut that bypassed the need for permission systems. This receptor uses antibody-style recognition to lock onto a cancer cell and is wired directly to CD3ζ, a signaling molecule that triggers the T cell’s internal “kill switch.” The moment the receptor engages its target, it flips the internal switch, activating the cell’s killing program.

In laboratory experiments, these first-generation CAR-Ts were formidable, displaying antigen recognition and potent killing power against tumor cell lines. Yet, this in vitro prowess vanished in patients and did not yield durable clinical responses. Understanding why this happened, though, was not simple. The failure could have been caused by a breakdown in in vivo antigen recognition, poor signaling strength, or other defects that only emerged after the cells were injected into the body.

Progress in understanding why came from treating first-in-human trials not simply as therapeutic attempts, but as opportunities to learn. These information-dense studies were conducted throughout the mid- and late-2000s and were relatively small (usually enrolling fewer than ten patients). However, they were designed to be maximally revealing. Researchers used many tools to monitor CAR-T persistence and activity in the body, turning information from a small number of patients into a mechanistic understanding.

One such tool was quantitative PCR (qPCR), a lab method that detects and counts specific DNA sequences, which allowed researchers to measure how many CAR-T cells were in patients’ blood. This showed that CAR-T cells successfully entered the body and were easy to detect after infusion. But the signal quickly faded, suggesting that the cells died off quickly. Other experiments shed light on the problem: CAR-T cells could recognize and eliminate cancer cells in patients — meaning antigen recognition was working — but their functional activity fell over time, suggesting that something in the blood was blocking them.

At this point, the diagnosis of why first-generation CAR-T therapies were failing matched long-standing insights from basic immunology. Decades of research had shown that T cells are not governed by a single on-off switch: signaling through CD3ζ provides only the first activation signal. To keep working, T cells need additional “co-stimulatory” signals,³ delivered through receptors such as CD28 or 4-1BB. First-generation CARs had been designed to deliver signal one without signal two, which explained their poor performance.

This hypothesis guided the next wave of clinical experiments, which investigated whether adding a costimulatory domain would make CAR-T cells more effective at clearing tumors in vivo.

The field-defining result came from Carl June’s group at the University of Pennsylvania. June and colleagues explored a costimulatory domain called 4-1BB. In a first-in-human study published in 2011, they treated a patient with chronic lymphocytic leukemia using CAR-T cells containing both CD3ζ and 4-1BB. They also administered a dose that was remarkably small by cell therapy standards at that time: just 1.5 × 10⁵ CAR-T cells per kilogram of body weight. (A first-generation CAR-T trial targeting renal cell carcinoma, published in 2013, used a dose more than 100-times higher.)

What followed was unprecedented. The CAR-T cells multiplied more than a thousandfold in patients, at their peak comprising a large fraction of immune cells in the blood. The CAR-T cells also persisted for months. Now, at last, CAR-T cells were a long-lived and self-maintaining immune population. Many cancer patients treated with these second-generation CAR-T therapies have achieved complete remission.

The next step, though, was asking whether the same CAR-T behavior would work in a faster, more aggressive blood cancer, such as B-ALL. In 2013, Carl June’s lab reported striking results in two children with relapsed B-ALL, again showing that the engineered T cells could multiply, persist, and drive cancer into remission.

All of these lessons were built into ELIANA, the study that ultimately supported Kymriah’s approval. Led by Stephan Grupp, who had treated the earliest pediatric patients and worked closely with June, ELIANA translated the early insights into standardized practice. This trial codified chemotherapy given before CAR-T infusion, scaled up cell manufacturing, and measured success using tools like qPCR.

Viewed through this lens, clinical trials are not an alternative to basic science, but rather a mechanism within it that closes a feedback loop. Foundational immunology, antibody engineering, and molecular biology made first-generation CAR-T cells possible in the first place, but early human trials quickly revealed that these designs were incomplete and suggested ways to fix them.

Yet theory alone did not prove this would work; the expansion and persistence observed with 4-1BB–based CAR-T cells came as a genuine surprise even to the therapy’s designers. “It was unexpected,” they reported, “that the very low dose of chimeric antigen receptor T cells that we infused would result in a clinically evident antitumor response.”

This shows why the “casino biotech” critique is flawed. It assumes that experimentation simply reveals a fixed probability of success. But trials can change those probabilities. When clinical testing is understood as part of a continuous feedback system, optimizing trial efficiency is not about accepting failure but about learning fast enough to make success more likely.

The most discovery-rich experiments are often not massive Phase III trials, either, but small, academic, investigator-initiated studies that sit close to the design loop.⁴ These are also the trials most burdened by regulatory, institutional, and manufacturing bottlenecks. And also the trials that adopting an Australia-like model would impact.

How the Australian model works in practice

The Australian Clinical Trial Notification (CTN) framework differs fundamentally from the current U.S. system in how authority is allocated between ethics committees and the national regulator, and when regulatory scrutiny occurs. In Australia, the Human Research Ethics Committees (HRECs) conduct the primary scientific, ethical, and safety review of a proposed early-stage clinical trial. Once an HREC approves a study, the sponsor submits a streamlined online notification to the Therapeutic Goods Administration (TGA), and the trial may begin shortly thereafter.

There is no requirement to submit a full Investigational New Drug–style dossier to the regulator before initiation for these early-stage, bespoke trials. By contrast, in the United States, sponsors must submit a comprehensive IND application to the Food and Drug Administration (FDA) under 21 CFR 312 before beginning a clinical trial. The FDA performs its own scientific, toxicology, and chemistry, manufacturing, and controls (CMC) review, and Institutional Review Boards (IRBs) separately review ethical considerations under 45 CFR 46. This results in parallel review streams and often significantly longer timelines before a trial may commence.

In practice, the Australian model delivers what can be described as stage-appropriate review. Regulatory scrutiny is not diminished. Rather, it is calibrated to the specific risks and uncertainties that characterize that phase of development. Early-stage trials have high failure rates, and many investigational products never move beyond first-in-human studies. Requiring extensive upfront documentation and full manufacturing packages for programs that are likely to stop early adds significant cost and delay without meaningfully improving participant safety at that stage.

Early development is a learning process. Sometimes even the underlying hypothesis is reconsidered as real human data become available. A regulatory framework that allows this kind of rapid, well-monitored iteration better reflects how drug development actually works. By focusing oversight on the risks that are most relevant in early trials—rather than imposing requirements designed for later-phase studies—the Australian system promotes efficient evidence generation while still maintaining strong safety protections for participants.

Australian site activation averages approximately two months from final protocol, and TGA acknowledgment of CTN submissions occurs within days. In the United States, IND review and resolution of regulatory questions frequently extend for several months before enrollment can begin. Such a difference in timelines can be particularly important for an early-stage biotech company struggling to survive on limited cash.

Why has this not been done before?

Knowing that Australia has operated this framework safely for 30 years—and that these trials are the highest-leverage instruments in biology—the question becomes: why has this been left on the table? Why is the U.S. biotech engine idling at a red light that doesn’t need to be there?

The answer lies in a fundamental distortion of Washington’s health policy priorities. In D.C., “health policy” is almost exclusively synonymous with demand-side economics: Medicare, drug pricing, and reimbursement. The supply-side—the actual machinery of how we discover and test new biology—is remarkably ignored.

This policy stagnation is a classic “tragedy of the commons” problem. There is no natural, deep-pocketed constituency for early-stage trial reform:

• Big Pharma uses its political capital primarily to protect reimbursement and pricing. For a multi-billion-dollar incumbent, saving a few million dollars or six months on an early-stage trial is a rounding error. Furthermore, their model is often to let others take the early risks and then acquire the survivors once they’ve reached later-stage validation, meaning that they often are not the ones carrying out these trials. By the time a biotech has produced a drug that looks promising, a few millions extra due to early trial costs is nothing.

• Small Biotech would be a more direct beneficiary of these reforms, but these startups rarely have the time, money, or lobbying presence to advocate for systemic change in D.C. They are too busy trying to survive the very bottlenecks we are discussing.

• The Commons. Of course, the primary beneficiaries of these reforms are ultimately The Commons. If more drugs are tested in early-stage trials, all of us get more and better drugs. However, those doing advocacy on behalf of the commons have often had an approach that focuses much more on the distribution of what exists (“demand-side”) than creating abundance (so working at the “supply-side” level).

This is why the emergence of the “Progress” community is so vital. Organizations like The Roots of Progress and IFP are filling the gap left by this tragedy of the commons. They are doing the “un-incentivized” work of advocating for the structural plumbing of science—the boring, high-leverage regulatory reforms that Big Pharma ignores but that the future of medicine depends on.

We need progress-minded individuals to step into these gaps, supported by institutions that understand that supply-side policy reforms centred around Clinical Trial Abundance are key to the future of medicine. Senator Cassidy’s report is a rare sign that this message is finally reaching the most important echelons and it is an opportunity we cannot afford to waste.

I’d like to offer some thoughts of my own, particularly around the timing of this guidance and what it might mean. FDA’s drug center has been facing pressure to embrace Bayesian statistics for decades. I worked at the FDA for over a decade, and saw frequent pushes to get a Bayesian guidance released. So why is the guidance coming out now? And will it actually change how clinical trials are designed?

Why go Bayesian?

Before we dig into those questions, it’s helpful to understand the statistical approach drug companies have traditionally used to prove that their drugs work. For decades, companies have used an approach rooted in frequentist (non-Bayesian) statistics: they conduct two, “pivotal” randomized controlled trials, each designed to prove that the drug has a clinically meaningful benefit. The trials are considered successful if each produces a two-sided p-value no greater than 0.05. While FDA has never officially required trials to meet this standard, in the past they have endorsed this approach.

Advocates for the use of Bayesian statistics would point out several problems with this traditional frequentist approach. First, there is the p-value itself, which is defined as “the probability of getting results at least as extreme as the ones you observed, given that the null hypothesis [of no drug effect] is correct”. It’s a concept that strains the comprehension of even expert scientists (I had to copy the definition from Wikipedia to make sure I got it right, which is what I suspect most responsible scientists do when asked what a p-value is). P-values and frequentist statistics are useful in drug regulation, but Bayesian methods give us a simpler and useful metric: the probability, given the data observed in the study, that the drug meets a threshold of effectiveness. That is a value that is easier to understand and often more useful for decision-making.

A more significant shortcoming of the frequentist approach, from a Bayesian perspective, is its tendency to discard useful information. When evaluating whether a drug should be approved, the FDA has the opportunity to evaluate multiple sources of evidence: There are, of course, the drug company’s pivotal studies. But they might also examine clinical studies from earlier phases, studies of similar drugs for the same condition, or studies of the same drug for similar conditions. If the drug is already being used in the clinic, FDA might also want to look at patients’ experiences on the drug in the real world. All of these sources of evidence are valuable, but the traditional frequentist paradigm discards much of it, relying solely on those pivotal studies to provide evidence of the drug’s effectiveness.

Bayesian statistics can help with these problems: It lets reviewers consider all of the data about a drug – not just the data collected in a given study – and it produces results that are easier to interpret and more relevant to regulators, clinicians, and patients. These benefits come with caveats: the results of a Bayesian study depend on modeling decisions made upfront, including the choice of the Bayesian prior (more on that below). That’s why FDA’s guidance suggests that Bayesian studies should demonstrate strong “frequentist operating characteristics” and stress-test assumptions. But with rigor and careful planning, Bayesian statistics can help us make better use of the information we have.

So why has this taken so long?

The proximate reason we are seeing this guidance now is that FDA committed to release the guidance in its 2022 user fee negotiations with industry. But the truth is, this guidance could probably have been released sooner. FDA has been discussing the use of Bayesian statistics for decades – FDA’s medical device center published its own Bayesian guidance nearly 20 years ago. I think the real reason this guidance is coming out is that, for the agency, the benefit of applying a Bayesian approach to drugs finally outweighs the downsides.

And despite its merits, Bayesian statistics does have a big downside: it requires the construction of the Bayesian prior probability distribution, or prior. The prior describes the range of effect sizes we think are plausible before the trial begins based on existing evidence, and is crucial in determining how the study data will be interpreted. While there are best practices in constructing the prior, it inherently requires some degree of subjective judgment on the part of the study designers.

It’s easy to see how the prior might raise red flags for regulators. If the prior is subjective, how do you decide what it should be? And how do you keep drug companies from gaming the system by choosing a favorable prior? Yet while these concerns are real, I don’t think they’re the main reason that FDA has been hesitant to embrace Bayesian statistics. After all, there is always an element of subjectivity in drug review. FDA tries to make its standards clear, but they acknowledge that they must make subjective judgments on what evidence should be collected and how it should be weighed. And the FDA is perfectly willing and able to scrutinize study design choices and the study data itself. They’re not likely to be fooled by drug companies who play games with statistics – even if the statistics are Bayesian.

If I had to guess, I suspect the biggest problem for FDA was not the subjectivity of the prior; it was the exercise of putting numbers behind those subjective judgments. For fans of Bayesian statistics (and legions of statistics nerds), the fact that the Bayesian prior can be quantified is one of its greatest strengths. Capturing relevant information in a Bayesian prior feels more rigorous and rational than relying solely on subjective judgment – and it’s much better than simply throwing that relevant information away.

But I suspect this is not how most normal people – FDA reviewers included – feel. The FDA has long stressed the importance of “clinical judgment” in reviews. The Bayesian prior threatens that. After all, if a reviewers’ clinical judgment is a key factor in review decisions, any attempt to translate that clinical judgment into numbers risks diluting and distorting that judgment.

In the past, FDA has been even more explicit in rejecting the quantification of its review decisions. In a 2013 report, they considered and rejected the idea of using numerical weights of benefits and risks to help them review drugs. The process of assigning weights, they argued, involved “numerous judgments that are at best debatable and at worst arbitrary.” While the construction of a Bayesian prior does not preclude reviewers from exercising judgment, it does probably elicit a similarly negative reaction.

And yet, here we are, with the Bayesian guidance in hand. The guidance even goes so far as to suggest that companies could explicitly quantify benefits and risks in their trial in the form of a Bayesian “loss function.”

Why the shift? Perhaps it’s because the alternative approach has grown less tenable.

FDA’s existing statistical approach no longer made sense

Against the “debatable and arbitrary” assumptions required by Bayesian statistics, the perceived simplicity, objectivity, and consistency of a frequentist approach must have seemed appealing to the FDA. In exchange for ignoring prior data, the frequentist approach gives us a consistent statistical approach with a consistent interpretation. The frequentist approach must have been particularly appealing to FDA in the 1960s, when it first introduced its efficacy standard. At that time, the agency feared being barraged by poorly conducted studies of questionable products. Indeed, FDA articulated this fear as recently as 1998 in its guidance on how it evaluates drug effectiveness:

The inherent variability in biological systems may produce a positive trial result by chance alone. This possibility is acknowledged, and quantified to some extent, in the statistical evaluation of the result of a single efficacy trial. It should be noted, however, that hundreds of randomized clinical efficacy trials are conducted each year with the intent of submitting favorable results to FDA. Even if all drugs tested in such trials were ineffective, one would expect one in forty of those trials to “demonstrate” efficacy by chance alone at conventional levels of statistical significance.

In other words, the agency needed to filter out the junk from the “hundreds” of randomized studies they might see each year, and relied on frequentist statistics, hypothesis testing, and replication to help them achieve this.

Nowadays, the circumstances the agency faces are different. If the agency ever faced a barrage of “hundreds” of randomized trials, that is no longer the case. In 2024, FDA approved 50 new molecular entities on the basis of only 75 trials. And it probably felt unfair to assume that most or all of the drugs that underwent these trials were ineffective (even if many would ultimately fail); many of those trials were done only after extensive study, including prior use and research in humans.

The frequentist framework no longer made sense. Even before the Bayesian guidance was released, FDA understood that, despite their reliance on frequentist statistics, these trials could not be reviewed in an evidentiary vacuum. As far back as the 1990s and early 2000s, FDA has found ways to “borrow” from prior information to support efficacy determinations: FDA has made use of “seamless” phase 2/3 trials in cancer, natural history studies, and even mechanistic data to support efficacy determinations. They usually described this as making use of “confirmatory evidence” or “ the totality of evidence”. Now, FDA has departed further from the traditional approach, requiring only one trial instead of two.

More broadly, the key regulatory question has shifted: instead of “how do we filter out the junk science” – a goal well-supported by frequentist methods – FDA and drug companies are trying to use scarce clinical data as efficiently as possible to make informed determinations about each drug that crosses its desk. That’s where Bayesian statistics shines.

The choice FDA now faces is not whether or not to rely on prior evidence; it’s whether to do so awkwardly, in its current frequentist framework – in which that borrowing is implicit and unstructured; or using the more structured approach offered by Bayesian statistics. The publication of this guidance suggests that FDA is starting to embrace the more structured approach.

Will this guidance lead to change?

Are we about to embark on a new Bayesian era at FDA? Perhaps, but first, drug companies need to actually adopt the methods described in the guidance.

I expect the process to be slow. Despite FDA’s endorsement, this is still just a draft guidance, and few drug companies will want to be among the first early-adopters to try out new Bayesian approaches with agency review teams. Drug companies have famously risk-averse cultures, particularly when it comes to running their most expensive and high-stakes pivotal trials. And even though FDA has signaled that it is amenable to Bayesian approaches, most will wait to adopt them until they see FDA approving applications that make use of them.

Adoption will also be slowed by a lack of capacity and expertise. Drug companies and the FDA will both need to train their staffs and get familiar with the technical details of designing and approving Bayesian studies, negotiating over priors, and understanding the benefits and limits of the approach. In the meantime, drug companies may find themselves more comfortable working within the existing frequentist approach, knowing that the agency has grown increasingly comfortable with accepting “confirmatory evidence” of effectiveness in lieu of data from trials.

But in the longer term, I am betting that Bayesian approaches will become far more common. Drug companies are going to find it difficult to resist the opportunity to make more efficient use of limited clinical evidence; particularly in rare diseases, pediatric indications, and other areas where evidence is particularly scarce. If enough pioneering companies are willing to take the chance on something new, others will follow. And if we find new ways to strengthen the quality of prior evidence, whether through clinically validated AI-driven “virtual cells” or simply faster and more abundant early-phase human trials, the case for Bayesian methods will grow even stronger.

I’d like to thank my co-bloggers , , , and for their input on this piece.

That uncertainty doesn’t just slow individual programs, but also reshapes behavior across the biotech ecosystem in a negative way. Lack of regulatory clarity fosters a culture of defensive decision-making, where companies avoid innovative clinical development strategies and default to more conservative, expensive paths simply because they lack confidence in how the FDA might respond. Over time, this dynamic drives up trial costs, prolongs timelines, and ultimately reduces the number of truly innovative therapies that reach patients.

Unfortunately, a decision made by the FDA very recently threatens to make an already existing problem even worse.

Today, the U.S. Food and Drug Administration issued a refusal-to-file letter to Boston-based biotech Moderna regarding its Phase III trial for an mRNA influenza vaccine in adults 50 and older. The letter was signed by Dr. Vinay Prasad, the head of CBER (Centre for Biologics Evaluation and Research)—an unusual move at this stage of the process, suggesting the decision was not routine.

A refusal-to-file (RTF) letter means the FDA has determined that a company’s application is not sufficiently complete or adequate to even begin formal review. It halts the approval process before any full evaluation of safety and efficacy takes place, forcing the sponsor to address the cited deficiencies and resubmit.

The justification offered by the FDA is that Moderna used the wrong comparator in adults over 65. Specifically, the agency argues that because higher-dose or adjuvanted influenza vaccines are recommended for older adults, Moderna’s use of a standard-dose comparator rendered the trial not “adequate and well-controlled.”

If that requirement had been clear from the outset, this would be an unremarkable story about trial design. Companies adapt to clear standards and requirements from regulators all the time. But Moderna’s CEO, Stéphane Bancel, says the FDA had previously signaled that a licensed standard-dose comparator would be acceptable—and has only now changed its position after the trial was completed. If that is accurate, it would mean that roughly $750 million—what the New York Times reports the trial cost—was effectively committed under regulatory assumptions that no longer hold.

Adding to the confusion, entrepreneur Dr. Jing Liang noted on X that publicly available trial data appear to show Moderna conducted analyses involving higher-dose comparators in older adults. If accurate, it would raise further questions about the basis for issuing a refusal-to-file letter in the first place.

No matter which account ultimately proves correct, this episode leaves us with two uncomfortable possibilities. If the FDA changed its regulatory position after years of clinical development, that reinforces a persistent concern about regulatory inconsistency. If, alternatively, Moderna met the agency’s highest level of stated expectations, actually did use a high-dose comparator and still received a refusal-to-file letter, the implications are even more serious, raising questions about transparency and internal coherence at the agency.

In this post, I will focus on the “least bad” scenario—that the FDA revised its stance on a trial design it had previously indicated was acceptable. Even that interpretation carries consequences far beyond a single influenza vaccine. When regulatory expectations shift late in development, companies internalize the lesson and design more defensively, avoid innovative comparators and add layers of redundancy. Over time, this entrenches a culture of safetyism and regulatory aversion. Sponsors begin optimizing not for better science, but for minimizing the chance of procedural surprise. In time, this leads to a ballooning of costs.

Why regulatory inconsistency is bad

Even if we assume the “least bad” scenario—that the FDA has indicated a lower-dose comparator would be acceptable—it still reinforces one of the agency’s most persistent structural problems: regulatory inconsistency and opacity.

Translating science into a clinical product already takes more than a decade and billions of dollars and only about 10% of drug development programs succeed. Most of that time and capital is spent in clinical development—running trials, scaling manufacturing, and interacting with the FDA. Yet over a 10+ year program, a company may have only three to five formal milestone meetings with regulators. Those meetings determine pivotal trial design, endpoints, manufacturing standards, and statistical plans.

Given that formal meetings with regulators are limited and FDA guidance is often broad and caveated, companies therefore make pivotal decisions under real uncertainty. And when the stakes involve hundreds of millions of dollars and years of work, the rational response is caution. This is what Adam Kroetsch meant when he said that “vague regulation breeds safetyism.” Sponsors layer on extra monitoring, avoid novel trial designs, and default to conventional manufacturing processes even when more efficient technologies exist. Each individual decision makes sense. Collectively, they make drug development slower and more expensive.

The slow adoption of risk-based monitoring (RBM) is a clear example of the downsides of regulatory opacity. RBM focuses oversight on the highest-risk elements of a study and could reduce trial costs by up to 30% without compromising safety or data integrity. The FDA has encouraged its use for years. Yet, despite this clear advantage, adoption remains uneven. Surveys show that sponsors hesitate because they worry regulators may not accept RBM in practice—even if official guidance supports it.

The same fear-driven conservatism appears in other areas of trial design. One study found that roughly one-third of data collected in clinical trials may be redundant. Sponsors collect this data due to mistakenly believing this is what regulators might ask for. That redundancy translates directly into higher costs, and longer trial timelines.

It’s trying to solve the problem of regulatory opacity, which has such a negative downstream impact on innovation, that has led me to launch, with OneDaySooner, the openFDA Fund—a centralized repository of past regulatory interactions designed to bring greater transparency to the often opaque FDA decision-making process.

The future

Unfortunately, this is not the first late-stage reversal under the current FDA. In November 2025, biotech company uniQure announced that the FDA had reversed its prior position on the company’s gene therapy AMT-130 for Huntington’s disease. Although the agency had previously indicated that Phase 1/2 data—compared against an external natural history control—could support a Biologics License Application (BLA) under the accelerated approval pathway, it now determined that the existing data are insufficient for submission.

The shift came despite promising results showing statistically significant slowing of disease progression and supportive biomarker improvements. CEO Matt Kapusta expressed surprise, noting that the feedback represented a “drastic change” from guidance provided in late 2024.

The current FDA leadership has openly acknowledged growing competitive pressure from China and has put forward a number of thoughtful proposals to modernize the regulatory framework. But reform does not end at changing rules. These rules must also be predictable. Regulatory clarity and consistency are foundational and as important as the quality of the regulations themselves. For the sake of the US biotech ecosystem, I hope that the FDA will reverse course in what seems a trend towards decreased regulatory consistency and clarity.

The first statin, mevastatin, was discovered in 1976 by Akira Endo, a biochemist at Sankyo pharmaceuticals in Japan, from a fungal mold growing on rice samples at a grain shop in Kyoto.1

The drug was clearly effective in reducing cholesterol levels, but in 1980, Sankyo abandoned its clinical trials after studies in dogs appeared to show intestinal tumors. The details of these findings were never formally reported; at the time, there were only rumors about what had led the company to shut down its trials.

Meanwhile, Merck was testing a nearly identical compound2, lovastatin, and heard about the decision. They expected it would be a multimillion dollar drug, and so took the sudden halt seriously: what if lovastatin would cause the same side effects?

Merck paused its own trials, and asked Sankyo for further details, but the latter declined to share them.3 So Merck conducted further toxicology studies of their own to understand whether the risks were real, which eventually showed the changes were benign and could be reversed. Three years passed before Merck resumed their clinical trials. When they were completed, they showed a large reduction in blood cholesterol reduction and few side effects. Lovastatin (now known as ‘Mevacor’) became the first statin approved in 1987.4

Decades later, after millions of people have been treated and monitored, the evidence shows no increased risk of cancers from statins. They have likely saved millions of lives by reducing the risks of heart attacks and strokes by roughly 20%, and annual mortality rates by 10%.

I’ve been wondering, what if Sankyo had shared its data publicly?

It’s possible that researchers might have analyzed the data further and reached the conclusions sooner. Merck could have avoided redundant studies, and both companies might have reached the finish line, speeding up the arrival of a drug class that would go on to save millions of lives.

Another possibility is that Merck might have instead funded trials for another drug candidate – one that was less similar in structure to Sankyo’s drug, as some within Merck suggested doing. Alternatively, if Merck had never heard about Sankyo’s halted trials at all, they might simply have proceeded, and still correctly found no evidence of harm in their own clinical studies in human patients.

The counterfactual from the story is ambiguous, but it shows that pharmaceutical companies respond to data from other firms. It also highlights a tension: transparency, or even partial transparency, can affect experimentation. When researchers or firms learn early that a similar approach has failed, they may shift toward safer, more predictable projects, at the cost of unexpected successes.

At the same time, access to the underlying data would have allowed other scientists to scrutinize the findings directly, rather than rely on rumors. That would have allowed for better decisions about what precisely to investigate further – helping understand which doses it occurred at and what types of cellular changes were seen, for example – and make more informed decisions about their own research and development plans.

Subscribe now

Medicine is an area where transparency has been considered valuable for a long time, and genuine progress has been made in improving it, as I’ll describe in a future post. The consequences extend well beyond the research phase; they can shape decisions about which drugs regulators approve, which treatments insurers cover, and which treatments physicians prescribe.

The stakes are unusually high. In the case of drugs like statins, such decisions touch tens of millions of patients each year5, with hundreds of millions – sometimes billions – of dollars resting on the results of trials.

Transparency at various stages of the process can shape decisions on whether to continue research into drugs, test them in clinical trials, approve them, and how to price them.

If transparency matters this much, what kind of data should be available in practice?

It can seem like an ambitious reform to ask for data at the level of individual patients so the findings can be verified and potentially re-analyzed, because research can involve sensitive personal information, and preparing datasets and documentation for public sharing can be time consuming.

But most highly-cited clinical trials say they will share data, and many say this will be anonymized data from individual patients. Moreover, a few recent efforts have tried to address this problem, including Vivli.org and ClinicalStudyDataRequest.com, with data platforms where clinical trial researchers can deposit their datasets securely.

In practice, however, data from clinical trials remains largely inaccessible and enforcement by journals is weak.

I think there are probably many benefits of transparency in clinical trials, beyond verifying data. They include:

• Better meta-analysis. When more trial results are available, researchers can perform meta-analyses that combine evidence across studies. This improves precision and allows for stronger conclusions than single studies can support, for example by helping to identify effects on rare outcomes, such as the effects of flu vaccines in reducing mortality rates, or rare but serious harms.

  • A recent meta-analysis of 19 clinical trials on statins, for example, found no increase in the risks of most side effects listed on the drugs’ labels (such as brain fog, diarrhea, pain, vision impairment, and 58 other outcomes that had showed no difference in risk between the drug and placebo). Only a few side effects were validated, such as muscle weakness, rhabdomyolysis in rare cases, and a slight increase in diabetes. With these conclusions, the researchers suggested that drug labels should be updated.

• Understanding inconsistent results. Clinical trials studying similar treatments sometimes reach different conclusions, which leaves uncertainty about whether the differences reflect chance, patient populations, study design, or other factors. By pooling data across studies, researchers can explore these sources of variation more systematically. They could, for example, try to identify when apparent contradictions stem from targeting different biological mechanisms.

• Further exploration. Transparency makes it easier to ask new questions of old data. Researchers may want to explore hypotheses that weren’t part of the original study, or revisit results in light of new evidence. When prior trial data are available, many of these questions can be answered without launching entirely new trials. This might include, for instance, when drugs initially developed for one condition show unexpected benefits for another.

• Better clinical decision-making. Doctors often have to choose between many drugs for the same condition, without having sufficient data for head-to-head comparisons. Individual trials can rarely answer questions like “Which drug performs best for these patients?” But with more data, techniques like network meta-analyses can help indirectly compare multiple treatments against each other, or derive better conclusions on how to tailor decisions to patients’ characteristics.

• Learning how to run trials better. Pooling data across many trials could also help answer questions about studies’ operational characteristics: How does remote monitoring compare with on-site testing? Which trial sites reliably deliver high-quality data? Which eligibility criteria slow down recruitment? Which practices reduce the chances of patients dropping out of studies? These questions could help design trials more efficiently, even though individual research groups rarely run enough trials to study these questions on their own.

• Reducing redundancy and wasted effort. Finally, transparency can help avoid repeating failures. Researchers can learn from what succeeded or failed to refine their hypotheses and improve their chances of developing effective drugs in the future. More recent evidence suggests that when reporting the headline results of trials became mandatory, pharmaceutical companies actively monitored competitors’ results and adjusted their own research plans accordingly, avoiding similar studies.

And the value of AI in performing many of these analyses is also constrained by access to the underlying data; the benefits depend on the availability and quality of data available to study.

Taken together, these benefits point to a broader function of transparency in medical research: it allows knowledge to accumulate more efficiently.

By making individual patient data available, studies can move beyond fixed headline results and become part of a cumulative evidence base, giving researchers and clinicians a fuller picture of data that they can use to answer broader questions.

This flow of information can influence the pace of basic research, shape drug development, guide clinical decisions, and ultimately, affect the health of millions of people.

Share

Subscribe now

Clinical trials are how new drugs are tested before becoming widely available. In most countries, like the US and UK, where we live, trials are expensive and slow. Effective drugs can be stuck in clinical trials for years, sometimes a decade or more, before the results are clear and those drugs are available to patients who need them. Clinical trials are also generally treated as a high-stakes, one-shot test to produce a final binary decision on whether a drug works, rather than as part of an ongoing evidence loop that could help accelerate innovation.

We think things could be different. Clinical trials could run more efficiently through modular components, innovative trial design, streamlined ethical approval, faster recruitment, and many other improvements. They could collect more informative data to help discover new, effective biomarkers, not just confirm older ones. And by sharing the data from them, they could help improve drugs in the future too. All of this could allow us to test more drugs in parallel and find out what works (or doesn’t) faster.

Despite major advances in AI, we believe that testing in humans – not just in animals or simulations – will remain essential for understanding effects in complex biological systems like humans, and effects that may vary between individuals and produce unexpected side effects.

Each of us – Adam, Manjari, Ruxandra, Saloni, and Witold – has been writing about these ideas on and off for years. We’ve started this blog because we want to have a common platform where we can publish more frequently, respond to news and policy change, share thoughts and start discussions, and seed ideas that can be refined and explored further.

Subscribe now

Here’s a short introduction to each of us:

is interested in accelerating medical innovation and scaling up efforts in global health. She’s previously written about why randomized controlled trials matter, why it took 141 years to develop a malaria vaccine, a prediction of why we’d get Covid vaccines in a year, and some ideas we can take from pandemic clinical trials to speed up innovation more broadly.

She works full time at Works in Progress magazine as an editor and writer, advises Coefficient Giving on clinical trial reform, and hosts the podcast Hard Drugs with Jacob Trefethen. Previously, she was a researcher at Our World in Data on global health and medicine, and holds a PhD in psychiatric genetics from King’s College London and the University of Hong Kong.

Kroetsch is interested in system-level reforms to make trials more abundant. Through the Clinical Trials Efficiency Project, he is exploring how we can make trials faster and cheaper through better policies, regulations, incentives, and infrastructure. He has written about why trials are so expensive and what clinical trial abundance might look like. He has also written about the FDA’s history and how the FDA evaluates clinical trial evidence.

Adam publishes his work on his Substack, Policy and Practice. Previously, he worked at the FDA and the Duke-Margolis Institute for Health Policy. He holds a Masters in Public Policy and Management from Carnegie Mellon University.

is interested in accelerating methodological falsification in science. As a BRAINS fellow at Speculative Technologies she explored an ambitious program to improve preclinical forecasting evaluations in drug development. Manjari writes about biomarkers, neuroimaging, and statistics at Neurostats. She runs the Surrogate Science Project, a new community effort to bring the formal logic of good proxy development from biostatistics, econometrics and machine learning to all fields.

As a consultant, she advises biotech startups on early biomarker R&D, and collaborates with tech companies to improve scientific causal reasoning of AI agents. As a member of industry working groups in pharmaceutical statistics, she seeks to bring statistical rigor to earlier stages of biomarker development. She is a sought after invited speaker in several areas of biomedical research including multi-modal brain stimulation, psychiatric neuroimaging biomarkers, theoretical metascience, and statistical innovation in biopharma. Manjari’s early academic research focused on signal processing and nonparametric decision theory. Her subsequent graduate and postdoctoral work blended causal machine learning with biostatistics to understand and intervene on complex systems found in biology and medicine. She advanced uncertainty quantification and decision science for gene therapy at Dyno Therapeutics. She holds a PhD in Electrical Engineering from Rice University; her graduate work received the ENAR distinguished paper award in Biostatistics.

is interested in how we can reverse Eroom’s Law – the observation that pharmaceutical R&D efficiency declined, despite staggering advances in basic science. Convinced that making clinical trials more efficient would play a key role, she started the Clinical Trial Abundance Initiative, a policy effort focused on clinical trials at the Institute for Progress (IFP) in 2024. She has also written extensively about fertility and biomedical progress more broadly.

She is now a Fellow at Renaissance Philanthropy, where she continues to work on Clinical Trial Abundance. Her focus at the moment is on streamlining Phase I trials, FDA transparency and Drug Development Tools (DDTs). You can read her policy work on the IFP website. She holds a PhD in Genomics from Sanger Institute, University of Cambridge.

is a statistician working on problems in global health, epidemiology, and development economics. Witold’s current research revolves around making vaccines more efficient, meta-analysis, low-cost life-saving interventions, and metascience.

Witold is a Consulting Director at Development Innovation Lab at University of Chicago, where he works on designing randomised controlled trials, a Senior Fellow at the Institute for Progress and a Research Fellow at the Center for Global Development. Previously he spent over a decade working as a consultant in the pharmaceutical industry. His PhD was on Bayesian approaches in biostatistics.

We hope you enjoy reading!

– Saloni, Adam, Manjari, Ruxandra and Witold