Yet at the same time, I suspect that for the typical viewer it was hard to tell what that exciting revolutionary thing actually was. The headline made it seem simple: The FDA was piloting “Real-Time Clinical Trials… that will report endpoints and data signals to the agency in real time.” Today, FDA sees trial results only after the drug companies who run the trials, called sponsors, have collected the data, cleaned it, analyzed it, summarized it, and packaged it for review. This new approach could allow the agency to receive useful information earlier, reducing the amount of “dead time” between study phases.

But if you watched the FDA’s press conference, you might start to think that this announcement was about something else entirely. During the event, the pilot participants each shared some remarks. Most of them did not focus on real-time submission of data to FDA. Instead, they spoke about other things: a representative from Amgen talked about pragmatic trials. A researcher at UPenn talked about how the new pilot would ease the burden on their staff. Kent Thoelke, CEO of Paradigm Health, the technology developer for the pilot, spoke about the importance of bringing trials into community and rural settings. This is all great stuff. But what does it have to do with real-time submission of data to the FDA?

But there is a good reason that the pilot participants were speaking in such sweeping terms. The benefits of real-time trials could go far beyond just faster submission to the FDA. Real-time clinical trials represent a new, more efficient way of running trials that affects everybody involved—including study site researchers and patients. Beyond just reducing “dead time”, they hold the promise of making trials faster, less expensive, and more accessible to patients. In this post, I’ll explain why this program could change how trials are run, and what it will take to make it successful.

How trials got re-engineered to allow for real-time data

To understand why FDA’s pilot is potentially transformative, it’s helpful to understand how trials are run today, and how the real-time trials pilot changes that. As long-time readers of this blog already know, today’s trials are deeply inefficient and labor-intensive, largely owing to the cumbersome way in which we collect and clean trial data. Today, trial data is collected from a broad range of different “source” systems; EHRs, lab systems, and—far too often—on paper. Then, traditionally, those data are manually transcribed into an electronic data capture system (EDC): the system that is used to collate and transmit the trial data to the drug company. This manual transcription is time-consuming, error-prone, and can introduce lag between when data is first captured and when it’s submitted.

After the drug company receives the data from the EDC, it undergoes multiple layers of quality checks, which add more work and introduce even more delay. The drug company reviews the data for discrepancies and missing values, and if it finds an issue it will return a “query” to the study site. The site will then need to go back to the source documentation to resolve the issue. Then, there’s an even more labor-intensive step: A study monitor, hired by the sponsor, will fly out to the study site and review all of the data captured in the EDC, comparing it against the source data.

Meanwhile, the trial is collecting adverse events and endpoints. Each of these undergo extensive manual review. Adverse events are reviewed by the sponsor to determine whether they are serious, unexpected, and plausibly associated with the study drug. If so, they are shared with the FDA immediately. Endpoints are often reviewed by expert adjudicators, who confirm that the study endpoint was actually reached.

This is why real-time submission of data to the FDA is difficult. In the current model, each piece of data needs to be transcribed, validated, verified, and potentially adjudicated before it can be analyzed. The process can be long and drawn out. No sponsor or site would want to share “raw” data with FDA before this process was completed. This process is also a big part of the reason that we see so much dead time between phases. After the last patient visits a study site, there is still much work to be done: unaddressed queries, source data verifications, and adjudication. Only after that work is done for every data element will the sponsor sign off on a “database lock”, at which point the actual analysis of the trial data begins.

The data bottleneck begets a decision bottleneck: between study phases, drug companies must review their compiled data and decide what they want to do next. So after the database lock, the sponsor will analyze the data and make their critical go/no-go decision. They’ve probably already given some thought to how they will design their next phase, but after looking at the data they’ll confirm their design choices and share them with the FDA. They often will need to schedule a meeting with the FDA to share their proposed approach and make any modifications the FDA recommends (these interactions are especially common before phase III begins). Finally the drug company will start the long and laborious process of recruiting sites and patients for the next phase. Today, each of these steps is done sequentially, creating costly delays between phases.

How the RTCT pilot changes things

But what if we could do away with many of those steps? What if, instead of data being entered into disparate systems, the source data was entered electronically, then compiled and automatically imported into the sponsor’s electronic data capture system (or just sent directly to the sponsor)? Then, much of the manual work of data transcription, checking, and verification could be automated. And thanks to AI, we can even automate the downstream work: After the data is submitted to the sponsor and validated, AI could help classify the seriousness of adverse events and even adjudicate the endpoints. Data that previously took weeks to enter, clean, and validate can be available nearly instantly to analyze—and, of course, to share with the FDA.

While I can’t claim to know precisely what Paradigm Health’s software is doing in these pilots, I suspect it is automating many of these tasks. And those kinds of automations yield many benefits beyond just enabling real-time submission to the FDA. We heard about many of them at the press conference. First off, the approach integrates data from disparate sources, reducing manual transcription work and errors. Sites will spend far less time dealing with source data verification and time-consuming queries. This saves an enormous amount of time and effort. Source data verification alone is believed to consume 25-40% of the trial budget.

This explains why FDA’s press conference kept drifting into the topic of site burden, community access, and pragmatic trials. The same infrastructure that makes real-time clinical trials possible also makes it easier for sites that don’t traditionally do clinical trials—like community and rural sites—to involve their patients in research. With much of the work automated, participation is simpler, especially when the trials themselves are designed to make use of the data collected in routine care.

Finally, when all of the systems that collect and use trial data are integrated and automated, it opens the door for even more innovation. Rapid, automated data analysis could make it easier to run adaptive and Bayesian trials by providing quicker feedback to sponsors about when a key milestone or threshold was hit. And the infrastructure that makes rapid data analysis possible also could be extended to automate many other tedious trial execution tasks, like patient recruitment, scheduling, and follow-up.

Will This Scale?

To be clear, the idea of using AI and automations to streamline trial data collection isn’t new; companies have been working on this for years. But FDA’s involvement could have a huge impact in actually getting these approaches adopted in a very conservative and risk-averse industry.

Observers are already asking how we can scale up this approach and apply it to more trials. Sean Khozin, who originated the real-time trials idea during his tenure at FDA, posted about this in his Substack: “The hard work, the transformation, is the scale-up. It is the second sponsor, and the tenth, and the hundredth. It is the standards work that has to happen quietly behind the announcements, the EHR-to-EDC plumbing, the sponsor data systems that have to be re-architected, the reviewer workflows that have to be redesigned around streams instead of submissions.”

If you are a longtime reader of my blog, you already know why scaling innovations in trials is so difficult. The trial industry is deeply fragmented, poorly coordinated, and risk-averse. And regulatory uncertainty makes companies fearful of doing things differently. You might also know that the industry has a long and inglorious history of piloting better ways of doing things, only to go back to doing things the old way after the pilot has ended.

The limits of this particular pilot illustrate the challenges. For the pilot, they picked a relatively simple task: the trials appear to be using one software vendor in just two health systems (this might change, but I expect the final number of vendors and sites to remain small). For this approach to be truly transformative, though, it will need to be scaled up to a much larger set of trials, including the large, expensive late-phase trials. These trials are conducted across multiple sites, often in multiple countries, using software from many vendors and pulling data from multiple EHRs and hospital systems. I don’t want to understate the technical achievement that Paradigm Health pulled off in developing the system used in this pilot—it’s difficult and impressive work. But expanding this approach will require a lot more technical work, including, as Sean Khozin noted, standards and plumbing.

FDA will need to be actively involved in this scale-up. Part of the reason that we do things in the current, inefficient way is because sponsors believe that FDA expects it. The cumbersome system of audits, source-data verification, and adjudications were not an inevitable part of trials; they grew over time to meet the perceived demands of regulators. If we want to automate those tasks with AI, sponsors will need to work closely with regulators to “de-risk” this approach. Pilots alone won’t be sufficient; FDA will need to set expectations, change policies, and even change its culture to accommodate a new approach to running and reviewing trials.

What Should the FDA Do Next?

As excited as I am about the potential of this new approach to running trials, the initial rollout was not exactly confidence inspiring. The FDA didn’t really tell a coherent story about why real-time trials matter, how the agency intends to use the data, or what it will take to scale up this work. But there’s still plenty of time for the FDA to provide more clarity—and I suspect they’re busy working on addressing these questions.

With that in mind, I’d offer three recommendations for the FDA:

First, the FDA needs to define the operating model for its use of real-time trial data. If you just read the press releases and materials, you might expect FDA reviewers to be monitoring a stream of data like a manager might monitor their corporate dashboard. Perhaps FDA reviewers will wake up in the morning and watch the trial results roll in while they drink their coffee. This, to be clear, would be a terrible idea. The last thing that sponsors need is FDA staring over their shoulders as they run the trial, (over)reacting to safety signals as they come in or prematurely drawing conclusions from interim views of study endpoints.

A more prudent model might look something like the one used in FDA’s real-time oncology review program, where the availability of early readouts allows FDA to gain familiarity with the trial outcomes earlier and therefore issue decisions more quickly. For example, FDA might receive a draft readout of the trial results the very day of the last patient visit before the trial’s “database lock”. This would give FDA a better sense of what to expect when the sponsor comes in later with its more polished data package.

But that is just one model of many the FDA should consider. Along with the real-time trials announcement, FDA published a request for information seeking input on potential pilots of AI-enabled methods to optimize early-phase trials.1 This could be a great opportunity to explore how these newer operating models might work in practice and anchor them in established principles for trial design and statistical analysis.

Second, FDA needs to help build the standards and infrastructure to scale this approach beyond just one vendor and a few trial sites. Today, this work is only possible because one company, Paradigm, worked closely with sites to integrate their disparate data sources. That is useful for a proof of concept, but it is not a scalable model for the broader clinical trials enterprise. Large late-phase trials (and even many early phase trials) are conducted across many sites, often in multiple countries, using multiple EHRs, EDCs, labs, imaging vendors, clinical outcome assessment tools, and monitoring platforms. For real-time trials to work, they’ll need to operate on common standards that all of these disparate systems can use.

FDA could play a role in setting data standards for real-time trials. FDA should work with ONC and standards development organizations to establish standards for how clinical trial systems share and transmit clinical and operational data. These standards could both facilitate real-time submission of data and help build the infrastructure needed to make trials more efficient and automated. These standards should include the metadata needed to understand the data’s provenance, its role in the trial, and whether and how it was verified or adjudicated. This work could begin now, building on work already underway in this area by existing standards development organizations.

Third, FDA should set clear expectations for sponsors when it comes to data collection, verification, and validation. The pilot shows that we can use technology to make trials far more efficient by reducing or automating cumbersome practices like source data verification, queries, and manual adverse event review. But for this to be possible, FDA needs to clearly define the conditions under which it will accept automated or AI-assisted alternatives, and how it intends to inspect study sites that use these approaches without requiring them to maintain reams of paper documents and PDFs. Otherwise, companies will default to doing the traditional, “safe” approach to data collection, even when better approaches are possible.

FDA’s expectations-setting should go beyond ordinary guidance. As readers of this blog know, FDA has tried in the past to issue guidance offering alternatives to practices like 100% source data verification, only to find that the guidance was not specific enough to change industry practice. I’d suggest FDA get more specific, identifying standards, practices, and data submission formats that it would deem acceptable for purposes of real-time submission and review.

If the FDA can address these issues, real-time clinical trials could go beyond just pilots and become the foundation for a better, more efficient way of running trials. But if FDA does not clarify its operating model, help build standards, and set regulatory expectations, this could go the way of most other trial pilots, which have failed to make any lasting impact on how the industry operates.

But there’s also an important story about how the clinical trials industry got this way. Trials used to be far less bureaucratic, and many of the practices we take for granted in trials, like monitoring and source data verification, only became standardized in recent decades. The story of how trials transformed is a case study in how regulation can reshape an industry–and it can help inform today’s debates over how trials need to change.

The story I’m about to tell is most often recounted by a group I refer to as the “trialists”: the academic leaders who helped design the most important large-scale clinical trials of the 1980s and 1990s, and who continue to influence the debate over clinical trials today. To these trialists, today’s clinical trials industry exists in a kind of fallen state. In their account, the 1980s and 1990s represented an efflorescence in the science and practice of trials: we proved we could run trials at unprecedented pace and scale— and at remarkably low cost. Trials seemed poised to answer many of our most important clinical questions. Then came a decline: in the 1990s, the trials industry underwent a transformation that left it corporatized, bureaucratized, and diminished in ambition.

Here, I’ll recount the trialists’ story. I want to explore what that story gets right, what it misses, and what it tells us about the prospects for real reform.

Subscribe now

The Rise of Large Trials

Within the story of clinical trials, there are two narratives running side-by-side. One is a story of continued scientific progress. The modern clinical trial is more than just a form of scientific experiment: it is a kind of truth-finding technology, refined and improved over decades, that has helped us unlock life-saving new treatments and improve care. Since the creation of the modern trial in the 1950s, the clinical trial has undergone a series of reforms that have cemented its status as the “gold standard” for clinical evidence.

But alongside the story of scientific progress is one of operational stagnation. Even as the science of trials advanced, the business of trials – sometimes referred to as the “trials enterprise” – has struggled to produce the evidence we need to develop new treatments and improve care. Most significantly, we have lost much of our capacity to run simple trials, cheaply, at scale.

To understand how this happened, we need to retrace the story of the trials enterprise. (If you’re interested in learning more about this story, I recommend this piece by Alexander Janaroff, which was a key source for this post, as well as FDA’s own history.)

Today, the drug industry’s trials are a big business; most trials are sprawling enterprises run by large organizations across multiple sites. But before this modern system of clinical trial research was developed, drugs were largely tested in an ad-hoc way by independent clinician-investigators. In the first half of the 20th century, the quality of those investigations could be mixed. At their best, investigators made genuine efforts to control their experiments, taking care to make adequate and unbiased comparisons between patients who took the drug and those who did not. At their worst, they relied on what can best be described as “vibes”. Drug companies might ask clinicians if the drug appeared to work, and use testimonials to justify their product’s safety and effectiveness. We had relatively few standards for what good, unbiased clinical research ought to look like.

Over time, however, the standards for research tightened, particularly by the 1950s as the principles underlying modern trials were codified. That process was pushed along in the 1960s as the FDA started systematically reviewing drugs’ efficacy. But even well into the 1960s, as scientific rigor increased, the fundamental operating model of the trial hadn’t changed.

To test a drug, a company would typically send samples to clinician investigators, who would take responsibility for figuring out whether the drug worked. FDA’s own statutes and regulations still reference this model: they stress the importance of bringing on “qualified investigators” to study drugs. The idea that trials might not be run by individual investigators but rather by large organizations across multiple institutions was relatively new and did not fit well with FDA’s model of regulation (a problem that persists today, incidentally, despite some subsequent regulatory changes).

In contrast to the large, multi-center studies that researchers were running as early as the 1940s, most early investigator-led studies were small, limited to just one institution. That created a problem: a single investigator in a single academic institution could study only so many patients. By the 1960s, it had become clear that these small studies were inadequate to the task of studying the latest drugs. The drugs themselves had changed: many drugs developed in the postwar period, such as antibiotics, had dramatic and obvious effects that were noticeable in small studies. By the 1960s, newer classes of drugs, including cardiovascular drugs, arose. Researchers and regulators were interested in studying these drugs’ effects on “hard” endpoints - like death, heart attack, and stroke - rather than just measuring their effect on biomarkers. These hard endpoints were observed relatively infrequently (fortunately, heart disease patients usually live for quite a while before succumbing to their illness).

These new drugs required a different approach to running trials. Given how rare these hard outcomes were, researchers realized that their studies would need to be far larger if they wished to detect an impact on mortality in a reasonable timeframe. In the 1970s, modern methods of study size calculation, advanced by researchers like Richard Peto, emphasized that the efficacy of many drugs could only be determined if thousands of patients were studied.

This realization led to the creation of a new approach to running studies, the “large simple trial.” In a pioneering paper by Salim Yusuf, Rory Collins, and Richard Peto, the authors described what this kind of trial should look like. First off, it should be very large: powered to detect small changes in mortality. That would require them to be run at many sites, each adhering to the same protocol. The authors emphasized that the protocol should be simple, and not too burdensome. They wrote: “Many clinicians involved in the management of patients are already overworked, and in practice a really large-scale trial is likely to succeed only if it adds little or nothing to their existing workload.”

The authors also stressed that simplicity need not sacrifice rigor or accuracy. In fact, simple protocols with simple treatment regimens would help improve patient and researcher compliance. While proper randomization was absolutely vital, other aspects of the trial might even vary a bit between study sites without biasing the results. As long as randomization was done properly, some “noise” should not threaten the validity of the trial. The largest trials of this era reflected this philosophy. One of the first large simple trials, the first of the “International Studies of Infarct Survival” (ISIS-1), which started in 1981, only required researchers to submit a 1-page form with patient data and had no monitoring or endpoint adjudication.

While these large simple trials never made up more than a small portion of overall trial activity, they had a major impact on the field. For example, the second ISIS study randomized over 17,000 patients and showed that streptokinase and aspirin used together could significantly improve the survival of heart attack patients. Trials like the ISIS studies and GISSI (an Italian study of heart attack treatments) each enrolled tens of thousands of patients, and their findings transformed care for heart attack patients and saved many lives. More broadly, the principles espoused for large simple trials–the importance of rigorous randomization, sufficiently large study sizes, and simple protocols–became important general principles of good study design.

The large simple trials also transformed the trials industry. Their rise led to the creation of a new kind of organization, the “Academic Research Organization” (or ARO): university-based organizations that specialized in running large multicenter trials, also sometimes called “megatrials,” for both academia and industry (examples include the Duke Clinical Research Institute and Oxford’s Clinical Trials Unit). The rise of the megatrial helped build the reputation of a number of renowned trialists who remain influential in the field today. This group included Peto, Yusuf, and Collins, as well as Robert Califf (who would later become FDA commissioner), and Martin Landray (who would later become famous for leading the RECOVERY trial during COVID).

Subscribe now

Transformation and Industrialization

With the growth of megatrials, the trials industry reached a critical pivot point. As early as the 1980s, industry had begun to demand more trials, more control, and more speed than university researchers and their AROs could provide. That led to the growth of a new kind of organization: the private contract research organization (or CRO). Over time, CROs displaced AROs as the primary operators of multi-site clinical trials. They were able to keep pace with rising industry demand, and even expanded those trials’ reach, extending their operations globally.

The rise of the CRO coincided with–and was facilitated by–another important development: the establishment of global good clinical practice (GCP) guidelines and intensified regulatory oversight over trials. While FDA oversight over clinical trials began in 1962, the 1990s GCP guidelines ushered in a new era of regulation. Trials began to face closer regulatory scrutiny, and monitoring and documentation began to increase.

These two developments reinforced each other. CROs had the organizational capacity to adapt to the GCP guidelines and help drug companies meet them. In turn, the heightened oversight under the new GCP guidelines gave FDA greater confidence in the ability of CROs to manage trials. Prior to this, there was an expectation that an impartial, unbiased trial should be run by academics–not by the private sector.

By the 2000s, CROs were running the majority of research; the relative importance of academic medical centers and AROs had greatly declined. In 1992, CROs earned just $1 billion, but as of 2024, CRO revenues are estimated to have reached nearly $60 billion. Nowadays, they are truly gigantic organizations: the biggest CROs, such as IQVIA, ICON or Parexel, each employ tens of thousands of people and support many hundreds of studies every year - far larger than the AROs ever were. This transition changed the nature of the trials industry. Trials were industrialized, at least, in part (I’ll argue soon that this industrialization was incomplete), and were conducted at a larger scale than ever before. Trials were also bureaucratized: the rise of the CRO coincided with the rise of the extensive system of monitoring, auditing, and documentation that defines today’s trials.

Meanwhile, the large simple trials pioneered by the leading trialists had begun to decline, driven by several factors. Funding had fallen off: the pharmaceutical industry still funded some trials run by AROs, but academic medical centers cut back their own research budgets under increasing financial pressure. Meanwhile, NIH, which had played a crucial role in shepherding the rise of the large simple trial, cut back their support for trials too. The decline in larger-scale confirmatory trials was especially stark: NIH-funded phase 3 trials fell from 230 in 2005 to 62 in 2015.

There was also a shift in interest – driven in part by evolving science. By the 2010s, biomarker-based research and precision medicine would come to dominate the drug development landscape. Drug developers moved away from large-scale cardiovascular trials as they pursued products for narrower uses tested in smaller studies.

But a crucial change reinforced–and perhaps even drove–these other trends: increased bureaucracy. In an effort to comply with regulatory requirements, drug companies and CROs added additional procedure, complexity, and bureaucracy into their trials. As bureaucracy increased, costs rose. Even publicly-funded trials and academic-funded trials were affected, since both academic and industry-led trials were run by the same staff in the same sites. Rising costs made it difficult for academic medical centers and NIH to continue to fund large trials. Even the drug industry lost interest in large trials as they grew more expensive.

Subscribe now

The end of trials’ golden age

To the trialists, the decline of the large simple trial might have felt like the end of a golden age. The trials enterprise that had given birth to large simple trials was gone, replaced with something more bureaucratic, more expensive, and less capable of answering the questions that actually mattered to doctors and patients.

The trialists directed sharp criticism at the FDA and other drug regulators’ “Good Clinical Practice” (GCP) guideline. Published in 1996 by the International Consortium on Harmonization (ICH), a global consortium of regulators, the guideline was intended to serve as a global standard for how regulated clinical research should be conducted.

The trialists lodged many complaints against the guideline, both general and specific. On the specific side, they pointed to a number of seemingly wasteful practices encouraged by the guidelines: expensive on-site monitoring of study sites, verification of every transcribed data element, extensive and exhaustive reporting of even irrelevant adverse events, and time-consuming documentation to support inspections and audits.

More broadly, research leaders argued that the GCP guidelines–unlike the clinical guidelines issued by professional societies–were created with no scientific input, had no logical basis, and had no evidence backing their recommendations. Moreover, the guidelines’ entire emphasis was wrong: rather than promoting the good scientific practice that had been espoused by the designers of the large simple trials, they emphasized extensive documentation, auditability, and a nitpicking approach to data quality that had little bearing on the overall validity of the study.

Worse yet, the regulations were applied too broadly. The GCP’s scope was initially limited to regulated research–that is, research whose results had to be submitted to regulatory agencies like the FDA. Gradually, however, the ICH GCP’s scope expanded: in the EU, the ICH guidance was required to be applied to all research. In the US, the guidance’s legal status is “non-binding” but, in practice, it too is applied to all clinical research conducted in the country. The leading trialists and academic researchers decried this change; likening it to a “straitjacket” on their research.

If the guidelines got most of the blame, the CROs and the drug industry came in for a close second. They interpreted FDA’s requirements in a maximally risk-averse way. Then their interpretations, reinforced by a fear of FDA inspections, created a stifling environment that drove up costs and bureaucracy at the expense of research quality.

There was an even darker edge to this concern. As several leading trialists presented suggestions to streamline trials, they feared opposition: “Certain entities have benefited from the complexity of the current regulatory environment — not just contract research organizations and companies providing training in the ICH-GCP guidelines, but also regulatory groups in pharmaceutical companies and other institutions, which have seen their revenue and influence increase substantially — and they too may oppose streamlining.” This view was part of a broader concern; that a kind of compliance-industrial complex had emerged, in which regulators, the pharmaceutical industry, and the contract research organizations shared an interest in retaining an elaborate and risk-averse bureaucracy.

As the trialists lodged their complaints, the pillars of the large simple trials movement were toppling, one by one. The trialists advocated for trials that were simple, large, and rigorously randomized. The industry was moving in the opposite direction. Trials were growing more complex, and CROs and the drug industry managed that complexity with tighter procedures and more layers of bureaucracy. This made large trials prohibitively expensive, so industry shifted its focus to smaller trials for complex biomarker-driven indications, and advocated for regulatory reforms that could make the trials even smaller and shorter. The last pillar to fall was randomization itself, as the drug industry encouraged the FDA to accept observational studies instead of trials. To the advocates of large simple trials, this was anathema; industry could more easily imagine doing away with trials altogether than making them simpler and cheaper.

Subscribe now

The Response to the Backlash

The trialists pushed back hard against these trends the best way they know how; through the institutions of science. In the 2000s, a series of journal articles and commentaries came out with titles like: “Randomized clinical trials: Slow death by a thousand unnecessary policies?”; “The Good Clinical Practice guideline: a bronze standard for clinical research”; and the more measured (but equally critical) “Clinical trials bureaucracy: unintended consequences of well-intentioned policy.” Over time, they would launch coalitions, host convenings, and lead scientific roundtables aimed at reversing the tide of bureaucracy in trials.

While these sorts of articles and convenings may have limited impact on the broader public discourse, they matter a great deal to FDA and other regulators, who depend on the support of scientists and academics. So as the criticisms mounted, the FDA engaged with the critics. In 2007, they partnered with Duke University to found the Clinical Trials Transformation Initiative (CTTI), aimed at making trials better and more efficient. The initiative’s founding CEO Judith Kramer, articulated the problem: “The investment in time, money, and human resources for clinical trials has been rapidly increasing in recent years, without a commensurate increase in the number of new products entering the market place. There is a growing consensus that clinical trials are inefficient and too costly.”

In the years that followed, FDA, informed by the work of CTTI and the trialists themselves, issued new rules and guidance that tried to address the most egregious examples of bureaucracy and waste. For example, in 2010, FDA published a rule limiting how often drug companies needed to submit expensive and time-consuming “expedited safety reports” during their trials. In 2013, FDA issued guidance clarifying that sponsors did not need to manually verify that every data element collected in the trial was accurately transcribed – a practice called ‘100% source data verification’.

Yet in each case, the rules were undermined by cautious, risk-averse interpretations. After the finalization of the 2010 safety reporting rule, CTTI found that expedited reports submitted to FDA actually increased. Likewise, years after FDA recommended against routine use of 100% source data verification, the vast majority of trials still adopted the approach. Similar efforts at reform, including regulations aimed at simplifying informed consent forms and reducing the scope of IRB review also had limited impact.

Despite these setbacks, efforts at reform continued, culminating in the 2025 issuance of the revised good clinical practice guideline. This guideline was explicitly designed to respond to the criticisms leveled by trialists. It cautions against overly complex trial protocols, and pushes hard for a “risk-proportionate” approach to tasks like study monitoring. Not only is the reform comprehensive, but–unlike the FDA-driven reforms that preceded it—the ICH guideline is global, raising hopes for more widespread adoption. But even as the guidelines were being developed, trialists expressed concerns that over-cautious interpretations of the guidelines could limit their value.

Subscribe now

“Back to the Future”: Reviving Large Simple Trials

Amid the backdrop of regulatory reforms, another important story was taking shape: an ongoing effort to revive the large, simple trial; or at least, to bring back some of its key principles. No story more exemplifies the power – and limits – of this approach than that of the COVID-19 RECOVERY Trial.

At the center of this story was Martin Landray, one of the advocates of simple trials we mentioned earlier. For years, he had helped design and run large simple trials alongside his colleagues at Oxford’s Clinical Trial Service Unit (CTSU) – an organization that had pioneered the concept of large simple trials. As trials grew more complex and bureaucratic, he was one of the loudest voices in favor of reform, and led several of CTTI’s efforts to reform and simplify trials. He even launched his own alternative trial guideline, “The Guidance for Good Randomized Clinical Trials” as a scientifically-grounded complement (and perhaps alternative) to the documentation-heavy and bureaucratic GCP guideline.

With the spread of COVID in early 2020, Landray had a chance to prove that a streamlined trial could be used to identify effective treatments for COVID. Before the pandemic had reached the UK, he had already begun to design the trial that would later become RECOVERY. He borrowed heavily from the large simple trials playbook: a one-page case report form, simple criteria for eligibility, and a simple protocol that could readily be administered at scale. But he modernized the approach, using web-based forms to collect data, an adaptive platform design to compare multiple treatments, online training for study sites, and the use of additional data collected from NHS’ electronic health record systems. It cost just $500 per patient.

The streamlined design allowed the trial to quickly recruit patients and produce results. Within just a few months, RECOVERY had enlisted every acute hospital in the UK (along with sites in many other countries) and had identified life-saving treatments for COVID. Those treatments quickly became part of standard COVID care in hospitals, saving millions of lives.

Like the 2025 good clinical practice guidelines, the success of the RECOVERY trial felt like a turning point. Landray and his fellow trialists had spent decades arguing for a return to streamlined trials, and, in this moment, they seemed to be vindicated. If RECOVERY’s streamlined and broad-based approach could work for COVID, surely the model could be adopted for other diseases.

In fact, RECOVERY was part of a broader movement, pushing for new kinds of trials that could recapture the benefits of those older, large, simple trials. The trialists have outlined what those new trials might look like. Like the old trials, they would be streamlined to impose minimal additional burden on study sites. But unlike those old trials, these new trials could leverage the data in electronic health records to streamline trials even further and collect more useful data. Their reach could be broader too: “point-of-care” trials could be run in community settings outside of academic medical centers, and decentralized trials could be carried out remotely through telemedicine. These newer trials could build on the strengths of simple trials while creating something better and more efficient.

But these new kinds of trials face huge obstacles. The RECOVERY trial was run in a COVID state of exception. It’s not clear whether other trials of similar scope and ambition would be possible today, absent a major public health crisis. Nor, it seems, was such a trial possible here in the United States. Our own efforts to run large-scale trials during COVID were largely unsuccessful. Researchers continue to find ways to run streamlined trials, but they remain a very small part of the research enterprise.

Subscribe now

Why large simple trials still matter

Today, the problem of clinical trial inefficiency and bureaucracy feels more urgent than ever. There is newfound energy and enthusiasm around doing something about the problem. Amid that backdrop, many of the debates that preoccupied the clinical trialists are playing out again: we are asking ourselves how we can make trials more efficient, more streamlined, and less bureaucratic. That makes this a good time to assess what we can learn from the past two decades of efforts at trial reform.

It’s clear that, for all they have achieved, the trialists have not yet been able to address their most important goal: to make trials significantly simpler and less bureaucratic. While the regulations have improved, the trials’ costs and complexity have only risen. Even if we stemmed the tide of bureaucracy, there is probably no way to return to the trials enterprise we once had.

But if we can’t go back to the heyday of large simple trials, the good news is that we wouldn’t necessarily want to. We should acknowledge the accomplishments of our current industrialized bureaucratic trials system. The industry is vastly larger than it was in the 1980s “golden age”, and our capacity to learn from trials has never been greater. We’re running more trials across more sites and more countries, for a broader range of diseases. And despite the greater size and complexity of the trials, we retain confidence in their rigor and validity. Thanks to the rigor of our system, FDA’s leaders were able to make the argument that we only need one trial, instead of the previous two, to demonstrate a drug’s effectiveness.

The challenge, then, is preserving the benefits of today’s trial system while streamlining it and reining in its excesses. For that reason, we should keep learning from the era of large simple trials. The principles behind those large simple trials are still valid and deserve to be more widely known and followed. And more importantly, when we study the history of trials–and the work of pioneering trialists like Martin Landray–our minds open to new possibilities. We learn that there is nothing inevitable about the way our clinical trials system works today: it especially does not have to be so complex and bureaucratic. We’ve transformed the trials system before, and it can be transformed again.

Back in January FDA published draft guidance on use of Bayesian clinical trials. Two of us already covered it, Adam Kroetsch on this blog and yours truly on Andrew Gelman’s blog. Now Andrew, Erik van Zwet and I also put together a short commentary for JAMA:

FDA Draft Guidance for the Use of Bayesian Methods in Clinical Trials by Andrew Gelman, Erik van Zwet and Witold Więcek (2026) in JAMA

It’s only around 1,000 words, so you should read it. The overall assessment is similar to the piece I wrote in January: we think it’s a very positive step. In one sentence, the contribution of this guidance is to clearly define how relevant information can be used to create Bayesian priors for trials, which can make many of them more efficient. Now the question is whether trial sponsors are willing to do it more often.

But there is also the perspective of the regulator: are there any broader benefits to the FDA from being Bayesian? We think the answer is yes. Following these principles can lead to more consistent and transparent decisions. This may strike some people as counterintuitive. The most common critique of Bayesian methods in clinical trials is that they “sneak in subjectivity” under the guise of prior. So how could they lead to more consistency? As Adam nicely outlined in his piece, the status quo is that the FDA already relies very heavily on “clinical judgement” in its approvals, which does not sound dissimilar from having a Bayesian prior to me! We write:

Consistency and transparency in these decisions are important—uncertainty about the procedures would prevent drugmakers from conducting research and lack of transparency can expose decisions to political pressure and lobbying. However, a survey of 912 FDA applications [5] found that FDA decisions did not consistently cite prior reasoning and that some approvals, after additional review, reflected new interpretations of existing evidence rather than new evidence, raising concerns about consistency across review cycles.

This survey of 912 applications, Janiaud et al, is also a short and interesting read. It is co-authored by FDA staff and it broadly acknowledges that there is no consistency, “resulting in standalone, bespoke decisions”. And that is not automatically a bad thing! As we also say in the piece, flexibility is necessary in regulatory decisions. But there are notable examples where this becomes a problem, where the basis of these judgements becomes… questionable:

A recent high-profile example may be illustrative. In February 2026, the head of the FDA’s Center for Biologics Evaluation and Research issued a refusal-to-file letter for Moderna’s mRNA influenza vaccine, citing the failure to compare the vaccine against high-dose competitors in older adults [6]; however, this decision was reversed after a week.[7] The problem of comparisons among relevant subpopulations can be directly and transparently addressed using hierarchical bayesian methods. Influenza vaccines are among the best studied therapeutics in existence. Relevant information from other trials could be incorporated into the prior and data reanalyzed to conduct a risk-benefit analysis in older adults without conducting a new trial.

In other words, this new guidance specifically focuses on the topic of how to transparently incorporate information across other trials and incorporate benefit-risk into the definition of success. If the FDA reserves the right to move the goalposts somewhat, that is a valid concern! But why not use the principles from its own Bayesian guidance to reach this decision: based on all available data, how likely is it that the vaccine is not as effective in the elderly as its high-dose competitors? What sort of prior assumptions are required to reach this conclusion? The Bayesian principles in this guidance make this project feasible.

In sum, we should be scrutinising both the work of the drug maker and the decision maker. Yes, it is entirely the sponsor’s job to generate data that proves their therapeutic is safe and effective. But then it is the job of the regulator to prove to us that their approval decisions are rational—especially in the cases which lean heavily on judgement. Or, as we like to say in my line of work: show us the model!

ANNOUNCEMENT KLAXON

The Irish government is seeking volunteers for their Clinical Trials Advisory Council. If you have relevant expertise and are interested in shaping European regulatory frameworks, consider applying.

Our friends at 1Day Sooner, best known for their advocacy for challenge trials to speed up COVID vaccines, are hiring for a new role that could genuinely impactful in helping to accelerate clinical testing and getting breakthrough medicines get to patients faster.

They’re hiring a Policy Lead for their new Clinical Trial Abundance program. As you’ll know from this blog, Clinical Trial Abundance is the idea that we can accelerate medical innovation by making it easier to collect and generate experimental data, and do more with the evidence we already have. It’s something a number of us are interested in, and 1Day Sooner are now building out the policy infrastructure to move it forward.

This is a policy-focused role: involving regulatory strategy, drafting comment letters and legislative language, building coalitions, and working with decision-makers across FDA, HHS, NIH, and Congress. They’re looking for someone with 5+ years in health or science policy who can translate high-level goals into concrete wins.

It’s remote and full-time, with a preference for DC-based candidates (though they’ll consider applicants across the Western Hemisphere). Salary is $100–145k in the US.

If that’s you, apply here: https://www.1daysooner.org/jobs/

If it’s someone you know, please share this with them!

Share

Thank you :)

– Saloni

There is a vast gulf between how the pharmaceutical industry sees itself and how the public sees it. To the public, the pharmaceutical industry is venal and corrupt, putting profits above patients’ needs. But when I speak to those who work in the industry, they are deeply mission-driven: many enter the field with sincere hopes of discovering treatments and cures for the world’s most devastating diseases.

At the heart of this mission are the scientists who discover new drugs. They’re the reason the industry exists, and they represent how pharma wants to be seen. If you visit a drug company’s website, you’ll find stock photos of them: They’re the intrepid researchers, toiling away in their lab coats, on the cusp of their next breakthrough. If the industry has a ladder of prestige, these scientists sit at the top of it.

Also near the top of the prestige hierarchy are those who work on clinical trial design. They may not discover new drugs, but they make decisions that can make or break a drug development program. Their work requires deep scientific expertise, regulatory knowledge, and creativity.

Many rungs below sit the people who actually test the drugs and prove they work. These are the “boring things after the creative, exciting part is done,” including the mundane but necessary work of “clinical operations”: setting up trials, negotiating payments, recruiting patients, collecting data. Even among those who work in the field, this is often considered the “unsexy”, “boring” part of drug development.

The field of clinical operations lacks drama and gets little attention. The industry, its leaders, and its investors focus on big milestones: a “go / no-go decision”, a high-stakes FDA meeting, the start of a $100+ million dollar trial, the pivotal readout that determines a program’s fate. By contrast, trial operations are a day-to-day grind. There are few big moments to celebrate. I suspect many CEOs spend relatively little time thinking about it.

But we desperately need to improve trial operations if we want to make clinical trials more efficient.

Trial operations have gotten less productive

That’s because trial operations are what drives trial costs, and those costs are rising fast.

To understand the rise in costs, it’s useful to understand how drug industry trials are typically run. Drug companies pay for trials, but they aren’t the ones who carry them out. Rather, their main job is to develop the trial protocol, which lays out how the trial ought to be run: who is eligible participate in the trial, what procedures should be run, what data should be collected, and how that data should be assessed. The work of running the trial itself is usually outsourced: Contract research organizations (CROs) coordinate and oversee the trials, and research sites—including large academic medical centers and research clinics—carry out the protocol.

As research sites do the bulk of the hands-on work of running the trial, they also incur the largest portion of the trial’s cost. Site costs have risen rapidly in recent decades. One analysis showed that site costs for phase 3 trials rose from $11,630 per patient in 2000 to $23,711 in 2011—an increase of nearly 10% per year. While we lack more recent data on site costs, other data suggests they are continuing to rise: a more recent study which measured total trial costs (not just site costs) found that phase 3 trial costs had reached $55,567 per patient by 2017. Meanwhile, as costs have risen, trials have also gotten slower. They take longer to set up, longer to enroll patients, and longer to complete. Those costs and delays limit how many drugs ultimately make their way to patients.

One reason that trial costs are rising is that the trials have grown more complex. Drug companies are collecting more data, running trials across more sites, and measuring more endpoints. They are also placing tighter restrictions on who can enroll. Some of the increases in complexity are scientifically justified, but much of it looks like bloat. This complexity drives up costs and puts strain on the trial sites. Trials are being optimized for regulators and risk-averse managers—not for the sites that actually have to run them.

But complexity isn’t the whole story. If we take a closer look at site costs, we find that they have risen even after accounting for the increased complexity, and faster than inflation in the health and research sectors. In other words, the trials industry, which already lagged in its use of technology by the turn of the century, has probably gotten less productive over time.

This won’t surprise people who work in clinical operations. While drug companies and trial sites take great care to adhere to protocols, procedures, and regulations, there is very little about the process that is automated or streamlined. Brandon Li, a clinical trials entrepreneur, remarked that “clinical trials are $200m group projects held together by duct tape and project managers.” If you’ve ever worked for an over-stretched and understaffed organization, you can picture the failures: manual transcription across paper and computer systems, reliance on ad-hoc spreadsheets, lost logins and passwords, inboxes full of missed emails, stalled contract negotiations, and unpaid invoices. Somehow the work gets done, but it feels much harder and less efficient than it should.

If that picture sounds discouraging, I’d like to emphasize the good news here: these problems are fixable. Making trials dramatically more efficient doesn’t require us to invent new science or technology. It doesn’t require us to reform the FDA. We don’t even need AI. (All of these things would be helpful, though!) All we really need to do is fix the basic stuff: stop using paper, stop transcribing information manually between systems, stop wasting time on useless administrative tasks, and stop making the protocols so needlessly complex. In other words, we just need to do the kinds of operational modernization that other industries have been doing for ages.

Fixing trials is boring, hard work

If the fixes are so basic, why haven’t we done them already? I suspect it’s because the work is too boring–and not prestigious. In an industry that prizes scientific creativity and expertise, operations don’t get much attention. Fixing trial operations and designing trials that are easier to run requires sustained focus and coordination. It won’t make big headlines in the trade press.

A lack of attention makes reform harder. Consider the problem of risk aversion. I have interviewed industry experts and insiders - and nearly all of them mentioned that operations are run in a deeply risk-averse way. Managers at drug companies would rather stick to tried-and-true approaches to running trials than try to reform them and risk regulatory scrutiny. This risk aversion has downstream effects on the entire trials industry. Since drug companies are reluctant to embrace innovative cost-cutting measures, CROs face little incentive to cut costs. And without sponsor support, it’s hard for trial sites to improve their operations.

But behind many cases of risk aversion lies a deeper problem: a lack of leadership attention. Risk aversion isn’t unique to pharma. You will struggle to find a middle manager in any large organization who is willing to be bold unless they have visible backing from leadership. As long as industry leaders neglect clinical operations, managers will default to doing what they’ve always done and productivity will stagnate.

And trial operations need a lot of attention. It’s difficult to change such a fragmented, entrenched, and heavily regulated system. Anyone in the industry seeking to make even a small improvement in trial operations must navigate a tangled web of contracts, regulations, and SOPs. Each of their business partners—whether they are sites, sponsors, or CROs—will handle things differently. There are also larger-scale coordination problems: sponsors can’t run more efficient trials unless the sites agree to modernize their operations. Sites can’t modernize their operations unless all of their sponsors agree on what that modernization should look like.

That means that there are no quick fixes. We don’t lack the knowhow or technology to improve how trials run; we lack the ability to deploy improvements at scale. We can’t just send consultants in to make trials more efficient, and there’s no B2B SaaS tool that will transform the industry. Improving the efficiency of trial operations will take hard work and focused attention. Policymakers and the government will probably have to get involved too.

If we want to make trials more efficient, we need to pay more attention to the boring stuff. We need to measure it and prioritize it. That’s part of why I created the Clinical Trials Efficiency Project. When it comes to improving clinical trial efficiency, there’s a role for new trial designs, new science, new technology, and new ideas. But I also hope we can bring attention to the boring bottlenecks that make trials inefficient and have remained neglected for too long.

Adam

It’s easy to see that proposal as an ambitious reform, and in some ways it is. But the history of medicine shows us that clinical trials have already undergone a series of transformations that once seemed equally bold. Over time, the way we test treatments has become more rigorous and standardized. Improvements in transparency are a continuation of this trend.

In this post, I’ll give a brief history of clinical trial reforms and why transparency is the natural next step.

The rise of randomized controlled trials

One of the most consequential rigor-enhancing changes in clinical trials was the rise of the randomized controlled trial. In 1962, the United States passed the Kefauver-Harris Amendments, which required that new drugs demonstrate efficacy through adequate and well-controlled investigations before receiving approval. In practice, this pushed regulators and manufacturers toward randomized controlled trials (RCTs). This was seen as a dramatic shift.

At the time, clinicians had to rely on scattered uncontrolled studies, controlled studies without randomization, and case reports of claims of miracle treatments. They’d see some patients improving and others worsening, with no clear way to estimate the average effect or detect harms that only become visible at scale.

The requirement for controlled trials might have raised costs and overturned established practice. But it addressed a real problem: if you want to understand the effectiveness and safety of a drug using observational, uncontrolled data, you’d face many challenges:

• Regression to the mean: Many conditions bring people to the doctor when symptoms are at their worst, and some improvement would have happened anyway. Someone might seek treatment for severe back pain after a particularly bad week, only for the pain to ease as the flare-up subsides. The same pattern appears in mental health: a person may begin therapy during an especially intense depressive episode, then improve as the episode naturally wanes. If improvement follows treatment, it’s tempting to credit the intervention, even when part of the change reflects a return toward a person’s usual baseline.

• External events: Outcomes can also shift because the world around patients changes. For example, asthma admissions may spike during a period of high air pollution and fall when air quality improves, a heatwave can increase dehydration and kidney stress. Even public health campaigns, changes in food availability, or a stressful economic downturn can influence sleep, diet, and cardiovascular risk. If a treatment is introduced during such periods, its apparent effects may partly reflect broader changes.

• Selection into treatment: People who receive a treatment often differ in systematic ways from those who do not. Patients who opt for a new preventive medication might be more health conscious, adherent to medical advice, or able to afford regular care. Conversely, a specialist clinic may see the sickest patients, or certain drugs may only be prescribed at severe stages of a disease, making treatment look less effective than it truly is.

• Blinding and concealment address a different source of bias: the expectations and behaviours of the people involved in a trial. Even after randomization, participants or researchers may figure out who received the active treatment because the drug looks, tastes, or smells different from the placebo, or because it produces noticeable side effects. When this happens, it can partly reverse the benefits of randomization: if participants or researchers can identify who received the active treatment, differences between groups may reflect expectations and behavioural changes rather than the drug alone, meaning the estimated effect could be larger or smaller1 than the true effect of the treatment itself. Concealment is a first step, preventing researchers from knowing which treatment someone will be assigned to before allocation. Blinding goes further, keeping both participants and researchers unaware of group assignments throughout the trial.2

Without careful design, these differences can wrongly appear to be treatment effects. But adjusting for these factors is difficult, and the flexibility can also open the door to questionable research practices: analysing data in selective ways, choosing favourable subgroups, or stopping analysis when results look promising.

Randomization helps because it creates groups that, at the outset, have roughly equal propensities for the outcomes being studied. Darren Dahly has a clear explanation of this logic. And like my friend Julia Rohrer, I think of randomisation as the closest thing epidemiology has to magic.

In my view, there’s no question that some treatments did have good evidence for them even before the rise of randomized controlled trials. In particular, researchers could notice when treatments were followed by large, otherwise unexplainable changes – like the elimination of a disease following vaccination, or reversals of diseases that usually progressed quickly or fatally, as was sometimes the case after treatment with drugs like antibiotics and insulin.

But in most cases, it would have been hard to identify effective drugs, especially if their benefits were more modest, or if their benefits took a long time to become visible.

Subscribe now

It’s useful at this point to break down randomized controlled trials into their components: an experiment, with randomization, a controlled group, and often the practice of ‘blinding’ as well. Each of these components has had many precursors.

Take controlled groups as an example, which trace back – at least as far as I knew – to 1747, when James Lind compared treatments for scurvy aboard a Royal Navy ship and found citrus fruit dramatically effective. [After publishing this, Erin Braid pointed out that there were far earlier experiments with controlled groups, including in Ancient Greece; James Lind’s experiment is commonly named as the first one or one of the first ones.] I also found it interesting to learn that some 19th and 20th century controlled studies assigned patients to treatment in an alternate sequence, as they arrived to see a doctor, which is somewhat close to randomisation, but doesn’t involve blinding the researcher or participant to which treatment they received.

Blinding has an even more interesting history. It traces back to at least 1784, when King Louis XVI commissioned scientists including Benjamin Franklin and Antoine Lavoisier to investigate the claims of Franz Mesmer, who said he could cure illness through “animal magnetism” (later named ‘mesmerism’ after him). Franklin and Lavoisier designed an experiment in which they physically blindfolded subjects while performing the magnetic treatment; but crucially, they sometimes withheld the treatment without telling the subjects.

When they reported sensations regardless of whether they actually received the treatment, the experiment showed the effects were driven by their expectations rather than magnetism. It helped popularise the idea of blinding, in which patients are prevented from knowing which treatment they receive – although it was often the case that blinding was used to ‘debunk’ findings, rather than being open-minded to the results. Louis XVI’s interest, for example, was not purely scientific: mesmerism had become a fashionable and lucrative craze in Paris, and he likely worried about public disorder and the erosion of medical authority, giving the monarchy strong incentives to discredit it.

Combining these practices into randomised controlled trials, however, is relatively recent, at least historically speaking. The first widely recognised modern RCT was the UK Medical Research Council’s 1948 streptomycin trial for pulmonary tuberculosis.3 Patients were randomly assigned to receive the antibiotic streptomycin (a new breakthrough at the time) plus bed rest or bed rest alone. The trial showed clear improvements in survival and lung disease according to radiology scans in the treated group, but they also found early evidence of antibiotic resistance emerging.

As the twentieth century progressed, RCTs became more common. A very interesting example I’ve read of recently was the US National Institutes of Health’s Coronary Drug Project, conducted in the late 1960s and early 1970s, before statins were introduced.

In the decades leading up to it, researchers were experimenting with a surprisingly eclectic set of ways to lower cholesterol. If high cholesterol drove coronary heart disease, then lowering it ought to prevent heart attacks, so investigators tried hormones, vitamins, industrial resins, and metabolic drugs. Doctors claimed various drugs might be effective:

• Thyroid hormone (notably dextro-thyroxine)

• Oestrogens

• Niacin (vitamin B3)

• Bile-acid sequestrants such as cholestyramine

• Fibrates

The Coronary Drug Project trial allowed for a proper, large-scale testing of these drugs in a randomized controlled trial to see whether they actually reduced heart attacks and mortality. The results were surprising – niacin and cholestyramine showed meaningful benefits, but some other treatments, including oestrogen therapy and dextrothyroxine, actually slightly increased mortality – and changed clinical practice.

By the 1970s, randomized controlled trials were widely accepted as a gold standard for evaluating therapies. And after the 1962 reforms in the United States, along with similar regulatory shifts elsewhere, controlled evidence became the norm for drug approval.

Subscribe now

Preregistration

Randomization solved the design problem, but new challenges emerged in how results were analyzed. By the late twentieth century, the environment surrounding drug regulation had changed profoundly. To bring a medicine to market, companies were now expected to demonstrate both safety and effectiveness, usually through at least two well-controlled clinical trials, and typically using randomized controlled designs.

This was a major improvement over earlier eras, but it also raised the stakes dramatically. When development costs run into the hundreds of millions and a successful drug may earn billions annually, a failed trial can mean the loss of an enormous commercial opportunity.

The 1980s and 1990s saw a wave of breakthrough therapies reach patients. Statins transformed the prevention of cardiovascular disease. Angiotensin-converting enzyme (ACE) inhibitors improved survival in heart failure. Antiretroviral drugs began to turn HIV from a fatal diagnosis into a manageable condition.

But not every promising therapy proved effective, and even successful drugs often looked more impressive in early studies than they did later in routine practice. Understanding why requires a brief detour into how trial results are judged.

In most trials, success is judged using a statistical test that produces a p value, roughly the probability of seeing a result at least this extreme assuming the treatment has no real effect. By convention, results are called “statistically significant” if this probability falls below 0.05.

At the margins, this threshold creates a temptation. When a trial narrowly misses its endpoint, researchers may search for a significant result elsewhere: in a secondary outcome, a composite measure, or a subgroup analysis that, by chance, clears the bar. Given the enormous financial investment involved in developing a drug, there are strong incentives to tweak analyses at the margins, for instance, by changing which outcome is analyzed after seeing the data, a practice called outcome switching.

This flexibility undermines the validity of results. While randomisation improved how trials are designed, it did not fully address how results are analysed and reported; and flexibility after seeing the data creates opportunities for finding results that are favorable but unreliable.

To reduce the scope for this type of post hoc reinterpretation, regulators and journals began requiring trial protocols to be registered in advance. Starting in the late 1990s and expanding in the early 2000s, investigators were asked to document their study design and prespecified outcomes before data collection was complete. In the United States, these requirements broadened over time and were formalized through federal law and FDA regulations, with trial protocols and outcomes posted on ClinicalTrials.gov. Registration also became a prerequisite for publication in major journals, reinforcing the norm across the research community.

Analyses of trials funded by the National Heart, Lung, and Blood Institute found that before prespecification became standard, published trials were more likely to report positive results. After registration became mandatory, the pattern changed: many trials found no effect, and some detected harm. It became harder to redefine what success meant.

Even so, the value of prespecifying analyses is not about eliminating judgement but in improving transparency. It allows readers to see what researchers initially set out to measure, what changed, and why. There are countless examples from history of how both successes and failures in science have contributed to knowledge; science advances by testing predictions, learning from the results, and refining theories over time.

Subscribe now

Trial reporting

By the late twentieth century, clinical trials had become the gatekeeper for drug approval. Regulators increasingly required convincing evidence of both safety and effectiveness, usually from multiple well-controlled studies, before a medicine could reach the market. This shift improved the reliability of medical evidence. But even as trial design grew more rigorous, a problem remained: what happened to the results once a study ended.

The results of many clinical trials were never published. Studies showing clear benefits were far more likely to appear in medical journals, while trials finding no effect or suggesting harm often remained unpublished. The problem became increasingly visible in the 2000s, as systematic reviewers began documenting “missing” trials.

Analyses of antidepressant trials comparing journal publications with the full set of studies submitted to regulators found many studies went unpublished, and were disproportionately those with negative or null results (see here for a more recent analysis, which finds improvement). The published literature suggested overwhelming effectiveness, while the complete dataset showed more modest benefits. When meta-analyses relied only on published studies, average effect sizes were inflated, meaning clinicians and patients were often seeing an overly optimistic picture of drug effectiveness.

There are probably many reasons for this: Journals prefer novel and positive findings, and often require multiple submissions and long delays before acceptance, companies have little commercial incentive to highlight disappointing results, academic researchers face career pressures that reward publishing and success, and negative results seem less actionable. The path of least resistance is to simply move on, hopefully to something more promising.

But the implications are not merely academic. Incomplete reporting makes it harder to learn from failures and adjust drug development plans, to estimate the true average benefit of treatments, to detect rare harms, and to make informed decisions about patient care.

Since the 2000s, governments and regulators began introducing legal requirements for trial registration and results reporting. The 2007 FDA Amendments Act in the United States required many trials to post summary results in a public registry. In Europe, transparency rules evolved into the EU Clinical Trials Regulation and a centralized database designed to make trial information publicly accessible. These policies aimed to ensure that results entered the public record regardless of whether they were favourable.

Compliance did not improve overnight and enforcement was initially limited, but pressure emerged from multiple directions. The AllTrials campaign pushed for the registration and reporting of all trials, the Good Pharma Scorecard tracked company transparency commitments, and the FDAAA TrialsTracker made reporting performance publicly visible. In the United Kingdom, parliamentary scrutiny and reviews by the National Audit Office and the House of Commons Public Accounts Committee criticized failures to report trial results, increasing political and public pressure.

This combination of regulation, monitoring, and public accountability gradually shifted behaviour. Reporting rates rose substantially in the United States after reporting requirements took effect. Recent research by Cunningham et al. finds that academically funded trials initially had very low compliance rates, but rose rapidly. Industry funded trials began at higher compliance rates, and also increased. The researchers also found that this shifted the behaviour of other firms, which became more likely to wait for trial results before advancing similar drug candidates. As with earlier reforms, mandatory reporting was initially framed as a bureaucratic burden, but it’s increasingly considered basic infrastructure for trustworthy medicine.

Today, roughly 75–80% of clinical trials that meet the FDA requirement report their results within a year, with higher rates of reporting among industry-funded trials than academic trials.

Conclusion

This blog is about clinical trial abundance, but it’s worth stepping back to ask why we care about trials at all. Clinical trials are not an end in themselves; they exist to generate reliable information about whether treatments are safe and effective. In that sense, it’s not just clinical trials that we want an abundance of, but information from them. The reforms described above, randomization, preregistration, mandatory reporting, addressed ways that information was being distorted or lost. Data sharing is the natural continuation of that arc.

In my previous post, I argued that sharing individual patient data could unlock a range of benefits that go beyond what published results alone can offer: pooling data across trials allows meta-analyses to detect effects on rare outcomes and confirm or rule out rare harms; it helps make sense of inconsistent results by letting researchers explore whether differences across trials reflect chance, patient populations, or study design; it makes it easier to ask new questions of old data, including when drugs developed for one condition show unexpected benefits for another; it enables better head-to-head comparisons between treatments, and can help tailor recommendations to patients’ characteristics; and it reduces redundancy: when trial results are visible, researchers can learn from failures and avoid repeating them.

So why isn’t this already standard practice? For industry-funded trials, the commercial disincentive is straightforward: sharing data helps competitors at least as much as it helps you. A firm that shares its patient data is effectively subsidising rivals’ research programs. But if all firms shared, the field as a whole would benefit, and so would each firm within it, gaining access to a much larger pool of evidence than any single company could generate alone. The case for coordination, whether through regulation or shared norms, follows from that.

For academic trials, the commercial disincentive largely disappears, and most academic research is publicly funded, which creates its own obligation to make results available. But it’s still uncommon to share data, probably as a result of practical frictions, career incentives that reward further publications foremost, and the absence of strong mandates. The relative neglect of data sharing in academic trials is harder to justify.

The case for sharing individual patient data from trials is strong enough that I think it warrants the same kind of coordinated push that succeeded for trial registration and results reporting. The answer could be better incentives, data infrastructure, and in some cases mandates from funders and regulators to make sharing practical and beneficial rather than burdensome.

Scientists often say that the best research doesn’t just answer a question, it opens up new ones. That’s how I think about data sharing as well. Without individual patient data, we’re left mainly with headlines, with limited ways to verify or explore the data to answer those questions, or build upon accumulated knowledge.

I’d like to thank Matt Clancy, Dylan Matthews, Nisha Austin, Ruben Arslan, Jamie Cummins, and Adam Kroetsch for feedback that improved this post.

During a recent interview, Dwarkesh Patel and the CEO of Anthropic, Dario Amodei, discussed whether clinical trials will remain a meaningful bottleneck for drug development in the age of AI. Patel said that “most clinical trials fail because the drug does not work.” In response, Amodei speculated that as AI models get better at designing drugs, “clinical trials will be much faster … let’s say, they will take one year.”

This is a commonly voiced sentiment, but flawed. The truth is that the most significant barriers to progress today are rarely a lack of intelligence. London has a housing crisis even though the technology to design and construct homes has existed for centuries. The bottleneck in housing is not a lack of knowhow, but rather the weaponization of environmental regulations, planning, and NIMBYism. Much the same is true for clinical trials.

AI models can help design more elegant molecules, in the same way an architect can use AI to design more efficient floor plans, but neither intervention guarantees an efficient use of institutional machinery to make that design in the real world. Even the most promising drug candidates must be tested in human bodies which, in turn, need time to metabolize those drugs and develop side effects. Patients must be recruited and followed over time, and regulators must be satisfied. None of this is easily accelerated with AI.

Although I’m optimistic that AI will design better drug candidates, this alone cannot ensure “therapeutic abundance,” for a few reasons. First, because the history of drug development shows that even when strong preclinical models exist for a condition, like osteoporosis, the high costs needed to move a drug through trials deters investment — especially for chronic diseases requiring large cohorts. And second, because there is a feedback problem between drug development and clinical trials. In order for AI to generate high-quality drug candidates, it must first be trained on rich, human data; especially from early, small-n studies.

Clinical Variables

The Amodei interview conflates two distinct variables: the success rate of a trial (based on the quality of a drug), and the speed of that trial, understood as an operational process.

The first variable — the success rate of a trial — is the probability that a drug candidate will be both efficacious and safe in humans. The current success rate for a drug entering clinical trials is only about ten percent, meaning 90 percent of all drugs fail. Most AI efforts in biology aim to boost this success rate.

The second variable is the speed of data generation — the calendar time required to run an experiment after it has started. A clinical trial is just an experiment in human subjects, and the duration of that experiment is determined by both operational and biological constraints that are largely independent of how confident we are in the drug itself. Recruiting 1000 patients across 10 sites takes time; understanding and satisfying unclear regulatory requirements is onerous and often frustrating; and shipping temperature-sensitive vials to research hospitals across multiple states takes both time and money.

Amodei’s prediction that clinical trials could be done in a single year seems to assume that improving the first variable will also compress the second; but this is not so. Even if AI can help design more effective drugs, timelines will not compress until we solve the operational and regulatory bottlenecks of trials.

Admittedly, there is a tempting counter-argument: If AI does generate better drug candidates, then perhaps clinical trials will cease to be a meaningful bottleneck. If a drug is almost certainly going to work, then trials may become a “formality,” even if, in general, they remain unnecessarily costly and long.¹ This argument is also wrong, but understanding why requires being clear about what clinical trials are actually for.

Trials serve two distinct functions: validation — confirming whether a drug works and is safe — and learning, or generating biological data to refine our understanding of a disease, a compound, and the relationship between the two.

Validation is the primary goal of large-scale Phase III trials, which come later in the process and are typically designed to support regulatory approval. While data from these studies can deepen our understanding of drugs, their main goal is to figure out whether a treatment works under defined conditions. Learning, by contrast, is the dominant aim of early-stage trials. Conducted in smaller patient populations and often using exploratory designs, these studies are not limited to simple “yes or no” outcomes. Instead, they are experiments in the fullest scientific sense: they seek to uncover how a drug behaves in the human body, and how the disease itself responds. As argued in my earlier essay, Clinic-in-the-Loop, this makes these early stage trials active engines of discovery that close the feedback loop between hypothesis and human biology.

For large “validation” trials, is it plausible that their cost will simply cease to matter in a (theoretical) world where AI makes drugs with a high probability of success? I think the answer is no, for a couple reasons.

First, unless we increase the pace and volume of the early-stage “learning trials,” it is unlikely that we will ever approach such a level of certainty in drug discovery. Today, most AI systems in drug development are trained predominantly on in vitro data and animal models. While valuable, these sources only imperfectly capture the complexity of human biology. Without large amounts of high-quality data from actual humans, we should not expect AI to generate predictions that approach near-certainty about trial outcomes.

Second, even if improved modeling could compress early-stage development timelines, every successful drug must still demonstrate benefit on an endpoint; either a clinical endpoint or a surrogate endpoint.

For many diseases, however, the relevant endpoints take a very long time to observe. This is especially true for chronic conditions, which develop and progress over years or decades. The outcomes that matter most — such as disability, organ failure, or death — take a long time to measure in clinical trials. Aging represents the most extreme case. Demonstrating an effect on mortality or durable healthspan would require following large numbers of patients for decades. The resulting trial sizes and durations are enormous, making studies extraordinarily expensive. This scale has been a major deterrent to investment in therapies that target aging directly.

Lastly, the duration of a clinical trial does not merely determine how fast an individual therapy reaches patients. It also shapes which diseases attract serious investment and which do not. In a scenario where AI produces better drug candidates, yet trials remain slow, medicines will become unevenly deployed. In that scenario, capital and innovation will flow toward indications with clear, rapidly measurable endpoints — such as oncology — where trials can be completed relatively quickly. By contrast, fields like aging, where meaningful outcomes take years or decades to observe, will continue to lag unless there is genuine innovation in endpoint development.

Osteoporosis, a progressive bone disease that primarily affects post-menopausal women, illustrates these dynamics well. Firstly, it benefits from an unusually strong preclinical model in the ovariectomized rat (OvX model). Unlike many other chronic diseases, where animal models have poor predictive validity, the OvX model reliably recapitulates post-menopausal bone loss and predicts drug response. This rat model is so good, in fact, that Phase III trials for osteoporosis succeed 83.7 percent of the time, substantially higher than the cross-indication average of roughly 57.8 percent at the same stage.

Given the existence of a good pre-clinical model that allows us to select higher quality candidates and the scale of unmet need in osteoporosis, one might expect it to attract sustained and substantial investment. But instead, the opposite has occurred. Today, only two drug candidates remain in late-stage clinical development for osteoporosis.

The primary reason is that Phase III osteoporosis trials are exceptionally large, long, and expensive to run. The core challenge lies in the endpoint: fracture reduction. Fractures are relatively infrequent events, even in high-risk populations, and they happen unpredictably. To demonstrate that a new therapy meaningfully lowers fracture rates compared with standard of care, trials must wait for enough fracture events to accumulate to produce statistical confidence.

Because the event rate is low and influenced by many factors beyond bone strength — such as fall risk, age, and comorbidities — the signal-to-noise ratio is modest. As a result, Phase III osteoporosis trials typically enroll 10,000–16,000 participants and follow them for three to five years. The sheer scale and duration of these trials push costs to between $500 million and $1 billion. Thus, investment into osteoporosis drugs slowed not because the biology failed or drug candidates lacked promise, but because the cost of proving benefit became prohibitively high.

Osteoporosis is just one example where trial size and costs deter investment. But there is broader empirical evidence in this direction. A 2015 study examining oncology R&D found that hematological cancers — where the FDA accepts short-term surrogate endpoints in roughly 92 percent of approvals, allowing for shorter trials — attracted 112 percent more private R&D investment than solid tumors, where surrogate endpoints are used in only about half of cases. The authors traced this disparity to commercialization timelines. The shorter trials used for the former preserve more of a drug’s effective patent life, improving expected returns and drawing capital. Each one-year reduction in bringing a new therapy to market was estimated to increase R&D investment by between 7 and 23 percent.

If we want AI models to actually accelerate “therapeutic abundance,” then, we must first find ways to speed up these large validation trials. And to design better drugs in the first place, we must find ways to collect in-human data in early-stage “learning” trials much faster, too.

Regulatory Friction

The best way forward is to reduce operational and regulatory friction. AI tools can already help at the margins by automating submission drafting, improving site selection, matching patients more efficiently, and streamlining data workflows. But without deep regulatory reform, this is unlikely to shrink trial timelines or costs at scale.

One regulatory lever we could pull is to implement more high-quality surrogate endpoints. A clinical endpoint directly measures how a patient feels, functions, or survives — such as prevention of stroke or a reduction in fractures. A surrogate endpoint, by contrast, is a measurable biological marker or intermediate outcome that reliably predicts such clinical benefit. Instead of waiting years to observe clinical outcomes, trials that rely on surrogate endpoints can measure signals much earlier.

AI tools can contribute to the development of better surrogate endpoints, such as by identifying promising biomarkers, analyzing cross-trial datasets, and modeling causal relationships between intermediate signals and clinical outcomes. But here, too, technical capability is only part of the story. Institutional reform is likely to be the binding constraint. As my case study of the 12-year effort to qualify bone mineral density (BMD) as an endpoint for osteoporosis trials illustrates, the bottleneck was not scientific capability. Instead, the core barriers to faster progress were fragmented trial data scattered across sponsors, weak funding incentives for what is effectively a public good, and an unnecessarily lengthy and opaque regulatory pathway.²

For AI to generate high-quality candidates — the kind that might, one day, push success rates of drug candidates so high that trials become more of a formality — it also needs rich, dynamic data as input. But remember that such data can only come from trials in people (mice are nice, but most animal results simply do not translate.) This, in turn, creates a feedback loop: better AI models require better trial data, and better trial data requires running trials. The loop is only as fast as its slowest component, the trial itself.

A regulatory structure modeled after Australia’s Clinical Trial Notification (CTN) framework — administered by the Therapeutic Goods Administration — offers a concrete example of the kind of policy push that could speed up these types of trials. There, most early-phase trials proceed after approval by a Human Research Ethics Committee (HREC), with notification rather than pre-approval by the regulator. The regulator retains inspection powers and the authority to halt unsafe studies, but does not duplicate the scientific review already conducted by the clinician-scientists and toxicologists embedded in HRECs. The result is that clinical trial sites can begin giving drugs to patients much sooner (about two times faster than in the United States, according to informal interviews with industry leaders).

In the United States, by contrast, Phase I trials typically require submission of an Investigational New Drug (IND) application to the U.S. Food and Drug Administration before initiation. This dual review — by both an IRB and the federal regulator — creates redundancy that lengthens the feedback loop. A CTN-like model for Phase I trials could preserve safety oversight while shifting scientific and toxicological reviews to accredited, transparently governed IRBs with expanded expertise. The FDA would retain the power to inspect, impose clinical holds, and intervene in high-risk cases, such as for novel gene therapies. But for the majority of small-molecule first-in-human studies, the default could be notification rather than permission.

My criticisms are not meant to imply that AI is irrelevant to trials; that’s certainly not the case. But many of the bottlenecks that determine trial speed and cost are coordination, institutional and regulatory problems, and they cannot be solved by technology alone.

Australia’s system allows early-stage, investigator-led trials to start significantly faster—often twice as fast as in the U.S.—by bypassing the heavy-duty Investigational New Drug (IND) requirements intended for large-scale commercial trials. Australia has operated this way for 30 years with no difference in adverse safety events, yet the U.S. continues to regulate small, bespoke, tightly monitored studies as if they were massive Phase III trials.

It is, frankly, ridiculous that we have left this “Australian advantage” on the table for three decades while our own domestic pipelines have been throttled by administrative inertia. These small-scale, investigator-initiated trials are not just “preliminary” steps; they are the most information-dense experiments we can perform. In Australia, a researcher with a brilliant idea can move from bench to bedside in weeks.

U.S. companies themselves have increasingly started to move their early-stage trials to Australia where possible. So much so that in informal interviews with founders I have heard of the problem of the Australian trial system becoming “too clogged”.

The regulatory burden associated with launching clinical trials in the United States significantly harms American patients—especially those facing life-threatening illnesses such as cancer. Although the U.S. continues to lead the world in biomedical research and academic science, many terminal cancer patients struggle to access cutting-edge, experimental treatments. Instead, they are often forced to wait for full regulatory approval before these innovative therapies become available.

Implementing such changes would not only help current patients, but also future ones. It is perhaps the highest leverage thing we can do from a policy perspective to speed drug discovery. That is because it is precisely early-stage trials of this kind that enable something I call “Clinic-in-the-loop”: iterative learning in humans, the most important type of learning in biology.

We know this is crucial because some of the most revolutionary therapeutic breakthrough of the last decade (e.g. CAR-T cell therapy) was the hard-won product of two decades of small-scale, iterative “failed” early-stage trials.

To illustrate why these “information-dense” early-stage trials are the true engines of progress, I will draw heavily from an essay I wrote a couple of months ago for Asimov Press titled “Clinic-in-the-loop.”

Clinic-in-the-loop

A couple of months ago, I co-authored an essay with Jack Scannell, arguing that making trials more efficient and informative is essential to breaking Eroom’s Law. Critics of our essay, however, argued that making clinical trials more efficient risks treating biotechnology like a casino.

In their view, making it easier to run clinical trials would risk allowing more potentially harmful drugs to be tested in patients and, instead, biotechnologists should focus on making better drugs that are more likely to gain approval. These critics see Clinical Trial Abundance as accepting the status quo of drug development rather than challenging it.

But this is a misunderstanding.

In fact, Clinical Trial Abundance and better hypotheses for drugs are not merely compatible, but self-reinforcing. Faster testing in the clinic creates a feedback loop: ideas become trials, trials generate rich data (including both successes and failures), these data improve models, and better models inform the next generation of ideas. In this view, the clinic is not an endpoint of discovery but a central component of it.

To understand why clinical abundance is important, we must step outside the prevailing view of clinical testing as a mere “validation step” for scientific ideas. The familiar funnel metaphor of drug discovery, depicting a linear progression from basic science to regulatory approval, reinforces the flawed notion of clinical testing as a passive filter designed to screen pre-existing ideas. While this model is narrowly correct in a regulatory sense, it obscures the clinic’s role as an active engine of discovery.

The reality is that clinical trials rarely just deliver a “yes/no” verdict on a drug’s efficacy. Instead, the history of drug development shows that many successful therapies emerged only after initial versions failed in specific, informative ways. When a trial fails, it provides a unique physiological stress test that reveals exactly where a drug’s design fell short. By collecting data from “failed” trials, we can transform negative results into experimental corrections for the next iteration.

Consider CAR-T cell therapies. Once thought implausible or risky, CAR-T therapies now deliver long-term, treatment-free remissions in cancers where relapse had been almost certain.¹ In pediatric B-cell acute lymphoblastic leukemia (B-ALL) and aggressive B-cell lymphomas, for example, CAR-T has cured patients who, previously, had been given only months to live.

CAR-T therapy works by turning a patient’s own immune cells into living drugs. Doctors collect T cells from the blood, genetically reprogram them to recognize a protein on cancer cells, and reinfuse the modified T-cells into the patient. These engineered cells expand inside the body, move to tumor sites, and destroy malignant cells.

In 2017, the FDA approved Kymriah, the first CAR-T therapy, for children and young adults with relapsed or refractory B-ALL, a cancer of arrested development in which immature blood cells, specifically B cells, multiply out of control while failing to mature into viable immune cells. Relapsed B-ALL is the most severe form of the disease, because it means the cancer has returned after prior therapy. Even with aggressive care, only 10-20 percent of patients with relapsed or refractory B-ALL survived beyond five years.

Against this backdrop, Kymriah received accelerated approval from the FDA based on results from the Phase II ELIANA trial, a global, multicenter study sponsored by Novartis. In ELIANA, 82 percent of treated patients achieved complete remission, and subsequent follow-up analyses revealed that five-year survival rose to approximately 55-60 percent.

ELIANA was not a sudden breakthrough, though. It was, rather, the culmination of nearly two decades of clinical studies. During this period, CAR-T therapies evolved through repeated failure in the clinic, as careful studies of underwhelming results spurred new ideas to correct them. The ELIANA trial was led by investigators at the University of Pennsylvania, a group that had spent years studying CAR-T cells directly in patients well before regulatory approval.

In the mid-2000s, the earliest CAR-T therapies first entered human testing. And they emerged from a fundamental question: is it possible to engineer and redirect the T cell’s innate killing power against malignant cells?

Two well-established biological concepts made this seem plausible. First, T cells are extraordinarily cytotoxic.² However, their natural activation is governed by a “layered permission” system, meaning they cannot recognize targets directly, but must wait for other cells to process and present protein fragments in a precise molecular context. While this evolutionary safeguard keeps us from being attacked by our own immune system, it also provides cancer with many opportunities to evade detection by suppressing these signaling pathways.

To bypass these safeguards, researchers relied on a second insight: the ability of antibodies to bind directly and precisely to proteins on the surface of cells, called antigens. By equipping T cells with a synthetic Chimeric Antigen Receptor (CAR), bioengineers created a functional shortcut that bypassed the need for permission systems. This receptor uses antibody-style recognition to lock onto a cancer cell and is wired directly to CD3ζ, a signaling molecule that triggers the T cell’s internal “kill switch.” The moment the receptor engages its target, it flips the internal switch, activating the cell’s killing program.

In laboratory experiments, these first-generation CAR-Ts were formidable, displaying antigen recognition and potent killing power against tumor cell lines. Yet, this in vitro prowess vanished in patients and did not yield durable clinical responses. Understanding why this happened, though, was not simple. The failure could have been caused by a breakdown in in vivo antigen recognition, poor signaling strength, or other defects that only emerged after the cells were injected into the body.

Progress in understanding why came from treating first-in-human trials not simply as therapeutic attempts, but as opportunities to learn. These information-dense studies were conducted throughout the mid- and late-2000s and were relatively small (usually enrolling fewer than ten patients). However, they were designed to be maximally revealing. Researchers used many tools to monitor CAR-T persistence and activity in the body, turning information from a small number of patients into a mechanistic understanding.

One such tool was quantitative PCR (qPCR), a lab method that detects and counts specific DNA sequences, which allowed researchers to measure how many CAR-T cells were in patients’ blood. This showed that CAR-T cells successfully entered the body and were easy to detect after infusion. But the signal quickly faded, suggesting that the cells died off quickly. Other experiments shed light on the problem: CAR-T cells could recognize and eliminate cancer cells in patients — meaning antigen recognition was working — but their functional activity fell over time, suggesting that something in the blood was blocking them.

At this point, the diagnosis of why first-generation CAR-T therapies were failing matched long-standing insights from basic immunology. Decades of research had shown that T cells are not governed by a single on-off switch: signaling through CD3ζ provides only the first activation signal. To keep working, T cells need additional “co-stimulatory” signals,³ delivered through receptors such as CD28 or 4-1BB. First-generation CARs had been designed to deliver signal one without signal two, which explained their poor performance.

This hypothesis guided the next wave of clinical experiments, which investigated whether adding a costimulatory domain would make CAR-T cells more effective at clearing tumors in vivo.

The field-defining result came from Carl June’s group at the University of Pennsylvania. June and colleagues explored a costimulatory domain called 4-1BB. In a first-in-human study published in 2011, they treated a patient with chronic lymphocytic leukemia using CAR-T cells containing both CD3ζ and 4-1BB. They also administered a dose that was remarkably small by cell therapy standards at that time: just 1.5 × 10⁵ CAR-T cells per kilogram of body weight. (A first-generation CAR-T trial targeting renal cell carcinoma, published in 2013, used a dose more than 100-times higher.)

What followed was unprecedented. The CAR-T cells multiplied more than a thousandfold in patients, at their peak comprising a large fraction of immune cells in the blood. The CAR-T cells also persisted for months. Now, at last, CAR-T cells were a long-lived and self-maintaining immune population. Many cancer patients treated with these second-generation CAR-T therapies have achieved complete remission.

The next step, though, was asking whether the same CAR-T behavior would work in a faster, more aggressive blood cancer, such as B-ALL. In 2013, Carl June’s lab reported striking results in two children with relapsed B-ALL, again showing that the engineered T cells could multiply, persist, and drive cancer into remission.

All of these lessons were built into ELIANA, the study that ultimately supported Kymriah’s approval. Led by Stephan Grupp, who had treated the earliest pediatric patients and worked closely with June, ELIANA translated the early insights into standardized practice. This trial codified chemotherapy given before CAR-T infusion, scaled up cell manufacturing, and measured success using tools like qPCR.

Viewed through this lens, clinical trials are not an alternative to basic science, but rather a mechanism within it that closes a feedback loop. Foundational immunology, antibody engineering, and molecular biology made first-generation CAR-T cells possible in the first place, but early human trials quickly revealed that these designs were incomplete and suggested ways to fix them.

Yet theory alone did not prove this would work; the expansion and persistence observed with 4-1BB–based CAR-T cells came as a genuine surprise even to the therapy’s designers. “It was unexpected,” they reported, “that the very low dose of chimeric antigen receptor T cells that we infused would result in a clinically evident antitumor response.”

This shows why the “casino biotech” critique is flawed. It assumes that experimentation simply reveals a fixed probability of success. But trials can change those probabilities. When clinical testing is understood as part of a continuous feedback system, optimizing trial efficiency is not about accepting failure but about learning fast enough to make success more likely.

The most discovery-rich experiments are often not massive Phase III trials, either, but small, academic, investigator-initiated studies that sit close to the design loop.⁴ These are also the trials most burdened by regulatory, institutional, and manufacturing bottlenecks. And also the trials that adopting an Australia-like model would impact.

How the Australian model works in practice

The Australian Clinical Trial Notification (CTN) framework differs fundamentally from the current U.S. system in how authority is allocated between ethics committees and the national regulator, and when regulatory scrutiny occurs. In Australia, the Human Research Ethics Committees (HRECs) conduct the primary scientific, ethical, and safety review of a proposed early-stage clinical trial. Once an HREC approves a study, the sponsor submits a streamlined online notification to the Therapeutic Goods Administration (TGA), and the trial may begin shortly thereafter.

There is no requirement to submit a full Investigational New Drug–style dossier to the regulator before initiation for these early-stage, bespoke trials. By contrast, in the United States, sponsors must submit a comprehensive IND application to the Food and Drug Administration (FDA) under 21 CFR 312 before beginning a clinical trial. The FDA performs its own scientific, toxicology, and chemistry, manufacturing, and controls (CMC) review, and Institutional Review Boards (IRBs) separately review ethical considerations under 45 CFR 46. This results in parallel review streams and often significantly longer timelines before a trial may commence.

In practice, the Australian model delivers what can be described as stage-appropriate review. Regulatory scrutiny is not diminished. Rather, it is calibrated to the specific risks and uncertainties that characterize that phase of development. Early-stage trials have high failure rates, and many investigational products never move beyond first-in-human studies. Requiring extensive upfront documentation and full manufacturing packages for programs that are likely to stop early adds significant cost and delay without meaningfully improving participant safety at that stage.

Early development is a learning process. Sometimes even the underlying hypothesis is reconsidered as real human data become available. A regulatory framework that allows this kind of rapid, well-monitored iteration better reflects how drug development actually works. By focusing oversight on the risks that are most relevant in early trials—rather than imposing requirements designed for later-phase studies—the Australian system promotes efficient evidence generation while still maintaining strong safety protections for participants.

Australian site activation averages approximately two months from final protocol, and TGA acknowledgment of CTN submissions occurs within days. In the United States, IND review and resolution of regulatory questions frequently extend for several months before enrollment can begin. Such a difference in timelines can be particularly important for an early-stage biotech company struggling to survive on limited cash.

Why has this not been done before?

Knowing that Australia has operated this framework safely for 30 years—and that these trials are the highest-leverage instruments in biology—the question becomes: why has this been left on the table? Why is the U.S. biotech engine idling at a red light that doesn’t need to be there?

The answer lies in a fundamental distortion of Washington’s health policy priorities. In D.C., “health policy” is almost exclusively synonymous with demand-side economics: Medicare, drug pricing, and reimbursement. The supply-side—the actual machinery of how we discover and test new biology—is remarkably ignored.

This policy stagnation is a classic “tragedy of the commons” problem. There is no natural, deep-pocketed constituency for early-stage trial reform:

• Big Pharma uses its political capital primarily to protect reimbursement and pricing. For a multi-billion-dollar incumbent, saving a few million dollars or six months on an early-stage trial is a rounding error. Furthermore, their model is often to let others take the early risks and then acquire the survivors once they’ve reached later-stage validation, meaning that they often are not the ones carrying out these trials. By the time a biotech has produced a drug that looks promising, a few millions extra due to early trial costs is nothing.

• Small Biotech would be a more direct beneficiary of these reforms, but these startups rarely have the time, money, or lobbying presence to advocate for systemic change in D.C. They are too busy trying to survive the very bottlenecks we are discussing.

• The Commons. Of course, the primary beneficiaries of these reforms are ultimately The Commons. If more drugs are tested in early-stage trials, all of us get more and better drugs. However, those doing advocacy on behalf of the commons have often had an approach that focuses much more on the distribution of what exists (“demand-side”) than creating abundance (so working at the “supply-side” level).

This is why the emergence of the “Progress” community is so vital. Organizations like The Roots of Progress and IFP are filling the gap left by this tragedy of the commons. They are doing the “un-incentivized” work of advocating for the structural plumbing of science—the boring, high-leverage regulatory reforms that Big Pharma ignores but that the future of medicine depends on.

We need progress-minded individuals to step into these gaps, supported by institutions that understand that supply-side policy reforms centred around Clinical Trial Abundance are key to the future of medicine. Senator Cassidy’s report is a rare sign that this message is finally reaching the most important echelons and it is an opportunity we cannot afford to waste.

I’d like to offer some thoughts of my own, particularly around the timing of this guidance and what it might mean. FDA’s drug center has been facing pressure to embrace Bayesian statistics for decades. I worked at the FDA for over a decade, and saw frequent pushes to get a Bayesian guidance released. So why is the guidance coming out now? And will it actually change how clinical trials are designed?

Why go Bayesian?

Before we dig into those questions, it’s helpful to understand the statistical approach drug companies have traditionally used to prove that their drugs work. For decades, companies have used an approach rooted in frequentist (non-Bayesian) statistics: they conduct two, “pivotal” randomized controlled trials, each designed to prove that the drug has a clinically meaningful benefit. The trials are considered successful if each produces a two-sided p-value no greater than 0.05. While FDA has never officially required trials to meet this standard, in the past they have endorsed this approach.

Advocates for the use of Bayesian statistics would point out several problems with this traditional frequentist approach. First, there is the p-value itself, which is defined as “the probability of getting results at least as extreme as the ones you observed, given that the null hypothesis [of no drug effect] is correct”. It’s a concept that strains the comprehension of even expert scientists (I had to copy the definition from Wikipedia to make sure I got it right, which is what I suspect most responsible scientists do when asked what a p-value is). P-values and frequentist statistics are useful in drug regulation, but Bayesian methods give us a simpler and useful metric: the probability, given the data observed in the study, that the drug meets a threshold of effectiveness. That is a value that is easier to understand and often more useful for decision-making.

A more significant shortcoming of the frequentist approach, from a Bayesian perspective, is its tendency to discard useful information. When evaluating whether a drug should be approved, the FDA has the opportunity to evaluate multiple sources of evidence: There are, of course, the drug company’s pivotal studies. But they might also examine clinical studies from earlier phases, studies of similar drugs for the same condition, or studies of the same drug for similar conditions. If the drug is already being used in the clinic, FDA might also want to look at patients’ experiences on the drug in the real world. All of these sources of evidence are valuable, but the traditional frequentist paradigm discards much of it, relying solely on those pivotal studies to provide evidence of the drug’s effectiveness.

Bayesian statistics can help with these problems: It lets reviewers consider all of the data about a drug – not just the data collected in a given study – and it produces results that are easier to interpret and more relevant to regulators, clinicians, and patients. These benefits come with caveats: the results of a Bayesian study depend on modeling decisions made upfront, including the choice of the Bayesian prior (more on that below). That’s why FDA’s guidance suggests that Bayesian studies should demonstrate strong “frequentist operating characteristics” and stress-test assumptions. But with rigor and careful planning, Bayesian statistics can help us make better use of the information we have.

So why has this taken so long?

The proximate reason we are seeing this guidance now is that FDA committed to release the guidance in its 2022 user fee negotiations with industry. But the truth is, this guidance could probably have been released sooner. FDA has been discussing the use of Bayesian statistics for decades – FDA’s medical device center published its own Bayesian guidance nearly 20 years ago. I think the real reason this guidance is coming out is that, for the agency, the benefit of applying a Bayesian approach to drugs finally outweighs the downsides.

And despite its merits, Bayesian statistics does have a big downside: it requires the construction of the Bayesian prior probability distribution, or prior. The prior describes the range of effect sizes we think are plausible before the trial begins based on existing evidence, and is crucial in determining how the study data will be interpreted. While there are best practices in constructing the prior, it inherently requires some degree of subjective judgment on the part of the study designers.

It’s easy to see how the prior might raise red flags for regulators. If the prior is subjective, how do you decide what it should be? And how do you keep drug companies from gaming the system by choosing a favorable prior? Yet while these concerns are real, I don’t think they’re the main reason that FDA has been hesitant to embrace Bayesian statistics. After all, there is always an element of subjectivity in drug review. FDA tries to make its standards clear, but they acknowledge that they must make subjective judgments on what evidence should be collected and how it should be weighed. And the FDA is perfectly willing and able to scrutinize study design choices and the study data itself. They’re not likely to be fooled by drug companies who play games with statistics – even if the statistics are Bayesian.

If I had to guess, I suspect the biggest problem for FDA was not the subjectivity of the prior; it was the exercise of putting numbers behind those subjective judgments. For fans of Bayesian statistics (and legions of statistics nerds), the fact that the Bayesian prior can be quantified is one of its greatest strengths. Capturing relevant information in a Bayesian prior feels more rigorous and rational than relying solely on subjective judgment – and it’s much better than simply throwing that relevant information away.

But I suspect this is not how most normal people – FDA reviewers included – feel. The FDA has long stressed the importance of “clinical judgment” in reviews. The Bayesian prior threatens that. After all, if a reviewers’ clinical judgment is a key factor in review decisions, any attempt to translate that clinical judgment into numbers risks diluting and distorting that judgment.

In the past, FDA has been even more explicit in rejecting the quantification of its review decisions. In a 2013 report, they considered and rejected the idea of using numerical weights of benefits and risks to help them review drugs. The process of assigning weights, they argued, involved “numerous judgments that are at best debatable and at worst arbitrary.” While the construction of a Bayesian prior does not preclude reviewers from exercising judgment, it does probably elicit a similarly negative reaction.

And yet, here we are, with the Bayesian guidance in hand. The guidance even goes so far as to suggest that companies could explicitly quantify benefits and risks in their trial in the form of a Bayesian “loss function.”

Why the shift? Perhaps it’s because the alternative approach has grown less tenable.

FDA’s existing statistical approach no longer made sense

Against the “debatable and arbitrary” assumptions required by Bayesian statistics, the perceived simplicity, objectivity, and consistency of a frequentist approach must have seemed appealing to the FDA. In exchange for ignoring prior data, the frequentist approach gives us a consistent statistical approach with a consistent interpretation. The frequentist approach must have been particularly appealing to FDA in the 1960s, when it first introduced its efficacy standard. At that time, the agency feared being barraged by poorly conducted studies of questionable products. Indeed, FDA articulated this fear as recently as 1998 in its guidance on how it evaluates drug effectiveness:

The inherent variability in biological systems may produce a positive trial result by chance alone. This possibility is acknowledged, and quantified to some extent, in the statistical evaluation of the result of a single efficacy trial. It should be noted, however, that hundreds of randomized clinical efficacy trials are conducted each year with the intent of submitting favorable results to FDA. Even if all drugs tested in such trials were ineffective, one would expect one in forty of those trials to “demonstrate” efficacy by chance alone at conventional levels of statistical significance.

In other words, the agency needed to filter out the junk from the “hundreds” of randomized studies they might see each year, and relied on frequentist statistics, hypothesis testing, and replication to help them achieve this.

Nowadays, the circumstances the agency faces are different. If the agency ever faced a barrage of “hundreds” of randomized trials, that is no longer the case. In 2024, FDA approved 50 new molecular entities on the basis of only 75 trials. And it probably felt unfair to assume that most or all of the drugs that underwent these trials were ineffective (even if many would ultimately fail); many of those trials were done only after extensive study, including prior use and research in humans.

The frequentist framework no longer made sense. Even before the Bayesian guidance was released, FDA understood that, despite their reliance on frequentist statistics, these trials could not be reviewed in an evidentiary vacuum. As far back as the 1990s and early 2000s, FDA has found ways to “borrow” from prior information to support efficacy determinations: FDA has made use of “seamless” phase 2/3 trials in cancer, natural history studies, and even mechanistic data to support efficacy determinations. They usually described this as making use of “confirmatory evidence” or “ the totality of evidence”. Now, FDA has departed further from the traditional approach, requiring only one trial instead of two.

More broadly, the key regulatory question has shifted: instead of “how do we filter out the junk science” – a goal well-supported by frequentist methods – FDA and drug companies are trying to use scarce clinical data as efficiently as possible to make informed determinations about each drug that crosses its desk. That’s where Bayesian statistics shines.

The choice FDA now faces is not whether or not to rely on prior evidence; it’s whether to do so awkwardly, in its current frequentist framework – in which that borrowing is implicit and unstructured; or using the more structured approach offered by Bayesian statistics. The publication of this guidance suggests that FDA is starting to embrace the more structured approach.

Will this guidance lead to change?

Are we about to embark on a new Bayesian era at FDA? Perhaps, but first, drug companies need to actually adopt the methods described in the guidance.

I expect the process to be slow. Despite FDA’s endorsement, this is still just a draft guidance, and few drug companies will want to be among the first early-adopters to try out new Bayesian approaches with agency review teams. Drug companies have famously risk-averse cultures, particularly when it comes to running their most expensive and high-stakes pivotal trials. And even though FDA has signaled that it is amenable to Bayesian approaches, most will wait to adopt them until they see FDA approving applications that make use of them.

Adoption will also be slowed by a lack of capacity and expertise. Drug companies and the FDA will both need to train their staffs and get familiar with the technical details of designing and approving Bayesian studies, negotiating over priors, and understanding the benefits and limits of the approach. In the meantime, drug companies may find themselves more comfortable working within the existing frequentist approach, knowing that the agency has grown increasingly comfortable with accepting “confirmatory evidence” of effectiveness in lieu of data from trials.

But in the longer term, I am betting that Bayesian approaches will become far more common. Drug companies are going to find it difficult to resist the opportunity to make more efficient use of limited clinical evidence; particularly in rare diseases, pediatric indications, and other areas where evidence is particularly scarce. If enough pioneering companies are willing to take the chance on something new, others will follow. And if we find new ways to strengthen the quality of prior evidence, whether through clinically validated AI-driven “virtual cells” or simply faster and more abundant early-phase human trials, the case for Bayesian methods will grow even stronger.

I’d like to thank my co-bloggers , , , and for their input on this piece.

That uncertainty doesn’t just slow individual programs, but also reshapes behavior across the biotech ecosystem in a negative way. Lack of regulatory clarity fosters a culture of defensive decision-making, where companies avoid innovative clinical development strategies and default to more conservative, expensive paths simply because they lack confidence in how the FDA might respond. Over time, this dynamic drives up trial costs, prolongs timelines, and ultimately reduces the number of truly innovative therapies that reach patients.

Unfortunately, a decision made by the FDA very recently threatens to make an already existing problem even worse.

Today, the U.S. Food and Drug Administration issued a refusal-to-file letter to Boston-based biotech Moderna regarding its Phase III trial for an mRNA influenza vaccine in adults 50 and older. The letter was signed by Dr. Vinay Prasad, the head of CBER (Centre for Biologics Evaluation and Research)—an unusual move at this stage of the process, suggesting the decision was not routine.

A refusal-to-file (RTF) letter means the FDA has determined that a company’s application is not sufficiently complete or adequate to even begin formal review. It halts the approval process before any full evaluation of safety and efficacy takes place, forcing the sponsor to address the cited deficiencies and resubmit.

The justification offered by the FDA is that Moderna used the wrong comparator in adults over 65. Specifically, the agency argues that because higher-dose or adjuvanted influenza vaccines are recommended for older adults, Moderna’s use of a standard-dose comparator rendered the trial not “adequate and well-controlled.”

If that requirement had been clear from the outset, this would be an unremarkable story about trial design. Companies adapt to clear standards and requirements from regulators all the time. But Moderna’s CEO, Stéphane Bancel, says the FDA had previously signaled that a licensed standard-dose comparator would be acceptable—and has only now changed its position after the trial was completed. If that is accurate, it would mean that roughly $750 million—what the New York Times reports the trial cost—was effectively committed under regulatory assumptions that no longer hold.

Adding to the confusion, entrepreneur Dr. Jing Liang noted on X that publicly available trial data appear to show Moderna conducted analyses involving higher-dose comparators in older adults. If accurate, it would raise further questions about the basis for issuing a refusal-to-file letter in the first place.

No matter which account ultimately proves correct, this episode leaves us with two uncomfortable possibilities. If the FDA changed its regulatory position after years of clinical development, that reinforces a persistent concern about regulatory inconsistency. If, alternatively, Moderna met the agency’s highest level of stated expectations, actually did use a high-dose comparator and still received a refusal-to-file letter, the implications are even more serious, raising questions about transparency and internal coherence at the agency.

In this post, I will focus on the “least bad” scenario—that the FDA revised its stance on a trial design it had previously indicated was acceptable. Even that interpretation carries consequences far beyond a single influenza vaccine. When regulatory expectations shift late in development, companies internalize the lesson and design more defensively, avoid innovative comparators and add layers of redundancy. Over time, this entrenches a culture of safetyism and regulatory aversion. Sponsors begin optimizing not for better science, but for minimizing the chance of procedural surprise. In time, this leads to a ballooning of costs.

Why regulatory inconsistency is bad

Even if we assume the “least bad” scenario—that the FDA has indicated a lower-dose comparator would be acceptable—it still reinforces one of the agency’s most persistent structural problems: regulatory inconsistency and opacity.

Translating science into a clinical product already takes more than a decade and billions of dollars and only about 10% of drug development programs succeed. Most of that time and capital is spent in clinical development—running trials, scaling manufacturing, and interacting with the FDA. Yet over a 10+ year program, a company may have only three to five formal milestone meetings with regulators. Those meetings determine pivotal trial design, endpoints, manufacturing standards, and statistical plans.

Given that formal meetings with regulators are limited and FDA guidance is often broad and caveated, companies therefore make pivotal decisions under real uncertainty. And when the stakes involve hundreds of millions of dollars and years of work, the rational response is caution. This is what Adam Kroetsch meant when he said that “vague regulation breeds safetyism.” Sponsors layer on extra monitoring, avoid novel trial designs, and default to conventional manufacturing processes even when more efficient technologies exist. Each individual decision makes sense. Collectively, they make drug development slower and more expensive.

The slow adoption of risk-based monitoring (RBM) is a clear example of the downsides of regulatory opacity. RBM focuses oversight on the highest-risk elements of a study and could reduce trial costs by up to 30% without compromising safety or data integrity. The FDA has encouraged its use for years. Yet, despite this clear advantage, adoption remains uneven. Surveys show that sponsors hesitate because they worry regulators may not accept RBM in practice—even if official guidance supports it.

The same fear-driven conservatism appears in other areas of trial design. One study found that roughly one-third of data collected in clinical trials may be redundant. Sponsors collect this data due to mistakenly believing this is what regulators might ask for. That redundancy translates directly into higher costs, and longer trial timelines.

It’s trying to solve the problem of regulatory opacity, which has such a negative downstream impact on innovation, that has led me to launch, with OneDaySooner, the openFDA Fund—a centralized repository of past regulatory interactions designed to bring greater transparency to the often opaque FDA decision-making process.

The future

Unfortunately, this is not the first late-stage reversal under the current FDA. In November 2025, biotech company uniQure announced that the FDA had reversed its prior position on the company’s gene therapy AMT-130 for Huntington’s disease. Although the agency had previously indicated that Phase 1/2 data—compared against an external natural history control—could support a Biologics License Application (BLA) under the accelerated approval pathway, it now determined that the existing data are insufficient for submission.

The shift came despite promising results showing statistically significant slowing of disease progression and supportive biomarker improvements. CEO Matt Kapusta expressed surprise, noting that the feedback represented a “drastic change” from guidance provided in late 2024.

The current FDA leadership has openly acknowledged growing competitive pressure from China and has put forward a number of thoughtful proposals to modernize the regulatory framework. But reform does not end at changing rules. These rules must also be predictable. Regulatory clarity and consistency are foundational and as important as the quality of the regulations themselves. For the sake of the US biotech ecosystem, I hope that the FDA will reverse course in what seems a trend towards decreased regulatory consistency and clarity.

The first statin, mevastatin, was discovered in 1976 by Akira Endo, a biochemist at Sankyo pharmaceuticals in Japan, from a fungal mold growing on rice samples at a grain shop in Kyoto.1

The drug was clearly effective in reducing cholesterol levels, but in 1980, Sankyo abandoned its clinical trials after studies in dogs appeared to show intestinal tumors. The details of these findings were never formally reported; at the time, there were only rumors about what had led the company to shut down its trials.

Meanwhile, Merck was testing a nearly identical compound2, lovastatin, and heard about the decision. They expected it would be a multimillion dollar drug, and so took the sudden halt seriously: what if lovastatin would cause the same side effects?

Merck paused its own trials, and asked Sankyo for further details, but the latter declined to share them.3 So Merck conducted further toxicology studies of their own to understand whether the risks were real, which eventually showed the changes were benign and could be reversed. Three years passed before Merck resumed their clinical trials. When they were completed, they showed a large reduction in blood cholesterol reduction and few side effects. Lovastatin (now known as ‘Mevacor’) became the first statin approved in 1987.4

Decades later, after millions of people have been treated and monitored, the evidence shows no increased risk of cancers from statins. They have likely saved millions of lives by reducing the risks of heart attacks and strokes by roughly 20%, and annual mortality rates by 10%.

I’ve been wondering, what if Sankyo had shared its data publicly?

It’s possible that researchers might have analyzed the data further and reached the conclusions sooner. Merck could have avoided redundant studies, and both companies might have reached the finish line, speeding up the arrival of a drug class that would go on to save millions of lives.

Another possibility is that Merck might have instead funded trials for another drug candidate – one that was less similar in structure to Sankyo’s drug, as some within Merck suggested doing. Alternatively, if Merck had never heard about Sankyo’s halted trials at all, they might simply have proceeded, and still correctly found no evidence of harm in their own clinical studies in human patients.

The counterfactual from the story is ambiguous, but it shows that pharmaceutical companies respond to data from other firms. It also highlights a tension: transparency, or even partial transparency, can affect experimentation. When researchers or firms learn early that a similar approach has failed, they may shift toward safer, more predictable projects, at the cost of unexpected successes.

At the same time, access to the underlying data would have allowed other scientists to scrutinize the findings directly, rather than rely on rumors. That would have allowed for better decisions about what precisely to investigate further – helping understand which doses it occurred at and what types of cellular changes were seen, for example – and make more informed decisions about their own research and development plans.

Subscribe now

Medicine is an area where transparency has been considered valuable for a long time, and genuine progress has been made in improving it, as I’ll describe in a future post. The consequences extend well beyond the research phase; they can shape decisions about which drugs regulators approve, which treatments insurers cover, and which treatments physicians prescribe.

The stakes are unusually high. In the case of drugs like statins, such decisions touch tens of millions of patients each year5, with hundreds of millions – sometimes billions – of dollars resting on the results of trials.

Transparency at various stages of the process can shape decisions on whether to continue research into drugs, test them in clinical trials, approve them, and how to price them.

If transparency matters this much, what kind of data should be available in practice?

It can seem like an ambitious reform to ask for data at the level of individual patients so the findings can be verified and potentially re-analyzed, because research can involve sensitive personal information, and preparing datasets and documentation for public sharing can be time consuming.

But most highly-cited clinical trials say they will share data, and many say this will be anonymized data from individual patients. Moreover, a few recent efforts have tried to address this problem, including Vivli.org and ClinicalStudyDataRequest.com, with data platforms where clinical trial researchers can deposit their datasets securely.

In practice, however, data from clinical trials remains largely inaccessible and enforcement by journals is weak.

I think there are probably many benefits of transparency in clinical trials, beyond verifying data. They include:

• Better meta-analysis. When more trial results are available, researchers can perform meta-analyses that combine evidence across studies. This improves precision and allows for stronger conclusions than single studies can support, for example by helping to identify effects on rare outcomes, such as the effects of flu vaccines in reducing mortality rates, or rare but serious harms.

  • A recent meta-analysis of 19 clinical trials on statins, for example, found no increase in the risks of most side effects listed on the drugs’ labels (such as brain fog, diarrhea, pain, vision impairment, and 58 other outcomes that had showed no difference in risk between the drug and placebo). Only a few side effects were validated, such as muscle weakness, rhabdomyolysis in rare cases, and a slight increase in diabetes. With these conclusions, the researchers suggested that drug labels should be updated.

• Understanding inconsistent results. Clinical trials studying similar treatments sometimes reach different conclusions, which leaves uncertainty about whether the differences reflect chance, patient populations, study design, or other factors. By pooling data across studies, researchers can explore these sources of variation more systematically. They could, for example, try to identify when apparent contradictions stem from targeting different biological mechanisms.

• Further exploration. Transparency makes it easier to ask new questions of old data. Researchers may want to explore hypotheses that weren’t part of the original study, or revisit results in light of new evidence. When prior trial data are available, many of these questions can be answered without launching entirely new trials. This might include, for instance, when drugs initially developed for one condition show unexpected benefits for another.

• Better clinical decision-making. Doctors often have to choose between many drugs for the same condition, without having sufficient data for head-to-head comparisons. Individual trials can rarely answer questions like “Which drug performs best for these patients?” But with more data, techniques like network meta-analyses can help indirectly compare multiple treatments against each other, or derive better conclusions on how to tailor decisions to patients’ characteristics.

• Learning how to run trials better. Pooling data across many trials could also help answer questions about studies’ operational characteristics: How does remote monitoring compare with on-site testing? Which trial sites reliably deliver high-quality data? Which eligibility criteria slow down recruitment? Which practices reduce the chances of patients dropping out of studies? These questions could help design trials more efficiently, even though individual research groups rarely run enough trials to study these questions on their own.

• Reducing redundancy and wasted effort. Finally, transparency can help avoid repeating failures. Researchers can learn from what succeeded or failed to refine their hypotheses and improve their chances of developing effective drugs in the future. More recent evidence suggests that when reporting the headline results of trials became mandatory, pharmaceutical companies actively monitored competitors’ results and adjusted their own research plans accordingly, avoiding similar studies.

And the value of AI in performing many of these analyses is also constrained by access to the underlying data; the benefits depend on the availability and quality of data available to study.

Taken together, these benefits point to a broader function of transparency in medical research: it allows knowledge to accumulate more efficiently.

By making individual patient data available, studies can move beyond fixed headline results and become part of a cumulative evidence base, giving researchers and clinicians a fuller picture of data that they can use to answer broader questions.

This flow of information can influence the pace of basic research, shape drug development, guide clinical decisions, and ultimately, affect the health of millions of people.

Share

Subscribe now

Clinical trials are how new drugs are tested before becoming widely available. In most countries, like the US and UK, where we live, trials are expensive and slow. Effective drugs can be stuck in clinical trials for years, sometimes a decade or more, before the results are clear and those drugs are available to patients who need them. Clinical trials are also generally treated as a high-stakes, one-shot test to produce a final binary decision on whether a drug works, rather than as part of an ongoing evidence loop that could help accelerate innovation.

We think things could be different. Clinical trials could run more efficiently through modular components, innovative trial design, streamlined ethical approval, faster recruitment, and many other improvements. They could collect more informative data to help discover new, effective biomarkers, not just confirm older ones. And by sharing the data from them, they could help improve drugs in the future too. All of this could allow us to test more drugs in parallel and find out what works (or doesn’t) faster.

Despite major advances in AI, we believe that testing in humans – not just in animals or simulations – will remain essential for understanding effects in complex biological systems like humans, and effects that may vary between individuals and produce unexpected side effects.

Each of us – Adam, Manjari, Ruxandra, Saloni, and Witold – has been writing about these ideas on and off for years. We’ve started this blog because we want to have a common platform where we can publish more frequently, respond to news and policy change, share thoughts and start discussions, and seed ideas that can be refined and explored further.

Subscribe now

Here’s a short introduction to each of us:

is interested in accelerating medical innovation and scaling up efforts in global health. She’s previously written about why randomized controlled trials matter, why it took 141 years to develop a malaria vaccine, a prediction of why we’d get Covid vaccines in a year, and some ideas we can take from pandemic clinical trials to speed up innovation more broadly.

She works full time at Works in Progress magazine as an editor and writer, advises Coefficient Giving on clinical trial reform, and hosts the podcast Hard Drugs with Jacob Trefethen. Previously, she was a researcher at Our World in Data on global health and medicine, and holds a PhD in psychiatric genetics from King’s College London and the University of Hong Kong.

Kroetsch is interested in system-level reforms to make trials more abundant. Through the Clinical Trials Efficiency Project, he is exploring how we can make trials faster and cheaper through better policies, regulations, incentives, and infrastructure. He has written about why trials are so expensive and what clinical trial abundance might look like. He has also written about the FDA’s history and how the FDA evaluates clinical trial evidence.

Adam publishes his work on his Substack, Policy and Practice. Previously, he worked at the FDA and the Duke-Margolis Institute for Health Policy. He holds a Masters in Public Policy and Management from Carnegie Mellon University.

is interested in accelerating methodological falsification in science. As a BRAINS fellow at Speculative Technologies she explored an ambitious program to improve preclinical forecasting evaluations in drug development. Manjari writes about biomarkers, neuroimaging, and statistics at Neurostats. She runs the Surrogate Science Project, a new community effort to bring the formal logic of good proxy development from biostatistics, econometrics and machine learning to all fields.

As a consultant, she advises biotech startups on early biomarker R&D, and collaborates with tech companies to improve scientific causal reasoning of AI agents. As a member of industry working groups in pharmaceutical statistics, she seeks to bring statistical rigor to earlier stages of biomarker development. She is a sought after invited speaker in several areas of biomedical research including multi-modal brain stimulation, psychiatric neuroimaging biomarkers, theoretical metascience, and statistical innovation in biopharma. Manjari’s early academic research focused on signal processing and nonparametric decision theory. Her subsequent graduate and postdoctoral work blended causal machine learning with biostatistics to understand and intervene on complex systems found in biology and medicine. She advanced uncertainty quantification and decision science for gene therapy at Dyno Therapeutics. She holds a PhD in Electrical Engineering from Rice University; her graduate work received the ENAR distinguished paper award in Biostatistics.

is interested in how we can reverse Eroom’s Law – the observation that pharmaceutical R&D efficiency declined, despite staggering advances in basic science. Convinced that making clinical trials more efficient would play a key role, she started the Clinical Trial Abundance Initiative, a policy effort focused on clinical trials at the Institute for Progress (IFP) in 2024. She has also written extensively about fertility and biomedical progress more broadly.

She is now a Fellow at Renaissance Philanthropy, where she continues to work on Clinical Trial Abundance. Her focus at the moment is on streamlining Phase I trials, FDA transparency and Drug Development Tools (DDTs). You can read her policy work on the IFP website. She holds a PhD in Genomics from Sanger Institute, University of Cambridge.

is a statistician working on problems in global health, epidemiology, and development economics. Witold’s current research revolves around making vaccines more efficient, meta-analysis, low-cost life-saving interventions, and metascience.

Witold is a Consulting Director at Development Innovation Lab at University of Chicago, where he works on designing randomised controlled trials, a Senior Fellow at the Institute for Progress and a Research Fellow at the Center for Global Development. Previously he spent over a decade working as a consultant in the pharmaceutical industry. His PhD was on Bayesian approaches in biostatistics.

We hope you enjoy reading!

– Saloni, Adam, Manjari, Ruxandra and Witold