A simple strategy for valid inference in target trial emulations
A sample-splitting trick lets drug researchers trust their statistical results even when they adjusted the study design after peeking at the data.
The Thesis
Clinical researchers who study real-world patient data — rather than running formal randomized trials — have long faced a credibility problem: the moment you adjust your study design based on what the data looks like, your reported statistics become unreliable. This paper proposes a straightforward fix: split the observational dataset in two, use the first half to design the study, then run the actual analysis only on the second half. Because the analysis data was never touched during the design phase, the resulting confidence intervals and p-values have the mathematical guarantees they're supposed to have. The catch is that splitting the data in half reduces statistical power — you're working with less data at the inference step — and the authors do not empirically demonstrate how large that cost is across realistic healthcare datasets. If it holds up, this method could make comparative effectiveness research — studies that compare the outcomes of patients who received different treatments in real clinical settings — substantially more credible to regulators and payers.
Catalyst
Observational health databases (insurance claims, electronic health records, national registries) have grown large enough that splitting them in half is less painful than it once was — you may still have hundreds of thousands of patient-years to analyze after the split. Simultaneously, the FDA and other regulators have been actively developing frameworks for accepting real-world evidence in drug approval decisions, raising the stakes for methodological rigor. The target trial emulation framework — a structured approach to mimicking a randomized trial using observational data, developed prominently by Miguel Hernán and collaborators over the past decade — gave researchers a shared vocabulary that makes the sample-splitting discipline described here practical to implement.
What's New
The dominant prior approach in target trial emulation is to iteratively refine the study protocol using the full dataset, then analyze that same full dataset — a process that conflates exploration and confirmation in ways that inflate false-positive rates. Pre-registration offers a partial remedy, but it does not eliminate the problem when protocol choices were already shaped by data inspection before registration. This paper adapts the classical statistical technique of sample splitting — long used in machine learning for train/test separation and in econometrics for instrument selection — and applies it explicitly to the target trial emulation workflow, with the conceptual framing of pilot-to-phase-3 trial progression as a guide for practitioners.
The Counter
The core idea — split your data, explore one half, confirm on the other — is not new; it is textbook statistics. The paper's contribution is mostly conceptual reframing for an epidemiology audience, and it is not obvious that reframing alone changes practice. More importantly, the paper provides no empirical demonstration of the method on real datasets, so readers have no sense of how much statistical power is lost in realistic healthcare study sizes, which can vary enormously. For rare diseases or narrow patient subgroups, losing half the data could make a study completely uninformative — a cost the paper does not quantify. Regulatory acceptance is also far from guaranteed: the FDA's real-world evidence guidance emphasizes fit-for-purpose data quality over statistical formalism, and it is unclear whether this procedural change would actually move a skeptical reviewer. Finally, the method relies on investigators faithfully committing to the protocol after examining only the first split — a behavioral assumption that is hard to enforce or audit in practice.
Longs
- IQVIA (IQV) — largest provider of real-world health data used in exactly these study designs
- Veeva Systems (VEEV) — clinical data management infrastructure for pharma sponsors running RWE studies
- Flatiron Health (private, Roche subsidiary) — oncology real-world evidence platform
- ICON plc (ICLR) — contract research organization increasingly executing RWE alongside traditional trials
Shorts
- CROs and consultancies whose RWE practice is built on rapid, iterative full-data protocol refinement — the method requires a more disciplined, phased workflow that disrupts fast-turnaround study designs
- Pharma teams using observational data for label expansion whose existing studies lack the sample size to absorb the power cost of splitting
Enablers (Picks & Shovels)
- Large electronic health record networks (e.g., TriNetX, PCORnet) that provide the patient populations large enough to survive a 50% split
- The R 'TrialEmulation' package and similar open-source tooling that operationalizes target trial emulation protocols
- FDA's Real-World Evidence Program, which sets the credibility bar this method is designed to clear
Private Watchlist
- Aetion — real-world evidence analytics platform used by FDA and large pharma
- Concerto HealthAI — oncology real-world data and study design tooling
- Komodo Health — claims and clinical data infrastructure for epidemiology studies
Resources
The Paper
Target trial emulation has improved comparative effectiveness research by making the causal question, assumptions, and analysis plan explicit. However, target trial protocols are usually developed iteratively. After examining the data, investigators revise the protocol to reflect which target trials the observational data can realistically support. While this iterative procedure is part of normal scientific practice, it raises concerns about selective choices and invalid statistical inference. A simple procedure can address these concerns. This procedure is based on sample splitting. In the initial split, investigators explore the data to define a target trial protocol. When these choices are made, the target trial protocol is implemented on the second split. Although the investigators made data-informed choices to select the target trial protocol, the inference has the usual coverage guarantees. The procedure is created to mirror how trialists move from pilot studies to a phase 3 trial. First, they use data from pilots and early-phase trials to learn and decide on a final protocol. Then they implement this protocol and analyze a new set of data in a phase 3 trial.