Synthetic electronic health records are sold as a privacy-safe, deletable stand-in for real patient data. But the field judges their quality almost entirely on associational tests: can a discriminator tell real from fake? do the distributions match? does a model trained on synthetic transfer to real? The flagship use of this data is the opposite kind of question, real-world evidence: what is a drug's effect on an outcome, adjusting for confounders. Nobody was checking whether a "realistic" twin preserves that.
I built a causal-fidelity benchmark: reproduce a full generator zoo (copula, GAN, VAE, tree-based, nearest-neighbour) on three national primary-care cohorts, then measure how far each one moves a confounding-adjusted treatment-effect estimate away from the real-data answer. The key device is an oracle floor (the error you'd get from a perfect reproduction against a fresh real sample), which turns vague "it's a bit off" gaps into hard multiples.
- Confounding-adjusted log-odds-ratios and g-computation, cross-checked with a doubly-robust estimator so the finding is a property of the generator, not the estimator.
- A synthetic ground-truth model with a known effect, to confirm the benchmark's ranking tracks true recovery (Spearman 0.96).
- A new neighbour-based method that reimposes the exact real per-feature statistics by construction while mixing multiple neighbours, so almost no record is copied (audited at under 2.5% exact copies).
Every generator distorted adjusted treatment effects 5–20× above the oracle floor, on all three cohorts. The sharpest finding: the Gaussian copula, which has provably perfect per-feature statistics, is the single worst generator for causal inference. And the discriminator score everyone gates on turned out not to predict causal error at all. Even my own method, the least-detectable privacy-safe generator in the zoo, still mis-estimates the magnitudes. That's the strongest form of the thesis: even the best associational twin fails the causal audit.
A metric that everyone trusts is not automatically a sufficient statistic for the property you actually care about. Proving that cleanly, with an oracle floor, a counterintuitive counterexample, and adversarial cross-checks, is exactly the evaluation discipline that separates a system that looks safe to deploy from one that is.
Anonymous double-blind submission, under review; venue withheld to respect anonymity. Underlying cohorts are a licensed multi-national EHR database and cannot be redistributed; the released artifact includes a data-free synthetic-model reproduction. Figures transcribed from the working manuscript.