Synthetic data is being sold faster than governance can be developed. Insights leaders are under pressure to reach hard-to-find audiences, move faster, and do more with tight budgets. Providers often validate their own models with a single score as sufficient evidence. But is the data really strong enough to support the business decision? That’s the question KS&R set out to test.
01 The bottom line
Synthetic data is viable when it is governed. Every provider passed our simple study. Every model fell short on the complex one, but going deeper showed a clear path to where and how to improve. Don’t buy the promise. Buy the evidence.
Four synthetic data providers submitted production models for independent evaluation on two datasets, one simple, and one complex. Every submission was scored with the ECHO Index, a patent-pending methodology developed by KS&R.
Summary of findings
| Simple studies work | All four providers landed inside the ±3.5 fit-for-purpose window on a short, straightforward study. |
| Complex studies broke every model | All four models that took part fell far outside the fit-for-purpose window, scoring roughly fifteen to twenty times the threshold. |
| Failure is diagnosable | Question-level diagnostics showed where each model broke down. A model update and removal of selected questions for each improved fidelity. |
| Subject matter matters | A model developed by experts in banking, healthcare, and life sciences stood out on financial-literacy questions. No other provider came close to matching the human response patterns. |
| Governance takes time | Generating responses can be fast. Producing a defensible result takes collaboration and, often, several rounds of iteration. |
These results cover specific submissions on two datasets. They are not a verdict on synthetic data as a whole.
Continue reading
Complete the form below to access the executive summary.
