Measure an Automation with a Holdout Group
Keep a share of new contacts out of an automation and compare them with the rest.
How do you know a welcome series or a win-back flow adds sales, and that people would not have bought anyway? Hold a few contacts back and compare.
- Open the automation and choose the Holdout tab (owners and admins).
- Choose how many in 100 new contacts to hold back, from 0 to 50, and click Save. Held contacts get none of the automation’s emails.
Who is held is decided by a fixed formula on the automation and the contact, so a contact always stays in the same group. Changing the share only affects contacts who enter later.
The report
It compares contacts who got the automation with those held back: how many bought, orders, revenue per contact and the number of contacts on each side. Choose how many days after entering an order still counts (7 to 90).
- Significant means the difference is unlikely to be chance (a two-proportion z-test, p below 0.05).
- Not enough data or cannot tell yet is normal for a small holdout. Keep it running before you decide.