All articles › Automations

Measure an Automation with a Holdout Group

Keep a share of new contacts out of an automation and compare them with the rest.

How do you know a welcome series or a win-back flow adds sales, and that people would not have bought anyway? Hold a few contacts back and compare.

  1. Open the automation and choose the Holdout tab (owners and admins).
  2. Choose how many in 100 new contacts to hold back, from 0 to 50, and click Save. Held contacts get none of the automation’s emails.

Who is held is decided by a fixed formula on the automation and the contact, so a contact always stays in the same group. Changing the share only affects contacts who enter later.

The report

It compares contacts who got the automation with those held back: how many bought, orders, revenue per contact and the number of contacts on each side. Choose how many days after entering an order still counts (7 to 90).

  • Significant means the difference is unlikely to be chance (a two-proportion z-test, p below 0.05).
  • Not enough data or cannot tell yet is normal for a small holdout. Keep it running before you decide.
Was this helpful?

Copy for AI (Markdown)

Related articles

Mailnaptic · Help center