Your Shopify email marketing dashboard says a flow generated 30% of store revenue. That may be useful attribution, but it does not answer the harder question: how much revenue would disappear if the flow stopped?
Attribution gives credit when a purchase occurs inside a configured window after a message interaction. Incrementality compares customers who were eligible for the same flow but did and did not receive its marketing message. These are different measurements, so a founder should not use the larger number by default.
Start with the measurement gap
Klaviyo's current default for new accounts gives email opens and clicks a five-day attribution window. A shopper can interact with a message, return later through another route, and still have the purchase credited to email while the window remains open. Changing attribution settings can also recalculate historical reporting.
That credit is still useful. It helps compare messages, flows, and channels under a consistent rule. It just cannot reveal what the same shopper would have done without the message.
A holdout creates that missing comparison. The treatment group receives the flow message. The control group remains eligible but receives no experimental message. Both groups retain the same storefront, products, prices, checkout, and observation window.
Choose the smallest valid holdout
Klaviyo's native global holdout is built for programme-level measurement. It requires at least 400,000 profiles, applies across campaign and flow channels, allows one active holdout, and is recommended to run for about three months. That is too broad for many founders and too blunt for one-flow diagnosis.
For a mature automated flow, use a reversible flow branch instead:
- Freeze the eligibility rule and create an experiment ID, such as post_purchase_2026q3.
- Add a random-sample split before the marketing message.
- Send the current message on treatment. Send no experimental message on control.
- Add a profile-property action on each branch to record experiment ID, assignment, and assignment date.
- Observe both groups for the same fixed period, long enough to cover the normal purchase and refund lag.
Do not remove order confirmations, safety notices, consent messages, or service updates. Do not run the test across Black Friday, a price change, a stock clearance, or another event that affects the groups unevenly.
Measure the outcome the business keeps
Use total contribution per eligible profile, not email-attributed revenue per recipient.
Incremental contribution per eligible profile = treatment contribution per eligible profile minus control contribution per eligible profile.
Multiply that difference by the eligible treatment population to estimate the test-period contribution added by the message. Calculate revenue lift as a secondary view. Contribution should subtract product cost, discounts, payment fees, fulfilment, shipping subsidy, and refunds that are known by the decision date.
| Decision | Evidence |
|---|---|
| Scale | Positive contribution lift with acceptable unsubscribe, complaint, refund, and support guardrails |
| Extend | Direction is useful, but sample size or refund maturity is insufficient |
| Stop | Contribution is negative, a guardrail breaks, or contamination makes comparison unreliable |
| Repair and rerun | Assignment tags, eligibility, order sync, or exposure data are incomplete |
Klaviyo's built-in significance guidance for flow message variations uses at least 500 recipients per variation and 90% win probability. Treat that as platform context, not automatic validation of a custom no-message branch. Have an analyst size the test from your baseline conversion rate, minimum worthwhile lift, group ratio, and risk tolerance.
Audit contamination before reading lift
The control can still receive campaigns, SMS, paid retargeting, support outreach, or another flow. Record those exposures where possible. If the control receives more or less pressure than treatment, the measured gap is no longer the effect of one email alone.
Also check assignment counts, duplicate or merged profiles, late Shopify order sync, and branch filters each week. Klaviyo warns that deleting a split while contacts are waiting can make them exit the flow. To roll back, stop new control assignment, let queued profiles clear, preserve experiment properties, and return future profiles to the established path.
Setup effort is medium. The data requirement is higher than a dashboard read, but the integration is reversible when branch tags and queues are managed. Maintenance needs one owner and a fixed review date. Exit cost is low if history is preserved, and higher if the team deletes live branches or depends on vendor-only fields it cannot export.
Klaviyo currently offers a free tier up to 250 profiles and 500 monthly emails; public email pricing starts at USD 60 per month and scales with active profiles. Exact UK account pricing and feature access should be checked in Billing. The global holdout's practical gate is the documented 400,000-profile minimum, not an endorsement or a reason to upgrade.
Run one useful test
Choose one stable, high-volume flow whose message is commercially meaningful but not essential, such as a mature post-purchase cross-sell. Document eligibility, branch ratio, outcome window, minimum worthwhile contribution lift, guardrails, contamination rules, owner, and rollback before touching the live flow.
If the data volume cannot support that design, do not manufacture certainty. Keep attributed reporting for operational diagnosis, improve the event and cost data, and revisit the holdout when the result can change a real decision.
Inficial can help connect retention logic, Shopify order data, and a decision-ready growth experiment.
Sources
- Understanding Klaviyo message attribution, Klaviyo Help Center, accessed August 26, 2026
- Getting started with global holdout groups, Klaviyo Help Center, accessed August 26, 2026
- How to A/B test flow branches, Klaviyo Help Center, accessed August 26, 2026
- How to add an update profile property action to a flow, Klaviyo Help Center, accessed August 26, 2026
- Multi-Branch Splits, Klaviyo Help Center, accessed August 26, 2026
- Understanding statistical significance in flows, Klaviyo Help Center, accessed August 26, 2026
- Klaviyo pricing, accessed August 26, 2026
- Klaviyo email marketing, accessed August 26, 2026
