The Global Holdout

A global holdout is the first decision a marketing programme makes, and the one most likely to be reopened later, usually in a bad quarter by somebody who was not in the room when it was made. So it is worth writing the argument down once, properly: why the holdout should exist, what it is for, what was considered instead, what it costs, and how to run it so that it survives.

The numbers throughout come from a working prototype I built of the data science behind bank marketing: who to contact, what to send them, and when, aimed at Monthly Transacting Users. It runs on synthetic data calibrated to published retention and reactivation results, on a default book of 150,000 customers with a 15% holdout. The customers are not real. The method, and the failure modes, are.

1. The decision, in one table

Everything below is the reasoning behind each row.

2. The problem it solves

Response rate is a property of the audience, not of the campaign

A campaign that targets customers who transact every week will report a magnificent response rate. It will report that rate whether or not a single message changed anything, because most of those customers were going to transact anyway. The number is real, it is reproducible, and it is not evidence about the campaign.

This is not a subtle statistical point. It is the default state of marketing reporting, and it has a direction: it flatters exactly the campaigns that spend money on people who needed no persuading. In this prototype, customers who transacted in the last week show an uplift of +0.51pp. Customers who have been quiet for 61 to 90 days show +4.54pp. The first group is where a conventionally-run programme spends most of its budget, because it is where the response rates look best.

Without a control group, every number is a claim about something nobody observed

The quantity a marketing team wants is the difference between what happened and what would have happened otherwise. The second half of that sentence is never observed for anyone. Any method that produces a number is therefore supplying the missing half from somewhere: from last year, from a lookalike customer, from a regression, from an assumption about parallel trends. A randomised control group supplies it from a group of real customers who differ from the treated group in exactly one respect, by construction.

That is the entire argument. Everything else is a question of how much bias the alternatives carry and how much the randomisation costs.

We have a measured example of how wrong the observational answer gets

The send-time question is the worked example, because the prototype answers it both ways on the same audience.

Half of every campaign’s treated audience is randomly assigned to be delivered at the customer’s own estimated peak hour, against a broadcast-time control. The randomised comparison says send-time optimisation is worth +0.89pp on conversion inside the golden window. The observational version of the same question, cutting the same sends by whether they happened to land near the customer’s peak, says 3.19pp.

That is a 3.6x overstatement, and it is not sampling error. Customers whose peak hour sits near broadcast hours are more likely to be on payroll and older, and both of those predict response on their own. The observational estimate is measuring who those customers are, not what the send time did.

Nothing in the modelling detects this. The observational number is stable across re-runs, has a tight interval, and looks exactly like a result. Only the randomised arm reveals it, and only because the randomised arm exists.

And an example of how wrong the reporting metric gets

The same experiment moves opens by +6.63pp, clicks by +1.70pp and transactions by +0.54pp. Attention moved roughly twelve times as much as behaviour did. A programme reporting opens would have declared a decisive win on a change that is real but marginal in the only currency that matters.

The holdout is what makes the business metric readable at all. Without it there is no denominator for “compared to what”, so the funnel metric wins by default because it is the one that can be computed.

3. What the holdout is for, in priority order

1. A programme-level incrementality number that needs no model. MTU rate among contactable customers minus MTU rate in the holdout. A difference of two means over a randomised split. No features, no functional form, no assumptions that a hostile reviewer can attack. On the default book this reads +1.03pp, about 1,500 incremental transacting users a month. This is the number that answers whether the marketing budget is worth its money, and it is the only one that survives an audit.

2. An uncontaminated control arm for the uplift model and its evaluation. The X-learner needs both response surfaces, and the measured band table needs a control group that was genuinely not touched. Per-campaign controls contaminated by an overlapping calendar produce an uplift curve that is flattened toward zero, which is the failure described in section 5.2.

3. A standing regression detector. The contrast is computed the same way every month against a group that never changes definition. When a channel deliverability problem, a consent-record drift or a creative change quietly destroys the programme’s effect, the holdout contrast is the thing that moves. Response rates will not move, because response rates are a property of the audience.

4. A governance instrument. Somebody will ask to target the “best” customers, meaning the most engaged ones. The holdout is what converts that from an argument about taste into an arithmetic question with an answer.

What it is explicitly not for

It is not a per-campaign readout. A single campaign against a 15% global holdout is badly underpowered, and attributing a programme-level contrast to one campaign in the calendar is not something the design supports. If a specific campaign needs its own causal read, randomise within the contactable arm and read that campaign against its own control, keeping the global holdout untouched underneath.

It is not a substitute for a marketing mix model on above-the-line spend. The holdout identifies the effect of addressable, individually-targeted promotional contact. Brand, sponsorship and mass media reach the holdout too, so they are inside the counterfactual rather than measured by it.

It is not powered for small segments. Slicing the holdout by lifecycle, region and product will produce sign-flipping noise that reads as a finding.

4. The science

4.1 The estimand

For customer i write Y_i(1) for the MTU outcome if the customer is contactable by the programme and Y_i(0) for the outcome if they are not. Only one of the two is ever observed. The quantity of interest is the average treatment effect:

ATE = E[ Y_i(1) - Y_i(0) ]

Random assignment of the holdout flag makes assignment independent of the pair of potential outcomes:

( Y_i(1), Y_i(0) )  ⟂  H_i

which is what licenses the estimator being a difference of two sample means, and nothing more:

ATE_hat = mean( Y_i | not held out ) - mean( Y_i | held out )

Unbiasedness here comes from the design, not from the analysis. That is the property no observational method can buy at any price, and it is what makes this number the one to put in front of a CFO.

4.2 What is being identified is intention to treat, deliberately

Not every contactable customer is contacted. The planner contacts about 7.4% of the addressable book, because past that point a contact costs more than the behaviour it changes is worth. So the contrast measures the effect of being in the population the programme is allowed to work on, not the effect of receiving a message.

This is the right estimand for the question being asked. “Is the marketing programme worth its budget” is a question about the programme as run, including its own decisions about whom to leave alone. An estimate of the per-contact effect would flatter the programme by charging it only for the customers it chose to spend money on.

The per-contact effect is recoverable from the same design, by scaling the ITT by the contacted share, under the assumption that uncontacted customers in the contactable arm are unaffected. It is worth computing and worth labelling clearly, because the two numbers differ by more than an order of magnitude and they get quoted interchangeably.

4.3 Why the customer is the unit

Randomising at the send or the campaign level violates the assumption that one customer’s treatment does not affect another customer’s observed outcome, and it violates it through the customer’s own other treatments. A customer held out of campaign A and contacted by campaign B has an observed outcome containing B’s effect while being counted as A’s control. That is not a small leak. It is the dominant term as soon as two campaigns overlap, and campaign calendars overlap by design.

Randomising at the customer level and holding the assignment across all campaigns makes the exclusion complete. There is then exactly one treatment, “the programme”, and it is well defined.

Two forms of interference survive and are worth naming rather than assuming away. Household and device sharing means a held-out customer can see a treated customer’s message, which biases the estimate toward zero. Word of mouth about an offer does the same. Both are small for individually-targeted transactional offers and both make the measured effect conservative, which is the right direction for a number used to justify spend.

4.4 Why the contrast is over customers, not over customer-months

The panel is customer by month. Pooling all customer-months and running a two-sample test on that treats the same customer’s sixteen rows as sixteen independent observations, which they are not. It narrows the interval by roughly the square root of the number of months, and it does it silently.

The programme-level estimator collapses each customer to the share of months in which they transacted, and contrasts the arms over customers. Each customer contributes exactly once, the two arms are independent by construction, and the interval is honest. The monthly series is still worth plotting, as a diagnostic for something having broken rather than as a scorecard.

4.5 Power, and where the standard error comes from

For a binary outcome at rate p, split into a contactable arm of size n1 and a holdout of size n0:

se_pp  = sqrt( p(1-p) * (1/n1 + 1/n0) ) * 100
mde_pp = (1.96 + 0.8416) * se_pp

The term in the middle is the important one. Writing the holdout share as s on a book of n:

1/n1 + 1/n0  =  ( 1/(1-s) + 1/s ) / n

That expression is dominated by the smaller arm. It is minimised at a 50/50 split, and it rises steeply as s gets small, which is the entire content of the sizing argument in section 7.

On the default book of 150,000 at a 15% holdout the monthly detection floor is 0.94pp against a measured effect of +1.03pp. The margin is about ten percent. That is deliberate: it is the margin a real programme of this size actually operates on, and rounding the book up until the answer looked comfortable would have hidden it.

4.6 The cost the design imposes on the models

This is the part that is usually left out of the pitch, and it is a genuine scientific cost rather than an inconvenience.

Under a global holdout, every marketing-contact feature is identically zero for the control arm. Contact counts, open rates, past response after contact: all of them are deterministic functions of treatment assignment. An uplift model handed those features does not learn heterogeneous response. It learns to recognise which arm a row came from, and the two response surfaces stop being comparable. open_rate_3m is worse still, because it is undefined for the entire control arm, which is how the problem was found in the first place, via a binning error inside scikit-learn on an all-NaN column.

So the uplift model cannot use past responsiveness, which is the most natural proxy for how persuadable somebody is. An explicit exclusion list names the six features that are banned, and a test asserts that none of them reaches the model.

The same logic caught a subtler leak. Per-customer send-time profiles were originally built from transaction times and message opens together. Opens only exist for contacted customers, so the confidence attached to a profile became a partial proxy for treatment status, the control arm could never contain a high-confidence row, and the estimator would extrapolate into a region with no counterfactual data. The profiles are now built from transactions only, which are observed symmetrically and are twenty times more numerous.

Per-campaign holdouts would keep all of these features usable. That is a real advantage and it is not enough to save them, because losing past-responsiveness as a predictor is the cheaper of the two problems.

5. The alternatives, and what each one buys

Every option below is a real practice used by real teams, and several of them are better than a global holdout at something. The question is not which is best in the abstract but which one answers “is this programme worth its budget” on an always-on, individually-targeted, overlapping campaign calendar.

5.1 No control group: response rates and pre/post comparison

What it is. Report response and conversion rates per campaign, and compare this period against last period.

Identifies. Nothing causal. The counterfactual is supplied by last year, which differs in seasonality, acquisition mix, competitive pressure, product changes and app releases.

Pros. Free, familiar, requires no platform capability, and forgoes no upside.

Cons. Biased upward by construction, because campaigns select customers who were already likely to convert. The bias is largest for the campaigns that spend the most. It cannot detect a programme that has stopped working, because the metric moves with the audience rather than with the effect.

Verdict. This is the status quo the proposal exists to replace. Keep response rates as operational telemetry for deliverability and creative, and never as a business number.

5.2 Per-campaign randomised control groups

What it is. Inside each campaign, randomly hold back a share of the target audience. The traditional and entirely respectable design.

Identifies. The campaign’s ATE, provided no other campaign touches the held-back customers during the measurement window.

Pros. Per-campaign answers, which is what campaign managers actually want. No permanently unmarketed group, so it is an easier internal sell. Contact features stay usable, so the uplift model keeps its most natural predictors. The cost per campaign is small and visible.

Cons. The proviso does not hold on a real calendar. The five campaign templates here target disjoint recency windows and still run concurrently, so a customer held out of A gets treated by B in the same month, and A’s control arm quietly contains B’s treatment effect. During development this flattened a clean inverted-U uplift curve peaking at +7.9pp into a noisy band around +3pp. The signal was still there. The contaminated controls were hiding it.

The second problem is that per-campaign effects do not add up to a programme effect. There is no clean counterfactual for “all marketing”, because every customer is in some campaign’s treatment arm, and summing per-campaign lifts double-counts customers who received several messages.

Verdict. Not a replacement, but genuinely complementary. Run per-campaign randomisation inside the contactable arm when a specific campaign needs its own read. The global holdout sits underneath, untouched, and answers the programme question that per-campaign controls structurally cannot.

5.3 Matched or propensity-matched observational controls

What it is. Build a comparison group from customers who happened not to be targeted, matched on observable characteristics or on an estimated propensity score.

Identifies. The ATE under selection on observables: the assumption that once you condition on what you can see, treatment is as good as random.

Pros. No forgone upside at all. Works retrospectively on data that already exists, so it produces an answer this quarter rather than next year. Statistically well understood, and defensible when the targeting rule is known and fully recorded.

Cons. The identifying assumption is not testable, and here it is close to certainly false. The reason a customer was not targeted is usually the targeting rule, which was built from the same features the matching uses, so the residual difference between arms is precisely the part the features do not capture. Persuadability is the clearest case: in this simulator, roughly a third of the true persuadability trait is linearly recoverable from ordinary CRM attributes, and that is a simulator built to be recoverable. On real data it will be much less.

And there is a measured example of the size of the error, in section 2: matching on observables overstated the send-time effect by 3.6x on the same audience where the experiment gave the answer.

Verdict. Useful for generating hypotheses and for questions where no experiment is possible. Never as the number that justifies the budget.

5.4 Geo holdout and matched markets, read as difference-in-differences

What it is. Withhold marketing from whole cities or regions and compare against matched markets over time.

Identifies. A market-level effect under parallel trends, plus the assumption that treatment does not cross market boundaries.

Pros. No individual customer is singled out, which sidesteps most fairness and compliance objections. It captures spillovers, word of mouth and above-the-line interactions that a customer-level holdout is blind to. It is the standard design for brand and mass media, where individual assignment is impossible.

Cons. The effective sample size is the number of markets, not the number of customers, so a +1pp individual effect is nowhere near detectable. Parallel trends is an assumption that fails exactly when something interesting happens, such as a competitor launching in one region. Digital banking promotions are delivered to individuals through owned channels, so geography is a weak and leaky proxy for exposure: customers move, commute and are reachable regardless of where the market boundary was drawn.

Verdict. The right tool for a different question. Keep it in mind for above-the-line evaluation, not for this programme.

5.5 Synthetic control

What it is. Construct a weighted combination of untreated units that reproduces the treated unit’s pre-period trajectory, and read the post-period gap.

Identifies. The effect of a single large intervention on a single aggregate unit, under the assumption that the fitted pre-period relationship would have continued.

Pros. No randomisation needed, transparent, and strong when there is a long clean pre-period and a genuine donor pool. Well suited to one-off events: a market launch, a pricing change, a regulatory shock.

Cons. An always-on programme has no pre-period and no untreated donor pool, because everybody is being marketed to. Inference is weak with few units, and the method has no answer for a continuously varying treatment intensity.

Verdict. Not applicable to this problem.

5.6 Switchback and on/off time-based testing

What it is. Turn the programme off for the whole book in alternating periods and compare periods.

Identifies. The effect under the assumption that it does not carry over between periods.

Pros. No customer is permanently excluded, which is often the easiest design to get approved. Simple to operate, and it captures network and spillover effects because everybody switches together.

Cons. Carryover is exactly what this programme has. A customer reactivated in month three keeps transacting in every month after it, which is the same accumulation that makes the duration curve in section 8 rise. Switchbacks are for effects that dissipate within a period, and this one does not. Turning marketing off entirely for whole months is also a larger and far more visible commercial cost than holding out 15% continuously, and it confounds the effect with seasonality.

Verdict. Wrong instrument for an effect that persists.

5.7 Marketing mix modelling

What it is. Regress an aggregate outcome series on spend by channel, with adstock and saturation transforms.

Identifies. Channel contributions under a functional-form assumption, and only to the extent that spend varies independently of everything else.

Pros. Covers every channel including the ones no experiment can reach. Forgoes no upside. It is the standard tool for allocating budget across channels, and it answers a question the holdout does not.

Cons. A handful of monthly observations with heavily collinear spend series cannot resolve a +1pp effect on an individual-level outcome. The estimates are highly sensitive to the transform choices, and the model will always return a positive contribution for a channel that was on during a good quarter. Modern practice is to calibrate a mix model against experimental results precisely because it cannot identify them on its own.

Verdict. Complementary and downstream. The holdout is the experiment that calibrates the mix model, not the other way round.

5.8 Ghost ads and PSA controls

What it is. In a paid-media platform, record which users would have been served the ad and serve them something else, or nothing, then compare.

Identifies. The effect of ad exposure, with the selection effect of the platform’s own targeting removed, which is the thing intention-to-treat comparisons on ad platforms usually get wrong.

Pros. Nearly zero opportunity cost, because the control group receives a different ad rather than no marketing. It removes the “we cannot withhold” objection entirely. It is the state of the art for paid media measurement.

Cons. It only exists where a platform implements it. This programme runs on owned channels, push, in-app, email and WhatsApp, where there is no auction and no ghost inventory. It also measures a single channel’s ad exposure rather than the programme.

Verdict. Adopt it for paid media if the platforms support it. It does not touch the question this holdout answers.

5.9 The global holdout

What it is. One persistent randomised group, held out of all promotional contact, for as long as it takes for the arms to diverge.

Pros. Unbiased by construction, with no assumption a reviewer can attack. It is the only design here that produces a programme number rather than a channel or campaign number. It is immune to calendar overlap, which is the failure mode that kills per-campaign controls in practice. It provides a clean control arm for the uplift model. Its cost is proportional to the value it measures, which is a property no other measurement spend has.

Cons. It forgoes real upside from day one and returns nothing for two quarters, which is a hard sell against a quarterly target. It costs the uplift model its contact-history features, per section 4.6. It requires a platform gate that cannot be overridden, which is engineering work and, more importantly, political work. It does not measure above-the-line, and it cannot answer per-campaign questions. It is fragile in one specific way: a single contamination event costs the whole accumulated window.

5.10 Side by side

The honest summary of that table is that the global holdout is not the cheapest option, not the fastest, and not the most flexible. It is the only one whose validity does not depend on an assumption that fails silently.

6. What it costs, and what it buys

The objection, and the arithmetic that answers it

The objection is always some version of “we are giving up 15% of our marketing revenue”. That is not what holding out 15% of customers does. It forgoes 15% of the incremental effect of marketing, and that effect is about +1pp.

Two orders of magnitude separate the intuition from the number. The gap exists because marketing’s effect on MTU is small relative to MTU, which is itself one of the findings the holdout is there to establish.

The cost is self-scaling, which is unusual and worth saying out loud

The holdout’s cost is denominated in the same units as the thing it measures. It is always 15% of whatever incremental value the programme creates, so it can never cost more than the programme is worth. If the programme turns out to be worth nothing, the holdout costs nothing, and it will have been the thing that found that out.

No other measurement spend behaves this way. An analytics platform, a research agency and a mix modelling engagement all cost the same whether or not there is anything to find.

The costs that are not in the table

Two quarters of returning nothing. The first credible readout is a year away. Somebody has to carry that, and it is the single hardest part of the sell.

A model that is missing its most natural features. Section 4.6. Real, permanent, and the cheaper of the two problems.

Engineering and governance. A pre-send gate that a campaign manager cannot override, assignment history retained per customer with effective dates, send-level control assignment retained for over a year, and a named person with the authority to refuse a request to message the holdout. The engineering is a week. The governance is the actual work.

A fairness and compliance question that has to be asked, not assumed. Withholding promotional offers from a randomly selected group needs a compliance answer rather than a marketing one. It helps that the group is randomised, that the exclusion is promotional only, that entitlements are unaffected, and that membership rotates.

What it buys, in the same currency

The direct benefit is a defensible programme number. The larger benefit is the decisions it changes, and there are measured examples of both.

The send-time answer would have been overstated by 3.6x without the randomised arm. Any budget decision taken on the observational number, at any scale, would have been wrong by that factor.

Targeting by propensity rather than by uplift projects 2 incremental MTU against 347 for the same list size. That comparison is a projection rather than a result, and it is only computable at all because there is a control arm to fit an uplift model against.

Charging the incentive against everyone who redeems rather than against incremental conversions, and pricing free-to-send channels above zero, moved the recommended contact rate from 99.3% of the book to 7.4%. That is the same class of correction, and the holdout is what makes the corrected number checkable rather than merely more conservative.

Against Rp8.6m a month, the bar is low. The holdout pays for itself if it changes one budget decision a year by more than about a hundred million rupiah, and the examples above are larger than that.

7. Sizing: what the share actually buys

Size is negotiable, it trades against upside, and it is set by arithmetic rather than convention.

From section 4.5, the detection floor scales with sqrt( (1/(1-s) + 1/s) / n ). Holding the book at 150,000 and varying the share:

Read the last two columns against the third. Going from 15% to 25% improves the detection floor by about 17% and increases the cost by 67%. Getting the same 17% improvement by growing the book instead needs about 220,000 customers, and books grow for free if acquisition is working.

That is the quantitative version of the rule:

The holdout is not the constraint. The book is.

The contactable arm is already the larger of the two, so the variance is dominated by the holdout arm and adding to the holdout hits diminishing returns quickly. If a programme genuinely needs to read a +1pp effect monthly, the answer is a bigger addressable base or a longer measurement window, not a bigger holdout.

The book requirement is the harder constraint, and it scales quadratically:

Halving the true effect quadruples the book needed to see it. That is the single most important line in the sizing discussion, because the constraint tightens far faster than intuition suggests and no amount of modelling relaxes it.

Practical recommendation. On a book in the millions, 5% is well powered and costs under a tenth of a percent of MTU. On a book in the low hundreds of thousands, 15% is roughly the minimum that supports a monthly readout. Below about 120,000 customers, no share supports a monthly readout of a realistic effect, and the honest design is a smaller holdout read quarterly or annually rather than a larger one read monthly and quoted as though it meant something.

8. Duration: why time cannot be bought with size

The second objection is “how long do these customers get no promotions”. The answer is longer than anyone expects, and for a reason that is not about precision.

Here is the same readout computed at increasing durations on the prototype:

Read which column moves. The confidence interval barely improves, from ±0.60 to ±0.42 across thirteen months. Precision is not what the programme is waiting for.

What moves is the effect itself, from roughly zero to +1.03pp. The two arms have to drift apart before there is anything to measure, because a customer reactivated in month three keeps transacting in every month after it and the divergence compounds.

The consequence is the part that matters. A readout taken at six months is not merely imprecise, it is biased toward zero. It reads +0.03pp, and a reasonable person concludes marketing does not work. The programme would have paid the entire cost of the holdout and bought a false negative.

This failure cannot be repaired by making the holdout bigger. Size fixes precision. Only time fixes accumulation.

Plan for 12 to 18 months to a confident answer. The prototype first clears zero at 11 months, but a first crossing is partly luck, and the margin is not comfortable until about 13.

One pause costs the whole clock. Suspending the holdout for a launch contaminates the accumulated divergence, and the window restarts from zero. Shrinking the holdout permanently is always better than pausing it once. This is the single most important operational fact in this document, and it is the one that has to be understood by the people who will ask for the exception.

9. How to execute it properly

9.1 Assignment

Randomise at the customer level, independent of everything else.

Use a deterministic hash of a stable customer identifier plus a recorded salt, rather than a stored random draw. Deterministic assignment is reproducible from the identifier alone, survives a warehouse rebuild, and cannot silently drift when a record is reprocessed. Record the salt, the share, the effective date and the code version that produced them.

Assignment happens once for the whole book, not per campaign and not per audience. New customers acquired after the assignment date are hashed on arrival with the same salt, which keeps the share stable as the book grows and keeps their assignment independent of anything about them.

Check the realised share against the configured one with a tolerance that scales with the book. The prototype checks it at four sigma of a binomial, which is a band that stays meaningful at 15,000 customers and at a million.

9.2 Scope of the exclusion

Promotional messages only.

Servicing, transactional, security, regulatory and entitlement communications go to everyone. So does anything the customer has asked for. Write this down before launch, because the first exception request will be a servicing message that someone has classified as a campaign, and that request should be approved.

The exclusions that must hold are the ones with an offer or a nudge attached.

9.3 Enforcement: two gates, in two systems

The holdout is the one invariant that cannot be repaired after the fact. A leak does not raise an error, does not change the shape of any output, and quietly destroys the ability to measure anything for as long as those customers stay on the book.

So it is checked at every point where a list of customers could reach a delivery system, rather than assumed from the filter upstream.

Gate one, in the pipeline. A no-holdout-contact assertion runs wherever a customer list is produced, and the run fails rather than writing a contaminated target list.

Gate two, in the delivery platform. A pre-send exclusion that a campaign manager cannot override from the UI. This is the one that matters, because gate one guards the file this repo produces and not the audience somebody uploads next to it.

Two gates in two systems is the right number for an invariant with no recovery path.

And an audit, because a gate that is never tested is a gate nobody knows is broken. Reconcile actual sends against the holdout roster daily, count the leaks, and alert on any non-zero count. Report the leak count as a standing metric on the dashboard next to the incrementality number, so that the number and its validity are read together.

9.4 Validate the randomisation before launch

Run the balance check before the first campaign goes out, and after every re-randomisation. Compare the arms on tenure, lifecycle mix, product holdings, digital ratio, acquisition channel and pre-period MTU rate.

Then run an A/A period: measure the contrast for a month while both arms are treated identically. It should read zero. If it does not, the assignment or the measurement is wrong, and it is far cheaper to find that out before the clock starts than eleven months into it.

9.5 Governance

Name the person who can refuse a request to message the holdout, before the first request arrives. This is a real appointment, not a line in a document, and it should sit with whoever owns the measurement rather than with whoever owns the quarterly number.

Write down the rule that a pause restarts the clock, and get it agreed while nobody is under pressure. The request will come during a bad quarter, framed as a one-off, and the answer has to already exist.

9.6 What to record, and keep

The assignment history is the asset, and it is the thing campaign platforms routinely throw away.

Retain, per customer, the arm and the effective dates of every assignment round. Retain, per send, the treatment or control flag and any secondary arm such as the send-time assignment, for at least the full accumulation window plus a year. Many platforms purge send-level detail after 90 days, which is why this needs to be established on day one rather than discovered in month three.

Without historical control assignment the uplift model cannot be fit at all, and the clock starts from whenever the recording starts.

9.7 The readout

Monthly, three numbers, in this order.

1. Programme incrementality. The customer-level pooled contrast from section 4.4, with an interval.
2. Cost per incremental MTU. Campaign spend divided by incremental MTU, compared against the margin per transacting month.
3. The MTU bridge. New, reactivated, retained, churned, as the diagnostic rather than the scorecard.

Everything else, opens, clicks, response rates, is operational telemetry and belongs in the campaign platform.

Expect null months, and expect them in streaks. The measured effect sits just above the monthly detection floor, so a working programme will produce months that fail to clear zero. The correct response is to read the pooled customer-level number, not to widen the claim and not to go looking for a reason the last month was different. Agreeing this in advance is what stops a null month from becoming an argument to drop the holdout.

9.8 Refreshing membership, and the tension nobody resolves cleanly

“Permanent” is the wrong word and should be retired. The group is permanent. The membership is not. Re-randomising the whole book periodically means nobody is permanently unmarketed: across five years a given customer spends an expected three months in the holdout at a 5% share, or about nine at 15%.

There is a genuine tension here that is worth stating rather than smoothing over. Section 8 says the arms need 12 to 18 months to diverge. Re-randomising resets that divergence, because the new holdout is full of customers who were being marketed to last year. A refresh cadence shorter than the accumulation window means the long-window readout is never available.

The resolution:

• Freeze membership for the whole accumulation window. No refresh before the first credible readout has landed.
• Refresh at a quarter boundary, and no sooner than 12 months, with 18 to 24 preferred if the effect is small enough that the first crossing was marginal.
• Refresh by re-randomising the whole book. Never by swapping in customers who look similar, which destroys the randomisation and replaces it with matching, which is section 5.3 wearing a holdout’s clothes.
• Keep the previous assignment. The earlier window stays analysable as its own cohort, the new window starts its own clock, and every readout is computed within an assignment round rather than pooled across rounds.
• Record the date. Every readout is labelled with the assignment round it belongs to.

If the fairness concern about long tenure is binding, the lever to pull is the share, not the cadence. A 5% holdout on a large book gives each customer a one-in-twenty chance of a quiet year, costs 0.07% of MTU, and preserves the window.

9.9 Model hygiene

Maintain the banned-feature list, and assert it rather than remembering it. Any feature that is a function of contact history is a function of treatment assignment under this design, and it must not enter the uplift model. Name them in one exclusion list, and let a test fail if one reaches a model frame.

The rule to apply when adding any new feature: if the control arm cannot produce this value, it encodes the arm.

9.10 The first year

10. Failure modes, and what catches each one

11. What still has to be decided on the real book

These are the questions this analysis cannot answer and the marketing team can.

1. How many active customers are there? This sets the share directly, through section 7, and it is the first input to everything else.
2. Does an existing control or holdout practice exist anywhere in the programme today? If one does, it may already carry usable history.
3. Who has the authority to say no when somebody asks to message the holdout for a launch?
4. Do we retain send-level history with control assignment, and for how long? This decides whether the uplift work starts now or in two quarters, and it is the single biggest unknown in the plan.
5. How many campaigns run concurrently, and do audiences overlap? Overlap is the reason a global holdout beats per-campaign controls, and the degree of overlap is the strength of that argument on this book.
6. What is the real contact frequency a customer experiences across all campaigns, rather than per campaign?
7. Are there compliance constraints on withholding promotional offers from a randomly selected group? This goes to compliance, not to the room.

12. What this document is not claiming

The +1.03pp programme effect is measured on simulated data, calibrated to published retention and reactivation results. It is the right order of magnitude and it is not a forecast of what this book will show.

The duration curve is a property of this simulated world’s reactivation dynamics. The shape is general, because it follows from reactivation persisting rather than from anything specific to the simulator, and the specific month at which the effect clears zero is not.

The targeting comparison of 347 against 2 incremental MTU is arithmetic on the model’s own predicted uplift, not evidence. It establishes the shape of the argument, that ranking by propensity spends money on customers with no headroom, and it cannot establish the size of the prize. Settling that needs the live two-arm test in section 9.10, read against the holdout.

The cost figures scale linearly with the margin per transacting month, which is currently an assumption and needs a finance owner.