When You Cannot Run an A/B Test, How Do You Measure an Event's Causal Effect?

One line: Many live-ops actions in mobile games have to ship to the whole server — new content and new modes cannot be shown to half the audience — so A/B testing is simply unavailable. Propensity scoring pairs "players who took part" with "similar players who did not," which lets you infer a causal effect from observational data. It fixes self-selection such as "high-ARPU players were always more likely to join." It does not fix the causes you never measured.

When to Use It

All three conditions have to hold:

1. The action cannot be split. The test is observable: once this ships, can every online player see and use it? New modes, new chapters and new systems qualify. Popup copy, bundle pricing and push timing do not — run a real A/B test on those.
2. You already have historical data that separates participants from non-participants. On sample size, the participating payer group should be in the hundreds at minimum; below that the propensity model itself is unstable.
3. Your question is causal — "how much incremental revenue did this event produce" — not descriptive. "How much did participants spend" is description and does not need this method.

State the limits too: if the action has no control group at all (a forced update for everyone, a compliance change), propensity scoring cannot help. That calls for interrupted time series or synthetic control, which this card does not cover.

Why "Participants Had Higher ARPU" Proves Nothing

This is the most common sentence in a live-ops retrospective, and it is the target this method exists to hit.

The comparison usually looks like this: ARPU rose among players who joined the event; ARPU among non-participants barely moved. The conclusion writes itself: "the event lifted ARPU."

The problem is that these two groups were never the same people. Players who open the event, finish the prerequisite tasks, and log in during a limited window are already the more active, more willing-to-pay segment. Without the event, their ARPU would still have been higher than the other group's.

The analyst presenting this at CEDEC2021 put the problem bluntly: aggregate the observational data and you will generally see ARPU positively correlated with event participation — but from this data alone you cannot tell whether the event raised ARPU, or whether players who already had high ARPU were the ones who joined.

Those two explanations imply opposite actions. The first says "run more events." The second says "the event merely filtered out people who would have paid anyway; running more events just burns labor." While you cannot tell them apart, any decision to double down based on that retrospective is a bet.

The Method in Four Steps

Step 1: Describe the person — choose covariates

Covariates are the fields that describe what this player was like before. The talk treats this section as the crux, because the credibility of the whole method rests on whether the covariates are right.

One criterion: the field must be fixed before the event started, and must influence both participation and payment. The usual families:

• Historical payment: lifetime top-up, last-30-day top-up, first-purchase date, historical ARPPU
• Activity: logins in the last 7 and 30 days, average session length, days since last login
• Progress: level, main-story progress, whether they hold a battle pass
• Account age: days since registration

Three prohibitions, each of which corrupts the result:

1. Never use in-event data as a covariate. Logins during the event are themselves an outcome of the event; including them puts the effect on the cause side.
2. Never use labels the event itself generated, such as "claimed the event reward."
3. Never include the outcome variable — do not use "revenue during the event" to predict "participated."

Step 2: Estimate the propensity score

The propensity score is a player's probability of joining this event, estimated with a binary model (logistic regression is the default in the talk) over the Step 1 covariates.

It is not a model of who will pay. It produces a participation propensity per player. After estimation, the propensity distributions of participants and non-participants must overlap — this gets its own entry under failure modes, because it is where things most often break.

Step 3: Adjust using the score

The talk compares three approaches — correlation analysis, regression analysis, propensity score analysis — and notes that only the third is designed specifically to remove selection bias. Two ways to apply it:

• Matching: for each participant, find the non-participant with the closest propensity score and form a pair, then compare outcomes on the matched sample. Intuitive; costs sample.
• Weighting: skip pairing and weight each player (a common scheme weights the non-participant group by the inverse of the propensity score) to align the two covariate distributions.

Choose by sample size. Thousands of participants or more, matching's sample loss is tolerable. Only hundreds, prefer weighting.

Step 4: Report an interval, not a point

The goal of the talk is a quantitative estimate of an action's effect. One thing the talk does not spell out but engineering cannot skip: a propensity-score estimate is an estimate and must carry a confidence interval. Reporting "this event produced 1.2M in incremental revenue" with no interval treats an estimate as a fact.

Corroboration from a different corner

The same problem — no experiment available, causality must come from observational data — gets the same family of answers in advertising. A study using 500 days of campaign data from a US mobile game developer identified causality from a single global ad shutoff, with fixed-effects panel regressions providing the dollar-level estimate. Its identification strategy differs from propensity scoring (it leans on one exogenous shutdown), but it answers the same question and makes the same point: when experiments are unavailable, causal identification needs an exogenous shock in the data or a genuinely comparable pairing — not a straight comparison of two aggregated groups.

Common Failure Modes

1. No overlap between the two propensity distributions. Symptom: a cluster of participants scores above 0.9 while no non-participant comes close, so matching pairs whoever is available — you end up comparing fundamentally different people. How to spot it: plot both distributions and look at the overlap region. How to avoid it: drop the players outside the overlap and report honestly that "this conclusion applies to the middle segment only."
2. Outcome variables or in-event behavior leaking into the covariates (see the three prohibitions). The most insidious failure, because the model looks better for it.
3. Running it on too little data. With fewer than a hundred participating payers, the score itself is poorly estimated and weighting inflates variance further. The correct move is to admit the question cannot be answered, not to reach for a fancier model.
4. Treating a good propensity model as evidence of an effect. A well-fitting propensity model only says participation is explainable by these features. It says nothing about whether the event worked.
5. Ignoring unobserved confounders. Propensity scoring only balances the features you measured. If participation is largely driven by something you never instrumented — social ties between players, an off-platform discussion — this method cannot give an unbiased answer. That is its principled ceiling, not an execution problem.

What You Can Do Tomorrow

1. Take the most recent all-server event and run all three approaches — correlation, regression, propensity score — and put the three numbers side by side in one table. If they diverge widely, that table is the most valuable page of the retrospective: it quantifies how large the self-selection bias was.
2. Write down the covariate list and ask of each field: was it fixed before the event started? Any "no" leaves the model immediately.
3. Add one number-free causal sentence to the next event retrospective, shaped like "because of X, Y happened, so the data shows Z." If you cannot write it, the analysis is still sitting at the correlation level.

Appendix

Evidence and sources

- [CEDEC2021 — "How much revenue did that initiative contribute?" Applying statistical causal inference via propensity score analysis](https://cedec.cesa.or.jp/2021/session/detail/s60648834255dc.html) — practitioner conference talk (WFS, Japan, 2021-08-26). Source for this card's scenario criteria, covariate approach and three-method comparison. Note: this page is the session abstract, carrying the substantive "knowledge gained" and "session content" sections; the concrete figures and result tables live in the talk materials and the CEDiL archive, which this run did not obtain.
- [CEDiL archive page for the same talk](https://cedil.cesa.or.jp/cedil_sessions/view/2397) — archived record (retrieval-layer hit; body not fetched).
- [Advertising Spillovers in Mobile Apps: Evidence from Ad Shutoffs and Store Rankings](https://arxiv.org/abs/2504.16151) — independent research (academic paper), v2 dated 2026-07-10. Used as corroboration that causal identification without experiments follows the same logic in an adjacent domain.

Evidence and sources (assessment)

Evidence tier A, confidence medium-high.

• The source is a first-party account: a WFS analyst presenting the company's own monetization initiative at CEDEC. The stance is "here is a usable method," not a neutral evaluation. Per filtering.md, technical facts are kept and the stance is labeled.
• It does supply method detail and explicit scope — which covariates, how the three approaches compare, why A/B testing is unavailable in this domain — rather than slogans.
• Downgrade note: this run retrieved only the session abstract, not the talk materials with the actual figures and result tables. The four steps here are therefore a methodological skeleton, not a replication with numbers. Under filtering.md's downgrade rule this would normally force a tier drop; this card keeps tier A because the framework comes from a first-party practitioner statement and an independent study corroborates its direction — but the "key parameters must not be empty" rule is only partly satisfied. The reference values (minimum sample size, overlap check) are supplied by this card from first principles, and are flagged as such below.
• No undisclosed conflict of interest (a conference talk, not sponsored placement).

Scope and limits

• Mobile, live service, all-server actions. Premium and single-player products do not face this scenario.
• One player's participation must not materially change another's outcome (no strong social spillover). If the event spreads through friends — say, you can only join by bringing a guildmate — the core assumption breaks: one player's participation changes another's probability of participating. That setting needs interference-aware methods and is out of scope here.
• Covariates must be fixed pre-event. Fields you never instrumented are permanently excluded from this method, which caps how good the conclusion can be.
• The specific reference values here (participating payers in the hundreds, the overlap check) are engineering rules of thumb, not source data. The source gives no numbers. Recalibrate against your own scale.

What would falsify this

1. A propensity-score estimate pointing the opposite way from a randomized experiment on the same data. If a product ran a genuine A/B (say, shipping a new mode to only some servers) and propensity scoring on the same action estimated the opposite direction, the covariate set has a systematic defect and the method is unusable in that setting.
2. Large-scale non-overlap. If the two propensity distributions barely overlap and dropping the non-overlapping players leaves almost no participants, the premise — that you can construct a comparable pair of groups — fails, and a different identification strategy is required.
3. High sensitivity to the covariate set. If adding or removing one or two covariates flips the sign of the effect estimate, the conclusion is unstable and must not drive decisions.

The weakest link in the current inference chain: this card did not obtain the talk's actual data or result comparison, so the step "propensity scoring genuinely corrected the bias here, and the corrected conclusion differed from the regression" rests on methodological principle and an analogous external study, with no direct evidence.

What data would settle it: the CEDEC talk materials (marked in the program as "materials to be published") or the full CEDiL record; or any public retrospective that reports all three approaches side by side.

Unverified

• The actual magnitude of correction from propensity scoring in small-ticket, high-frequency mobile monetization (needs multi-product, multi-event comparison data).
• Whether stratified modeling is needed when an event is open to both new and veteran players but they participate through different mechanics.
• The usability of interrupted time series and synthetic control for forced all-server updates — not covered here; it is the remaining gap in C4.
• The talk's stated finding about how accuracy changes when the propensity model uses a different learner. Not obtained. To be added.

Terminology follows references/glossary.md. This card is one file; there is no separate .en.md.