Is "this change lifted revenue 50%" actually true? Four checks

In one line: Revenue rising after a change and revenue rising because of that change are two different claims. The first is just two numbers. The second needs a control group, a statistic that isn't hijacked by whales, a window covering a full cycle, and a set of guardrail metrics confirming you didn't break something else. Miss any of the four and what you have is a hypothesis, not a conclusion.

Terminology follows references/glossary.md.

When to use this

Triggers (observable, decidable):

• Someone wants a full rollout based on data whose shape is two consecutive time periods (the month before vs the month after) instead of a treatment group against a concurrent control
• Someone cites "great first-week numbers" to justify more resources, where that window is shorter than one complete event cycle
• An experiment report gives only a mean change on the primary metric, with no distribution and no guardrails

When you don't need it: the experiment already ran with a control group and pre-registered metrics and sample size — you have a conclusion; what's left is review. Purely qualitative work (interviews, usability testing) doesn't fit these criteria either.

Check 1: Is there a control group? If not, it's a hypothesis

A counter-example (covered in detail in today's companion card): the casual game Mergic replaced one 30-day battle pass per month with two 14-day passes, and IAP revenue rose about 50% within two months. Strong numbers — but it was a pre/post comparison, not a controlled experiment. The article says: "in the fall, the project saw an unexpected surge in in-app purchases." In a pre/post comparison, everything that changed alongside gets credited to the change — seasonality, concurrent updates, ad spend, competitor moves, platform featuring, general market swings. You can't separate them; they overlap perfectly in time. This isn't a rigour issue; the information simply isn't there.

Parameters:

• The control doesn't need to be large — 5–10% of users usually shows direction
• But it must run for at least two complete event cycles (see Check 3)
• The control must be able to participate normally. "We didn't ship them the new build" and "control players saw the old event" are different things — only the second is a meaningful control. Many teams' "control" is just users on the old build, possibly without even the same event entry point
• Where payment or pricing is involved, watch for control users learning about the treatment through community channels or guide sites — that contaminates the control

Strength of conclusion: with no control group, the strongest claim is "this change is worth running a real experiment on," not "this change works."

Check 2: Don't look at the mean — game metrics are long-tailed

The GDC talk "Evaluating Game Experiments: A More Robust Approach for Freemium Games" states in its overview: to run effective experiments you must understand the distributions of key metrics across players, and conventional evaluation methodology has weaknesses.

Why this is especially lethal in games: metrics like ARPU, LTV and purchase amounts are heavily skewed — a small number of high spenders produce most of the revenue. The misread:

The treatment mean is 3% higher, when the real cause is that the control group happened to be missing one whale, or the treatment group happened to gain one.

Means are not robust under long tails: one or two extreme values can flip the conclusion, with no warning.

Parameters:

• Report quantiles — at minimum P50 / P90 / P99, not just the mean
• Report both denominators: ARPU (all users) and ARPPU (paying users). These two can point in opposite directions — ARPPU up while ARPU is flat means existing payers pay more, not that new payers arrived
• For whale-heavy products, consider comparing layer by layer after segmenting on spend, or replacing the normal-theory t-test with bootstrap or permutation tests

Test: if the conclusion flips once you drop the top 1% by revenue, it is about that 1%, not your experiment.

Check 3: Cover a full cycle, and assume a novelty effect

Novelty effect: players respond more strongly to something new in the short term than the long term. New interfaces, new events, new reward structures all tend to look good in the first days — because novelty itself attracts, an increment unrelated to design quality. The opposite, primacy effect: existing users resist change briefly, so numbers dip then recover — and you may kill a good change by mistake.

The two run in opposite directions but share one property: both decay over time. Short-window data therefore proves neither effectiveness nor failure.

Parameters:

• The experiment must cover at least one complete event cycle; for seasons or battle passes, at least two
• Before the experiment starts, fix the minimum detectable effect (MDE) — say, D7 retention +1.5 percentage points — then back-calculate the sample size using that MDE, baseline variance, significance level (typically 0.05) and power (typically 80%)
• The order is not negotiable: fix MDE and sample size first, then look at data. Doing it the other way (use whatever data you have) essentially guarantees a "significant" result
• A cheap process check: run an A/A test before shipping (randomise users into two groups with identical builds). If it reports a significant difference, the bug is in assignment or instrumentation — the best-value debugging you will ever do

Check 4: Guardrail metrics — did the primary metric rise because something else broke?

A guardrail metric doesn't judge whether the change is good — only whether it caused harm elsewhere. The primary metric (the OEC, Overall Evaluation Criterion in Kohavi's framework) covers that; guardrails don't.

For an event redesign, the minimum set:

Test: when a guardrail degrades, a rise in the primary metric is not a net gain.

Back to Mergic: that article reports revenue and conversion but not retention. So even granting its pre/post framing, "this was a net gain" still lacks evidence. Not pedantry: revenue +50% with D7 retention -10% is entirely possible, and it describes a losing change.

Common failure modes

1. Sample Ratio Mismatch (SRM). The split should be 50/50 and comes out 48/52. Usually means dropped tracking events, a bug in assignment, or users being assigned twice. How to catch it: chi-square the observed split against the intended one; if the p-value is implausibly small, stop analysing and go find the bug. This is the commonest and most often skipped way an entire experiment is void.

1. Peeking. Checking daily and stopping at the first significant result. False positives inflate badly — you're picking the winner across many looks. How to avoid it: fix sample size and stopping rules in advance, and during the run look only at guardrails (to stop losses early), never the primary metric.

1. Concurrent experiments interfering. Two experiments both touch the same screen or flow, and effects become unattributable. How to avoid it: build experiment layers and mutual exclusion; run experiments in the same slot sequentially.

1. Twyman's law: any figure that looks interesting or different is usually wrong. When you see +50%, your first move should be hunting for a bug (did the definition change? double-counted events? two events stacked? did you land in peak season?), not writing the retrospective deck. Empirically, implausibly large effects are usually errors.

1. Writing two simultaneous events as one cause. This is the inevitable product of pre/post comparisons, and the most common error in retrospective reports. Mergic is the template: a shorter event and peak autumn season happened together and got written up as "shorter events produced +50%."

1. Definitions changing mid-experiment. Tracking version updates, metric definition changes, or settlement rule changes all make before-and-after data incomparable. How to catch it: pull the log of metric-definition changes covering the experiment window and check. Almost nobody does this, and it explains a lot of "inexplicable effects."

Three things you can do tomorrow

1. Pull the most recent "this change worked" conclusion and check one thing: was there a control group? If the shape is "month before vs month after," write "no control group, attribution unsupported" next to the conclusion. No tooling required — just look at where the data came from.

1. Give your primary metric three guardrails and put them in the experiment report template. Minimum: D7 retention, refund rate, crash rate. Changing the template beats changing the process — do that and every future experiment carries all three.

1. Before your next experiment, fix MDE and sample size in one sentence, then lock the primary metric until the run completes. "Detecting D7 retention +1.5pp, N per arm at 80% power, 14-day run." That sentence costs five minutes and decides whether the experiment has a conclusion at all.

Appendix

Source list

• Evaluating Game Experiments: A More Robust Approach for Freemium Games — GDC talk, GDC Vault index layer (title and Overview readable; full text is members-only). Source of the two points "you must understand the distributions of key metrics" and "conventional methodology has weaknesses." The talk body was not read
• Don't trust the HiPPOs: A/B Testing Online Games — GDC talk, GDC Vault index layer. The overview mentions bringing experimental design techniques from engineering and bio-medical fields into games, and trusting metrics over the HiPPO (Highest Paid Person's Opinion). The talk body was not read
• Kohavi, Tang & Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, 2020 — the methodological backbone of all four checks (OEC and guardrail metrics, MDE and sample-size back-calculation, novelty and primacy effects, SRM, Twyman's law). ⚠️ Not read this run: the publisher page returned 403 and the authors' public Chapter 1 PDF could not be fetched (the tool does not support PDFs). The framework's existence and terminology were confirmed at the search layer; the specific wording reflects this card's author's existing command of the methodology and was not verified against the original text
• We cut the duration of events in a casual game in half — and increased IAPs by 50% — first-party account (Azur Games), 2026-02-26. Cited here as a live instance of the uncontrolled pre/post comparison; case details and measurement caveats are in today's 2026-09-20-mergic-shorter-events-iap card

⚠️ How sourcing shaped the tier: this is a retrospective card, selected because C3 is a blank cell on the knowledge map. The methodology itself is mature and heavily cited, but access this run stopped at the search layer — two GDC talks yielded abstracts only, and the textbook body was unreachable. Under filtering.md's downgrade rule, the evidence tier drops one step and confidence cannot exceed medium.

Evidence and sources

• Source type: mature methodology (textbook framework) + index-layer abstracts of two industry talks + one counter-example case.
• Evidence tier A, confidence medium: the framework is standard, widely validated practice in online experimentation (readable in full, it would be tier S); the deduction comes entirely from this run's access depth, not from the methodology's reliability.
• Measurement definitions:
• Back-calculating sample size from MDE requires baseline variance. No concrete sample size is given here, because it depends entirely on your metric's baseline and variance — any universal figure like "1,000 per arm is enough" is wrong
• "At least one complete event cycle" is a floor, not a sufficient condition. Products with strong seasonality (holidays, school terms) need a full annual cycle to rule seasonality out
• An A/A test catches process bugs but not everything — it does not cover underpowered designs
• The four checks are ordered deliberately: without a control group, the other three cannot rescue attribution no matter how carefully done (Mergic's situation)
• This card's inference: stating "guardrail degradation means the primary-metric gain is not a net gain" as a hard rule is this card's judgement — the textbook framing is more conservative (guardrails detect harm; whether that vetoes the change is a business call). This card takes the stricter line because retention damage from event redesign tends to be lagging and hard to reverse
• The six-way categorisation of failure modes is this card's own organisation; in the sources these concepts appear scattered, not listed this way

Scope and limits

• Genre and platform: the four checks hold for F2P mobile, PC live service, and premium + IAP. They apply weakly to premium single-player titles, whose revenue concentrates in the launch window and which have no "event cycle."
• Minimum scale: guardrail analysis and quantile reporting both need enough payers per arm. For a low-DAU product (say, a few dozen payers per arm per day), there is simply no power to detect small effects — and the right move is not a cleverer design but accepting that this change cannot be adjudicated by experiment, falling back to judgement and accepting higher risk. This is the most important boundary here: experimental methodology does not apply to very small products, and forcing it produces pseudo-conclusions.
• Organisational precondition: guardrails require cross-functional agreement (ops, data, engineering all signing off). In an organisation that only measures revenue, guardrails get treated as process obstructing the launch — fix the incentives before the method.
• Industry stage: applies to live products with continuous iteration. Pre-launch testing phases (closed beta, wipe tests) have user bases structurally unlike launch, so conclusions extrapolate poorly.
• Compliance and ethics: experiments touching pricing, drop rates or minors' spending are regulated in many countries, and A/B testing itself raises consent and data-protection questions.

What would falsify this

1. If many uncontrolled pre/post conclusions are later confirmed correct by controlled experiments → this card's dismissal of pre/post is too strong, and it may be an adequate approximation under some conditions (very large effects, no concurrent changes)
2. If industry evidence shows mean comparison misleads no more often than quantile methods under long tails → Check 2's necessity should be revised down
3. If a product shows a guardrail (say D7 retention) declining while long-term LTV rises → "guardrail down means not a net gain" must soften to "depends on the guardrail"

The weakest link in the chain: the methodology was not read at source. The existence of the four checks is confirmed at index and search layers, but specific parameters (80% power, 0.05 significance, cycle floors) are given as textbook common practice and were not verified verbatim. Separately, treating every uncontrolled comparison as "only a hypothesis" ignores a real constraint — some changes (mandatory full-client updates, regulatory requirements) cannot have a control group at all, and the methodology should then shift to quasi-experimental designs (interrupted time series, synthetic control). This card does not cover those.

What data would settle it: the textbook text, to verify the parameter wording; internal industry data on how often pre/post conclusions are overturned by later controlled tests (almost certainly never published).

Unverified

• The year, full content and specific cases of the two GDC talks (only the Overview abstracts were retrieved)
• The original parameter wording on MDE, sample size and experiment duration in the textbook
• Measured misjudgement rates of quantile vs mean methods on game revenue metrics
• The usability of quasi-experimental designs (interrupted time series, synthetic control, propensity score matching) for game live-ops where a control group isn't permitted
• The typical decay period of novelty effects in mobile vs PC live service (two weeks, a month, a quarter?)
• Alternative adjudication methods for very small products (fewer than ~100 payers per arm)