Test → Applovin Scale Analysis v3.2

Methodology & Findings · Data Science Report · Andres Mendoza — Growth Data Analyst (Central Hybrid Team)

1 · Executive Summary

Five sentences for stakeholders

1. We analyzed 669 creatives that ran on both the Facebook auto-test pipeline (with mature D7 Lumina scores) and Applovin (with real impressions, clicks and spend across 4 months).
2. Applovin spend is extremely Pareto-distributed — top 1% of creatives captures 74% of recovered spend — so we used a top-decile binary classification (≥$2,820 spend) as the cleanest outcome.
3. Lumina D7 does carry predictive signal for Applovin scaling, but it is partially hidden by a sample-size confound: Facebook auto-test allocates more impressions to creatives that perform well early, so IPM/ROAS/installs all move together. After controlling for install volume, Lumina D7's partial correlation with scale rises from ρ = +0.02 to ρ = +0.12 (p = 0.001).
4. A 5-feature composite reaches AUC 0.77 — combining per-window-normalized Lumina with raw-level economic metrics and audience reach. A cleaner-causal alternative using only Lumina D0 + Lumina D7 + install_share + log(installs) reaches AUC 0.70 with unambiguous coefficient signs.
5. The recommendation is to publish the Lumina Cross-Network Scale Score alongside Lumina in the dashboard, not to modify Lumina itself; Lumina remains the right tool for within-cohort Facebook ranking.

2 · Business Problem & Why This Matters

What we wanted to answer

The Lumina Score is the team's north-star metric for ranking creatives in Facebook auto-test campaigns. The score is well-defined for its original purpose: which creative outperformed its window-mates in the same Facebook test cohort?

But the operationally meaningful question for the UAM team is different: "if a creative scored well on the Facebook test, does it actually scale on Applovin?" Applovin is where most of the production spend happens — Lumina's value rises if it predicts Applovin scaling, and falls if it doesn't.

Why this matters financially (with recovered-data caveat)

ALL_ creative spend on Applovin grew sharply from $950K (Feb) to $5.24M (Apr) in our recovered window, driven mostly by a few breakout creatives. The UAM team selects creatives for Applovin partly based on Facebook test signals. If that signal is weak, the team is essentially picking by intuition; if it's strong, the pipeline is justified.

Why this is harder than a typical correlation question

• Selection bias (see Section 3 — confirmed with data, not just hypothesized).
• Extreme outcome distribution: Pareto-on-steroids (Gini 0.91). 6 creatives carry 74% of all recovered spend.
• Tiny upper tail: only 17 creatives reach $10K+ in recovered Applovin spend. Insufficient sample size for top-tier modeling.
• Historical blind spot: 453 high-scale Applovin creatives (incl. R1426_V1 $480K, R1608_V1 $391K) were never in the auto-test pipeline — they're pre-October-2025 historical winners, confirmed.
• Cross-platform translation: Lumina is calibrated for Facebook; Applovin's algorithm rewards different signals than Facebook's.

3 · Operational Context — What Shapes the Cohort We're Modeling

Before interpreting predictive results, we examine which test-phase signals are systematically different between the creatives that reached Applovin and those that did not. This is not about evaluating any team's process — it is about characterizing the population our model is trained on, so that downstream conclusions (especially about Lumina) are read in context.

3.1 — Comparing test-phase metrics: propagated vs not-propagated cohorts

This snapshot comparison is valid for the full cohort (it does not depend on timing). We display the percentage difference between medians on a unified scale to make small but real operational differences visible — raw values on different units obscure the pattern.

3.2 — Propagation rate by Lumina quartile, concurrent windows only

Restricted to windows 17–24 (Feb–Apr 2026), where Applovin observation overlaps with test end. n=253. This is the data-honest slice for a direct propagation-rate measurement.

3.3 — Inventory: top-quartile-Lumina creatives that were not activated on Applovin

A direct artifact of the propagation pattern: which creatives ranked in the top quartile of their test window (by Lumina D0 or D7) but were not activated on Applovin? Below is the per-window count — useful as a candidate-audit list for retrospective review.

40 of 191 top-Q4-D0 creatives (21%) were never activated on Applovin. Five of them scored a perfect Lumina D0 = 100 (top of their window). The full inventory is in Q4_D0_not_picked_inventory.csv and Q4_D7_not_picked_inventory.csv. Top 15 examples:

3.4 — Implications for the modelling that follows

Because propagation is partly orthogonal to Lumina rank, the cohort of "activated + scaled" creatives we use for modelling is shaped by operational filters (CPI, CTR, installs). This has three consequences for how the results in subsequent sections should be read:

• Univariate AUC for Lumina-only models is diluted by the operational filtering in the propagation step — Lumina's true predictive power on a less-filtered population is likely higher than the 0.50 we observe here.
• The composite model in Section 9 partially compensates because it includes the same operational signals (IPM, ROAS, install_share) that propagation is implicitly using — giving the score access to both the rank-based view (Lumina) and the absolute-level view.
• The 40 "top-quartile-Lumina, not propagated" creatives in Section 3.3 are candidate audit cases, not necessarily mis-decisions. Some may have had operational reasons not visible in the test data; others may genuinely represent untapped scale potential. The inventory exists so that retrospective review can answer this.

4 · Methodological Approach

The full analysis ran through 13 sequential approaches, each designed to address a specific weakness of the previous one. The methodology is summarized below; each step's rationale is explained in Section 7.

4.7 — Statistical methods used in this report

This subsection is a self-contained reference for the statistical tests cited throughout — readers can return here when they hit a "CV AUC = 0.77 ± 0.08" or "Spearman ρ = +0.12, p = 0.001" in later sections.

5 · Data Sources & Cleaning

Key cleaning decisions

The strict reliability funnel

Starting from 1,425 unique tokens (test ∪ Applovin), we filter down to a 669-creative cohort that meets every reliability criterion. Every analysis in the results section uses this cohort.

6 · Outcome Distribution & Why It Shaped the Method

Tier composition of the reliable cohort (n=669) — corrected presentation

The previous version of this chart used dual y-axes which made it hard to read the actual counts. Below the two views are split apart for clarity.

The four MEGA creatives ($100K+) account for 68% of all reliable-cohort spend; the 13 SCALED ($10K–100K) add another 13%. Together, 17 creatives carry 81% of the recovered spend; the remaining 652 share $603K (19%). This is why any "what makes a winner" question has a small-sample problem.

7 · Tests Conducted & Why Each One

The analysis ran 13 approaches. Each was motivated by a specific question that arose from the previous step. The honest progression — including dead-ends — is documented here.

8 · Statistical Results

8.1 — Univariate predictive power (single metric → top decile)

How well does each test metric, used alone, rank creatives by their probability of reaching the top decile of Applovin spend? AUC = 0.5 means random; AUC = 1.0 means perfect.

Key observations:

• install_share is the strongest single feature (AUC = 0.65) — creatives that captured a larger share of installs in their test window are more likely to top-decile-scale on Applovin.
• log(ROAS D0) and log(ROAS D7) tie next at AUC ≈ 0.63 — but with inverted sign (see Section 8.4 for explanation).
• Retention metrics cluster around AUC 0.55–0.59 — directional but inconsistent.
• Lumina D7 used alone sits at AUC = 0.50 on the propagated cohort — but this is an observational ceiling shaped by the operational filters that select which creatives reach Applovin (Section 3). It is not a verdict that Lumina is uninformative; in the multivariate model Lumina D7 receives the largest positive coefficient (Section 8.5).
• The next-weakest single feature is CTR % (AUC = 0.54). Several other operational signals are also weak in isolation but complementary when combined.
• Lumina D0 is also weak alone (AUC = 0.51) — but together with the other features, they add multivariate value.

8.2 — Multivariate AUC progression with install_share included

8.3 — Tested alternative composites (incl. user-suggested maturation forms)

8.4 — The install-volume effect: why Lumina D7's true signal is masked

This section explains the most consequential mechanism we found: Lumina D7 has real predictive power for Applovin scaling, but that power is partially obscured because Lumina's components and absolute install volume are measuring overlapping economic quality. Understanding this is what makes the difference between a model that hits AUC 0.77 by leveraging the overlap and a model that hits the same AUC with coefficients you can defend operationally.

8.4.1 — Why Lumina's components and install volume are not independent

Lumina rewards high-IPM creatives (more installs per impression — a real efficiency signal) and high-ROAS-D7 creatives (long-maturation paying users — a real monetization signal). Both of these intuitions are correct. The complication is that, under our fixed-budget auto-test design, install count and IPM are essentially the same axis: a creative with high IPM mechanically accumulates more installs at the same total spend. Same with CPI: a creative that's cheap to acquire installs at gets more installs. As a result, the four "different" Lumina components (IPM, CPI, ROAS_D7, RET_D7) are not measuring four independent dimensions of quality on this dataset — they're measuring the same underlying conversion-efficiency signal from four highly-correlated angles.

The bucketed view makes the mechanism concrete. Across 6 install-volume buckets in the reliable cohort:

The gradient from <30 installs to 150+ shows IPM rising from 0.69 to 3.27 (4.7× higher), CPI falling from $1.24 to $0.17, and Lumina D7 climbing from 20.6 to 66.8 in lockstep. Within a single test cohort, the operational metrics and the composite that uses them are largely co-determined by sample size.

8.4.2 — Partial correlations: Lumina D7 emerges as the cleanest signal once installs are controlled for

To separate volume-driven contribution from quality-driven contribution, we compute partial Spearman correlations between each test metric and log(Applovin spend), controlling for log(test installs):

8.4.3 — What the negative IPM and ROAS_D7 coefficients are really doing

The negative coefficients on log(IPM) and log(ROAS D7) in the 5-feature composite are not a claim that low-IPM creatives scale better. They are the model performing an algebraic decomposition: extracting the parts of Lumina D7 that come from other components (retention, CPI residual) by subtracting out the parts that come from IPM and ROAS_D7.

Recall the Lumina D7 raw formula, expanded with its actual transforms:

The composite uses log(IPM + 1) and log(ROAS_D7 + 1) directly — i.e. the same transform Lumina applies internally, minus the z-score step. That is exactly the point: z(log(metric + 1)) = (log(metric + 1) − μ_window) / σ_window, so the un-z-scored log(metric + 1) carries the absolute-level information that the z-score step removes. When the composite has access to both Lumina D7 and the raw log-transforms, the model can decompose:

That is: the negative IPM/ROAS coefficients remove the IPM and ROAS pieces from Lumina so that the surviving signal is the retention and CPI residual — plus the pure cohort-rank information that Lumina's z-scoring preserves. Under our fixed-budget test design, log(IPM + 1) carries the absolute conversion-efficiency level, log(ROAS_D7 + 1) carries the absolute payback level, and Lumina D7 carries those plus retention and CPI plus the within-window rank. Subtracting the first two from Lumina isolates the rest. The composite isn't reaching for a different metric than Lumina uses — it's giving the model access to the un-normalized version of the same metric, which carries the level information that the z-score step strips out.

This is verifiable by tracking how the IPM coefficient changes as we add features:

The shrink from −0.71 to −0.18 (IPM) when log(installs) is added explicitly is the cleanest evidence of the mechanism. The IPM coefficient was carrying the absolute-volume signal because nothing else could; once log(installs) is given the job directly, IPM gets to keep just its smaller, true marginal effect.

8.5 — Standardized coefficients of the recommended composite

8.6 — Lumina quartile rates by outcome

If Lumina were a strong predictor on its own, top-decile rates would rise monotonically Q1→Q4. The data shows weak, non-monotonic patterns — consistent with the selection-bias evidence in Section 3.

9 · The Lumina Cross-Network Scale Score — Two Recommended Versions

Given the install-volume mechanism described in Section 8.4, we present two recommended composites. They make different trade-offs between predictive performance and coefficient interpretability — both are deployable, the choice depends on the downstream use case.

9.1 — Composite A: High-AUC version (recommended for ranking)

Use this version when: you need maximum top-decile ranking accuracy and the coefficients will not be shown to stakeholders for direct interpretation. The negative coefficients on IPM and ROAS_D7 are statistical corrections (see Section 8.4) but can be confusing.

9.2 — Composite B: Clean-causal version (recommended for stakeholder explanation)

Use this version when: you need to explain to non-data stakeholders why a creative scored high. All quality signals carry positive coefficients in the natural direction; the only negative coefficient is the install-volume control, which has a clear statistical justification ("we already account for the fact that high-install creatives looked artificially good on Facebook").

9.3 — Comparison of all considered composites

9.4 — Why the volume control matters

The key insight from Section 8.4: Lumina D7 has real predictive power that is largely hidden by the install-volume confound. The two composites above handle this differently:

• Composite A uses the negative coefficients on log(IPM) and log(ROAS D7) to implicitly subtract the install-volume contribution. The model is more accurate, but the negative signs invite misreading.
• Composite B uses explicit log(test_installs) as a control variable. The remaining coefficients on Lumina and install_share carry only the residual quality signal, with the natural positive sign throughout. Slight AUC cost, big interpretability win.

9.5 — What does NOT work (rejected alternatives)

10 · Findings Ranked by Reliability

11 · Known Limitations & Caveats

12 · Actionable Recommendations

Now (no new data needed)

1. Add the Lumina Cross-Network Scale Score to the dashboard alongside Lumina D7. Use the 5-feature composite as the basis. Keep Lumina D7 unchanged for its original purpose (within-cohort Facebook ranking).
2. Brief the UAM team on the selection-bias finding. They've been using CPI + CTR + raw installs — that's not wrong, but it's not Lumina either. The new composite formalizes a signal closer to what's actually predictive.
3. Build a sustainability score separately. Retention D1 + ROAS predicts which creatives stay active 4+ months (AUC 0.59). Different audience use case from "will this peak-scale."
4. Investigate the playtime inversion qualitatively. Look at the 17 winners' creative content — what's similar about them?

Next 1–3 months (data accumulation)

1. Push for finer Applovin spend granularity via API access (the monthly export limit is the binding constraint right now).
2. Increase test installs per creative — target 200+.
3. Validate the 5-feature composite prospectively. Score the next 50 test creatives, see if predicted top-deciles actually scale.
4. Log UAM picks with reasons to make the selection model more learnable.

Longer-term (months 3–6)

1. Re-fit the composite weights against Applovin scale labels. Once ≥50 $10K+ winners exist.
2. Pull pre-October-2025 Facebook auto-test data. Would fill in the 453-creative blind spot.
3. Build an Applovin-side companion score based on Applovin's own D1/D7 metrics.

13 · Appendix: File & Reproduction Index

Master tables (canonical)

Cleaning & joins

Models & coefficients

Crosstabs & winner inspection