Creative Intelligence — Creative Learnings & Production Iteration (tag system)

Andres Mendoza · Growth Data Analyst (Central Hybrid Team) · Analyst Edition

1 · Executive Summary

Six headline findings

1. Concept choice dominates. 40.5% of variance comes from the concept (idea). 3.3% from all 9 tags. ~56% is unexplained (execution + noise). Tags are still the only knob a producer directly turns — see §3.2.
2. 3 specific tag values reliably lift Lumina at p<0.01: gameplay_narrative_arc: Fail → Win (+24.5%, n=117), hook: Problem Statement (+20.2%, n=81), strategic_angle: Cognitive Challenge (+18.3%, n=71). Everything else is noise or worse.
3. Lumina D7 has 4 components: IPM, ROAS_D7, RET_D7, CPI (inverted). For any underperforming creative we can isolate which component is dragging the score — that's the diagnostic foundation for every recommendation.
4. 4 of the 9 tag categories define the concept's identity (gameplay_narrative_arc, hook, strategic_angle, creative_focus). They carry 61% of the information distinguishing one concept from another. The other 5 are peripheral — execution choices that vary across versions of the same concept.
5. Hard Tag Specialists exist: Alexandre de Crozals (+14.0% on hard tags, n=48) and Julie Droz (+10.9%, n=131). For Hard-difficulty tags the engine routes to them.
6. Iteration is not currently improving creatives. 98 descendants beat their ancestors; 104 underperformed them. Roughly even. Caveat: only 14% of ancestors are in the dataset, so this isn't conclusive.

2 · Business Problem & Approach

Business Problem

• Creative is the #1 lever for UA efficiency, yet creative decisions remain largely intuition-based.
• High production volume, low signal: hundreds of variations without systematic feedback.
• No feedback loop: creative teams don't know which elements to repeat, scale, or avoid.
• Performance data exists but is disconnected: Lumina tracks metrics (IPM, CPI, ROAS, Retention) but there's no link to creative attributes.
• No systematic way to diagnose why a creative underperformed.

Core Thesis

Creative performance is not random. Specific tag values consistently lift Lumina, and we can name them. The system uses these patterns to inform briefs — it's a guidance tool, not a forecasting model. Tags explain ~3% of total variance, so a recommendation is "best evidence we have," not a prediction.

The 4 Layers of Analysis

3 · Framework & Methodology

3.1 The Lumina Formulas (D0 and D7)

Lumina is a composite score. D0 measures same-day performance (≤24h after the creative ran); D7 measures mature-cohort performance (≥7 days). Each horizon uses metrics from its own time window — D7 does not borrow D1 retention, because doing so would double-count retention and reward same-day engagement at the D7 layer.

Where:

• z(·) = z-score normalization within test window (both D0 and D7) — every component is standardized against its window cohort.
• p_installs = 0.5 if installs < threshold (D0=10, D7=25) else 1.0; p_impressions = 0.8 if impressions < 2000 else 1.0.
• D7 mature = test window ended ≥7 days ago AND roas_d7 + retention_d7 are both non-null.
• Final score range = 0 to 100, rounded to 2 decimals. Higher is better.

Verification: the persisted lumina_score_d7 in creative_level_analysis.csv matches independent recompute to 0.0000 absolute difference across all 916 D7-mature rows.

3.2 The Variance Ceiling — what's actually predictable

Source: variance_breakdown_summary.csv + variance_decomposition.json (regenerated against corrected 4-component D7 formula; total D7 variance = 528.26). Eta-squared via ANOVA, grouping per-creative D7 scores by orion_concept (n = 789 D7-mature creative-window rows, 236 concepts).

Within-concept version variance — where iteration pays off

A concept's average Lumina tells you whether the idea works on average. The within-concept standard deviation tells you whether iteration on its versions pays off — high std means the best version beats the worst by a lot, so testing more versions captures real upside. This is the explore-vs-exploit signal at the concept level. Each dot below is one concept (≥3 versions); marker size = number of versions.

Source: concept_level_analysis.csv (filtered to concepts with ≥3 versions, benchmarks excluded). Quadrant reading: top-right = HIGH-CEILING BETS; top-left = LOTTERY EFFECTS; bottom-right = CONSISTENT WINNERS; bottom-left = STABLE UNDERPERFORMERS. Median Lumina and median std reference lines split the quadrants.

3.3 Concept DNA — which tags carry concept identity

Mutual information between each tag category and orion_concept, computed on de-duplicated unique creatives. Higher MI = the tag does more work distinguishing one concept from another. The top 4 tags cover 61% of the total — these are the concept's structural DNA. The other 5 are peripheral execution choices: cosmetics that vary across versions of the same concept without changing what the concept fundamentally is.

Source: concept_dna_ranking.csv built via sklearn.feature_selection.mutual_info_classif on 998 de-duplicated creatives. Concept entropy = 7.54 bits.

3.4 Reading guide

Each subsequent section is one of the 4 levels. Within each section, charts come from the CSVs listed in the appendix. Every recommendation has an evidence tier — see Section 9 for definitions.

L1 4 · Descriptive — What happened?

4.1 Portfolio overview

The dataset spans 998 unique creatives across 236 concepts, 22 producers, and 24 test windows. Of these, 916 have a computed Lumina D7 score.

4.2 Weekly Lumina D7 trajectory

Source: weekly_lumina_d7_trajectory.csv. Hovers above ~37–41 across the period. No clear improvement trend over 23 windows — consistent with the "net learning ≈ zero" finding.

4.3 Per-tag-category performance — multi-metric panels

For each tag category, six side-by-side views in two rows. Row 1: (1) share of production — how often each value is used (currently equivalent to share of spend, since auto-test creatives receive equal spend); (2) average IPM — installs per 1000 impressions; (3) average CPI — cost per install (lower is better). Row 2: (4) average ROAS D7 — return on ad spend at day 7; (5) average Retention D7; (6) average Lumina D7 — the composite north-star. Bars colored green or blue when the value is in the favorable direction vs the production-weighted overall average; grey otherwise. Dotted line = overall average. CPI is colored inversely (lower is better).

DNA tags carry concept identity; peripheral tags are execution choices. Read the DNA tag charts as "if you change this, you're effectively testing a different concept"; read the peripheral tag charts as "safer within-concept tweaks."

L2 5 · Diagnostic — Why did it happen?

5.1 Concept-class taxonomy

Every concept is classified into one of the categories below based on its mean Lumina, variance, sample size, and ceiling. This is the explore-vs-exploit map at the concept level.

Source: concept_level_analysis.csv column concept_class. 15 CONSISTENT WINNERS to scale, 19 HIGH-CEILING BETS to keep iterating, 42 STABLE UNDERPERFORMERS to kill.

Distribution of Lumina D7 within each concept class

The bar chart above shows class size; the boxplot below shows the actual distribution of per-creative Lumina D7 within each class. Read the spread (whiskers and outliers) to see how distinct the classes really are — a CONSISTENT WINNER's median ought to sit clearly above an AVERAGE PERFORMER's, and a LOTTERY EFFECT should have a wider IQR than a STABLE UNDERPERFORMER.

Source: creative_decision_facts.csv. Benchmark and single-version concepts excluded. Box = IQR, vertical line = median, dashed line = mean.

5.2 Variance decomposition

Source: variance_breakdown_summary.csv. The "Concept" row groups creatives by orion_concept (e.g. C396, R1617) — the creative idea — not by individual creative_id. Confounding via Cramér's V on the categorical pairs.

5.3 DNA tag execution difficulty

Source: dna_difficulty_matrix.csv joining tag_execution_difficulty.csv, tag_metric_correlations.csv, and producer_difficulty_analysis.csv.

5.4 Per-creative weakness diagnosis

Each D7-scored creative has a "weakest component" — the term in the Lumina D7 formula that contributed most negatively to its score. Distribution across 916 D7-scored creatives (4-component formula: IPM, ROAS_D7, RET_D7, −CPI):

• ROAS_D7 drags 311 creatives (34%)
• Retention_D7 drags 272 (30%)
• IPM drags 203 (22%)
• CPI drags 130 (14%)

Composition split by concept class:

5.5 Metric × tag heatmap — the cross-section view

For each tag value (rows, n≥30 only), the lift % vs creatives without the tag, across all 8 KPIs (columns). Cells are red→green, centered at 0%; cells annotated with the lift value. ** = p<0.05, * = p<0.10. Read across a row to see whether a tag lifts everything or only one specific KPI; read down a column to see which tags are the strongest levers for a given KPI.

Source: tag_metric_correlations.csv filtered to sample_size ≥ 30. Note: lifts are univariate (one tag at a time, no concept fixed effects) so they double-count correlation between confounded tags. Use this view for hypothesis generation, not causal claims.

L3 6 · Prescriptive — What should we do?

6.1 The 4-component decomposition flow

This is the core diagnostic mechanism. Given an underperforming creative:

1. Pull its row from creative_decision_facts.csv.
2. Identify the most-negative component (the weakest_component_d7 column).
3. Look up that KPI in lift_cube.csv → ranked tag values that lift it.
4. Filter to evidence_tier ∈ {HIGH_CONFIDENCE, MEDIUM_CONFIDENCE}.
5. For the recommended tag, check producer_tag_performance for the best executor.
6. If the tag is Hard / Very Hard, route to a Hard Tag Specialist from producer_difficulty_analysis.
7. Cross-check tag_swap_recommendations.csv for a within-concept evidence-backed swap (EXPLOIT path).

6.2 Top metric drivers — Lumina D7

Source: metric_drivers.csv filtered to metric=lumina_score_d7.

6.3 Top within-concept tag swaps (EXPLOIT)

Source: tag_swap_recommendations.csv filtered to high-confidence rows. Note: only 31 of 169 concepts have a high-confidence swap available — the rest don't have enough version diversity for the swap algorithm to fire.

6.4 The engine on one creative — a worked example

Run suggest.py --creative <id> --window <id> on any creative and you get a full diagnostic. Here's the output for an actual underperformer in concept C504 (a CONSISTENT WINNER concept — meaning C504 versions normally do well, but this specific one didn't):

==============================================================================
CREATIVE: cmmon4dxm09ay0cpnm09lsu4b  |  Window: 19  |  Producer: Jeremy Laplatine
Concept:  C504  |  Class: CONSISTENT WINNER
          D7 Avg: 68.6 (p95), D7 Max: 78.4, D7 Std: 8.3, Versions: 5
------------------------------------------------------------------------------
Lumina D7: 57.4  (concept avg 68.6, -1.4σ vs concept peers)

DECOMPOSITION (sorted by drag, most negative first):
    Retention_D7           -0.95  ←  WEAKEST
    IPM                    +0.42
    ROAS_D7                +0.47
    CPI (lower better)     +1.06

EXPLOIT — within-concept tag swaps (n=0):
    [no high-confidence swaps available for this concept]

EXPLORE — cross-concept lifts on weakest KPI (ret_d7, n=4):
    • gameplay_narrative_arc: try 'Order → Chaos'
      +28.6% on retention_d7 (p=0.000, n=524, HIGH_CONFIDENCE)  [⚠ DNA tag]
      Difficulty: Hard | Success rate: 48% ← HARD, route to specialist
        → Best on this tag: Julie Droz (+19.0% D7 lift on this tag, n=45)
        → Best on this tag: Ayca Uyanik (+11.7% D7 lift on this tag, n=17)
    • strategic_angle: try 'Relax & Escape'
      +11.4% on retention_d7 (p=0.000, n=570, HIGH_CONFIDENCE)  [⚠ DNA tag]
      Difficulty: Hard | Success rate: 48% ← HARD, route to specialist
        → Best on this tag: Julie Droz (+15.6% D7 lift on this tag, n=51)
        → Best on this tag: Nikola Kachanski (+2.7% D7 lift on this tag, n=231)
    • hook: try 'Surprise / Unexpected'
      +2.8% on retention_d7 (p=0.019, n=284, HIGH_CONFIDENCE)  [⚠ DNA tag]
      Difficulty: Moderate | Success rate: 47%
        → Best on this tag: Vira Bilous (+13.2% D7 lift on this tag, n=18)
        → Best on this tag: Nikola Kachanski (+2.5% D7 lift on this tag, n=104)
    • hook: try 'None (No Hook)'
      +1.7% on retention_d7 (p=0.032, n=525, HIGH_CONFIDENCE)  [⚠ DNA tag]
      Difficulty: Hard | Success rate: 51% ← HARD, route to specialist
        → Best on this tag: Nikola Kachanski (+5.2% D7 lift on this tag, n=144)
        → Best on this tag: Yevhenii Hrushetskyi (+2.6% D7 lift on this tag, n=34)
==============================================================================

6.5 From one-creative diagnostics to the concept brief

§6.4 is the engine analyzing a single creative. The biweekly deliverable works at a different grain — a concept brief, not a per-creative fix. Briefs ship for new concepts and new versions; nobody re-makes a creative that already shipped.

The bridge: the same logic (decomposition, EXPLOIT/EXPLORE, producer evidence) gets rolled up across the portfolio. For every concept the engine answers three questions:

• What action? SCALE, ITERATE, EXPLORE, or KILL.
• Which peripheral tags should we vary in the next versions? Hold the 4-tag DNA constant; vary one peripheral per version.
• Who should execute it? The producer with the best track record on the concept's DNA combination, plus a capability-building option.

Apply this to all D7-mature concepts and you get the cycle deliverable: Brief_Backlog_v1.html. One concept block looks like:

─────────────────────────────────────────────────────────────────────────────
C428                                                       [SCALE]  CONSISTENT WINNER
                          5 versions shipped · avg D7 = 53.7 · std = 12.9
─────────────────────────────────────────────────────────────────────────────

Rationale:
  C428 is a CONSISTENT WINNER. Goal: hold the proven DNA and systematically
  vary ONE peripheral tag per new version. The 3 versions below test the 3
  peripheral swaps with positive cross-concept evidence — generating data the
  concept currently lacks.

DNA (held constant):
  strategic_angle: Relax & Escape
  hook:            None (No Hook)
  creative_focus:  Gameplay Mechanic
  gameplay_narrative_arc: Order → Chaos

CONCEPT-LEVEL PRODUCER ROUTING:
  EXPLOIT  Producer-X +N% weighted (n on DNA tags, K/4 covered)
  EXPLORE  Producer-Y n=K on DNA tags (build capability)

Versions (3):
  V1 — None (Silent) audio                              H-2026-05-05-001
       Primary: audio_narration: Music/SFX → None (Silent)
       Expected D7 lift: +10.8%   [HIGH_CONFIDENCE, n=107]

  V2 — Core Loop / Real-gameplay representation         H-2026-05-05-002
       Primary: gameplay_representation: Exaggerated → Core Loop / Real-gameplay
       Expected D7 lift: +6.6%    [MEDIUM_CONFIDENCE, n=28]

  V3 — Always-On Banner CTA                             H-2026-05-05-003
       Primary: cta_strategy: End-Card Only → Always-On Banner
       Expected D7 lift: +2.3%    [MEDIUM_CONFIDENCE, n=46]
─────────────────────────────────────────────────────────────────────────────

Full operational deliverable: outputs_3/Brief_Backlog_v1.html — 13 active concepts (5 SCALE + 5 ITERATE + 3 EXPLORE) + 3 KILL recommendations, ~40 hypotheses total. The concept-level routing block aggregates each producer's track record across the concept's 4 DNA tags; the version-level prescriptions are the same defensible peripheral swaps applied across all SCALE/ITERATE concepts (this structural repetition is the data-generating mechanism for cycle-over-cycle learning).

6.6 How the engine builds the backlog

Four steps

Tunable knobs

Every threshold that controls the backlog lives at the top of build_brief_backlog.py. Change a number, regenerate, ship. These are not statistical thresholds — they're operational levers the team can adjust as capacity or evidence appetite changes.

What the engine does NOT do

• No learned weights. Every threshold is named in the script. No hidden model.
• No campaign-level forecasting. "Expected D7 lift" is cross-concept evidence, not a prediction of what your specific creative will score.
• No automated assignments. Producer routing surfaces track records; the Creative Producer assigns.
• No approval authority. Every recommendation can be overridden. Overrides are themselves data — over time we learn which recommendation types the team trusts.

Honest about the limitations

Two known weaknesses worth stating up front:

• Top-N counts are operational, not statistical. Picking 5 SCALE / 5 ITERATE / 3 EXPLORE / 3 KILL is sized to roughly match motion-design capacity per biweekly cycle. If capacity changes, change the knob.
• The defensible-swap filter is observational. When we say a peripheral swap "lifts D7 by +13.7%," that's the cross-concept correlation — we don't yet control for which concepts the swap was tested on. The biweekly brief design (same swap applied across multiple concepts each cycle) is exactly what generates the controlled data needed to compute true causal lifts in 5+ cycles.

Every cycle's run prints exact selections to stdout: concepts picked, defensible swaps qualified, CV quantiles used. That's the audit trail.

The hypothesis tracker — how the engine learns

L4 7 · Strategic — Where should we invest?

7.1 Production concentration — OVER vs UNDER-invested tags

For each tag value, the chart shows how its mean Lumina D7 differs from the overall D7 average across the 825 D7-eligible non-benchmark creatives. Red = OVER-INVESTED (high volume + below-average D7 performance — these are the tags consuming most of your production capacity but pulling your average down; the rebalance candidates). Green = WELL-INVESTED. Orange = UNDER-INVESTED (low volume + above-average performance — scale candidates).

Source: outputs_3/production_concentration_d7.csv — the Lumina D7 version, with benchmark creatives and "No Tag Applied" rows excluded (benchmarks recur across windows and would inflate volume artificially). Built by outputs_3/build_production_concentration_d7.py. NEUTRAL-status tags (medium-volume) are hidden in this chart for clarity; see the CSV for the full list.

7.2 Underexplored DNA recipes — the EXPLORE frontier

9-tag recipes flagged is_underexplored=True by the pipeline (recipe appears far less often than expected) AND with at least one D7-mature creative. Sorted by avg Lumina D7. These are direct EXPLORE candidates.

Source: underexplored_combinations.csv. Sample sizes are intentionally small (n=3–5) — that's the definition of underexplored.

7.3 Producer overview & specialist routing

Hard Tag Specialists (sorted by lift on hard tags vs peers):

8 · The Suggestion Engine — Architecture

The engine sits on top of three new tables in outputs_3/:

1. creative_decision_facts.csv — atomic per-(creative, window) table with the 4-component Lumina D7 decomposition (IPM, ROAS_D7, RET_D7, CPI). Joins concept-class. 1,078 rows.
2. lift_cube.csv — (tag × KPI × producer) lookup. 523 rows × 29 cols. Pre-computed evidence tier (HIGH/MEDIUM/EXPLORATORY/INSUFFICIENT_DATA).
3. dna_difficulty_matrix.csv — DNA tag values × difficulty × success rate × specialist routing. 27 rows × 19 cols.

Plus the orchestrator: suggest.py. Run as:

Evidence tiers (used everywhere in the engine)

9 · Data Quality & Limitations

9.1 What this analysis does NOT claim

• Causal effects. All lifts are observational. Concepts that already perform well tend to use specific tags — causation runs both ways.
• Concept-controlled tag effects. tag_metric_correlations.csv and marginal_effects.csv do NOT control for concept. With concept explaining 40.5% of variance, this matters.
• Producer-specific causal claims. "Nikola lifts Cognitive Challenge by 7.7%" partly reflects which concepts Nikola was assigned, not pure execution skill.
• Predictions. R² on the multi-tag OLS model (marginal_effects.csv) is 0.13. This is a guidance system, not a prediction model.

9.2 Sample-size flags by analysis

9.3 Known data hygiene issues

• Typo splits a tag bucket: "Divergent / Fake Gampeplay" (n=2, typo) and "Divergent / Fake Gameplay" (n=46) are the same value with different labels.
• Producer role is constant: s_created_by_user_role = "Internal" for all 1,078 rows — no internal/external producer dimension is analyzable.
• recommendation_outcomes.csv is empty: the closed-loop tracker exists but has never been populated. Engine cannot self-evaluate.
• 84 untagged creatives (8.4% of total), including 6 benchmarks. Tag coverage = 91.6%.

9.4 Confounding warnings (Cramér's V)

• 16 of 36 tag pairs show V > 0.5 (severe multicollinearity). E.g., strategic_angle ↔ gameplay_narrative_arc = 0.55.
• 4 of 9 tags are confounded with producer at V > 0.5 — tag effects may just reflect producer style.
• 5 of 9 tags are temporally confounded with test windows (V > 0.3) — usage shifts over time.

10 · Roadmap & Open Questions

10.1 Highest-leverage upgrades

1. Add concept fixed effects to tag_metric_correlations.csv and marginal_effects.csv. Will reduce most "tag lift" claims, surface the few that survive — those are the real ones.
2. Close the recommendation_outcomes loop. Track which suggestions producers acted on, what happened. Without this we can't measure engine quality.
3. Increase ancestor coverage for inheritance_learning_analysis. Currently 14% — push to ≥50% to make iteration claims trustworthy.
4. Build per-creative explainability beyond the 4-component decomposition. Currently we say "IPM is low"; we don't say "IPM is low because the hook plays at 1.4 sec instead of 0.8."

10.2 Open questions for the team

• Are CONSISTENT WINNERS being deliberately under-varied? Most have no within-concept swap evidence — they're not teaching us anything.
• What fraction of "underexplored" recipes are recipes the team intentionally avoids (brand reasons)? Without this filter, the EXPLORE list overstates opportunity.

11 · Appendix — CSV Reference

Every file used in this report, where it lives, and which level uses it: