Skip to content

slides_outline.md

Latest commit

 

History

History
117 lines (86 loc) · 4.2 KB

slides_outline.md

File metadata and controls

117 lines (86 loc) · 4.2 KB

Panel Quality vs Quantity: Simulation Study

ITV Lantern Analysis

Slide 1: The Problem & Approach

The Core Question

When should we prefer a Large Sparse (LS) panel over a Small Rich (SR) panel for measuring advertising effectiveness?

Two Competing Panels

Panel Type Size Covariates Data Quality
Large Sparse (LS) 50,000 Basic demographics (age, gender, region, income) 85% ad tracking, 60% purchase linkage
Small Rich (SR) 4,000 + purchase history, online behavior, brand awareness 98% ad tracking, 95% purchase linkage

The Challenge: Confounding + Measurement Error

Confounding: Demographics drive BOTH ad exposure AND purchase probability

  • Women are 2-5× more likely to see ads (targeting)
  • Women are also 2-5× more likely to purchase (baseline behavior)
  • Naive analysis: "Ads caused huge lift!" Reality: It's confounding

Measurement Error: Neither panel observes everything perfectly

  • LS panel: Misses 15% of ad exposures, 40% of purchases
  • SR panel: Misses 2% of ad exposures, 5% of purchases

Simulation Design

Vary systematically across:

  • Confounding strength (1× to 10× gender effect)
  • True advertising effect (10-30% log-odds)
  • Measurement error (on/off)
  • Panel type (LS vs SR)

Evaluate on:

  • Statistical metrics: Bias, RMSE, coverage
  • Decision quality: Correct go/no-go decisions, expected utility loss

Slide 2: Results Interpretation

What We're Looking For

H1: SR wins under high confounding

  • Why? Rich covariates control for confounding better
  • Look for: Lower RMSE, higher decision accuracy at confounding ≥ 3×
  • Plot: [BIAS & VARIANCE BY CONFOUNDING STRENGTH]

H2: LS wins with weak confounding

  • Why? Large sample size → narrow confidence intervals
  • Look for: Similar utility but narrower intervals at confounding ≤ 1.5×
  • Plot: [CONFIDENCE INTERVAL WIDTH COMPARISON]

H3: Measurement error hurts LS more

  • Why? LS already has lower quality + attenuation bias compounds
  • Look for: Larger utility loss for LS when measurement error = TRUE
  • Plot: [UTILITY LOSS: WITH vs WITHOUT MEASUREMENT ERROR]

H4: Cross-over threshold depends on effect size

  • Why? Smaller effects harder to detect → need better confounding control
  • Look for: SR advantage emerges at lower confounding when true effect is small
  • Plot: [DECISION ACCURACY HEATMAP: CONFOUNDING × EFFECT SIZE]

Key Decision Metrics

Decision Accuracy: % of correct go/no-go decisions

  • Threshold: £100k campaign must generate break-even lift of [VALUE]%
  • Placeholder: LS: ___% correct | SR: ___% correct

Expected Utility: Average profit/loss from decisions

  • True optimal decision utility: £[VALUE]
  • Placeholder: LS achieves ___% of optimal | SR achieves ___% of optimal

Utility Loss: Cost of wrong decisions

  • Type I error: Waste £100k on ineffective campaign
  • Type II error: Miss profitable opportunity
  • Placeholder: Mean loss - LS: £[VALUE] | SR: £[VALUE]

The Bottom Line

When to use Large Sparse panel:

  • Confounding strength < [THRESHOLD]
  • True effect size > [THRESHOLD]
  • High data quality (measurement error < [THRESHOLD]%)
  • Budget constraints (cheaper to maintain)

When to use Small Rich panel:

  • Confounding strength > [THRESHOLD]
  • Small/moderate effects need precise measurement
  • Multiple confounders present
  • Decision stakes are high (utility loss matters)

Recommendation: [TO BE COMPLETED AFTER RESULTS]

Notes for Presentation

Data to fill in from results:

  1. Break-even lift percentage (from config)
  2. Cross-over threshold for confounding strength
  3. Mean decision accuracy by panel type
  4. Mean utility loss by panel type
  5. Specific scenarios where each panel wins

Plots to include:

  • bias_variance_plot.png - Shows bias/RMSE by confounding
  • decision_accuracy_plot.png - Correct decision rate comparison
  • utility_loss_plot.png - Expected utility loss by scenario

Additional slide ideas if needed:

  • Slide 3: Detailed methodology (DGP equations)
  • Slide 4: Sensitivity analysis
  • Slide 5: Business implications and recommendations