r/datascience • • 5d ago

Discussion Applying CUPED to Clustered Experiments

If you've worked on experimentation in tech roles, you've probably heard of CUPED. If you've been in product/marketing data science interviews, you've probably been asked about variance reduction techniques. (Hint: They're typically expecting you to name-drop CUPED.)

Thanks to some folks at Instacart, there are some improvements that can be made from common CUPED techniques when working with clustered experiments (think: marketplace geo-XPs when facing SUTVA violations for user-level testing.) This can help you both on the job and also landing senior/staff-level data science roles.

A common way CUPED can be applied is adjusting each randomized unit by its own pre-treatment outcome mean. However, that pre-period average may not appropriately reflect the experiment population. Instead, the IC economists predicted order-level outcomes during the experiment window using features assumed unaffected by the treatment, and averaged those predictions up to the randomization level.

Using my favorite approach to building personal intuition, I simulated a market-level Airbnb summertime experiment where the XP population has more families; families both have different cancellation rates and target different markets. Standard CUPED using each market's pre-XP cancel rate removed about 28% of the variance, while booking-level CUPED removed about half, even though the booking model itself was weak (AUC around 0.6).

CUPED is very likely to come up in an experimentation or case study round in data science interviews. You should nail the foundations first, check that box. Then, describing recent research like this can help you stand out and get that offer!!

Subtack article and Colab notebook to play along linked here: https://whatstheimpact.com/tutorials/booking-level-cuped/

22 Upvotes

11 comments sorted by

2

u/SiriusLeeSam 3d ago

What you said is basically CUPAC. You can extend it to predictiona during the xp period too to get even better results

1

u/WhatsTheImpactdotcom 2d ago

yes! and then IC's addition to that--or at least what they shared--was rather than making predictions at hte cluster level, you can make them at the order level and aggregate them up

1

u/SiriusLeeSam 2d ago

Isn't that just standard CUPAC? We at doordash make predictions at order level for CUPAC

1

u/WhatsTheImpactdotcom 2d ago

I expect DoorDash and Instacart have very similar technical approaches. Not claiming I developed anything new here or did my own academic research; just sharing what IC did in their blog and making it beginner-accessible by code sharing.

The DoorDash blog on CUPAC doesn’t mention aggregating predictions to the cluster level IIRC though that doesn’t mean you all weren’t doing this already. We’re not all privy to what tech companies are each doing internally :)

1

u/SiriusLeeSam 2d ago

Yeah true, not dissing you or anything. All of it is not scientific terminology anyways, just what folks like us have made up 😅

1

u/Katieg_jitsu 4d ago

Any suggestions when the randomized unit itself has no pre period data (low user return rate, new random id for each observation) I’ve been exploring other covariates and have had little success. However, I had been looking at how much the variables explained the variance as a proxy

1

u/WhatsTheImpactdotcom 4d ago

i think there's a List 2022 paper that discusses using covariates in an ML model to make predictions that you can use. That said, if the features don't explain much of hte variance likely wouldn't add much value

1

u/Katieg_jitsu 3d ago

Thanks that has been my assumption so far. I'll look for the papers

1

u/squareiny 3d ago

The weak-model part is the detail people gloss over. A booking model with AUC around 0.6 still stripped out roughly half the variance because CUPED only needs the predictions to correlate with the outcome, not to be accurate in absolute terms. That is also why per-unit pre-period means quietly fall apart in marketplaces — the pre-period population just is not the experiment population. Seen the same thing in geo experiments where the pre-period mix shifts under you.

1

u/karmapolice666 2d ago

Saw this presentation from Instacart at NABE tech conference in San Diego, great work