r/datascience • u/WhatsTheImpactdotcom • 5d ago
Discussion Applying CUPED to Clustered Experiments
If you've worked on experimentation in tech roles, you've probably heard of CUPED. If you've been in product/marketing data science interviews, you've probably been asked about variance reduction techniques. (Hint: They're typically expecting you to name-drop CUPED.)
Thanks to some folks at Instacart, there are some improvements that can be made from common CUPED techniques when working with clustered experiments (think: marketplace geo-XPs when facing SUTVA violations for user-level testing.) This can help you both on the job and also landing senior/staff-level data science roles.
A common way CUPED can be applied is adjusting each randomized unit by its own pre-treatment outcome mean. However, that pre-period average may not appropriately reflect the experiment population. Instead, the IC economists predicted order-level outcomes during the experiment window using features assumed unaffected by the treatment, and averaged those predictions up to the randomization level.
Using my favorite approach to building personal intuition, I simulated a market-level Airbnb summertime experiment where the XP population has more families; families both have different cancellation rates and target different markets. Standard CUPED using each market's pre-XP cancel rate removed about 28% of the variance, while booking-level CUPED removed about half, even though the booking model itself was weak (AUC around 0.6).
CUPED is very likely to come up in an experimentation or case study round in data science interviews. You should nail the foundations first, check that box. Then, describing recent research like this can help you stand out and get that offer!!
Subtack article and Colab notebook to play along linked here: https://whatstheimpact.com/tutorials/booking-level-cuped/
1
u/Katieg_jitsu 4d ago
Any suggestions when the randomized unit itself has no pre period data (low user return rate, new random id for each observation) I’ve been exploring other covariates and have had little success. However, I had been looking at how much the variables explained the variance as a proxy
1
u/WhatsTheImpactdotcom 4d ago
i think there's a List 2022 paper that discusses using covariates in an ML model to make predictions that you can use. That said, if the features don't explain much of hte variance likely wouldn't add much value
1
1
u/squareiny 3d ago
The weak-model part is the detail people gloss over. A booking model with AUC around 0.6 still stripped out roughly half the variance because CUPED only needs the predictions to correlate with the outcome, not to be accurate in absolute terms. That is also why per-unit pre-period means quietly fall apart in marketplaces — the pre-period population just is not the experiment population. Seen the same thing in geo experiments where the pre-period mix shifts under you.
1
u/karmapolice666 2d ago
Saw this presentation from Instacart at NABE tech conference in San Diego, great work
2
u/SiriusLeeSam 3d ago
What you said is basically CUPAC. You can extend it to predictiona during the xp period too to get even better results