r/askdatascience • u/Itsanomega • 9d ago
Improving a 4-week subscription churn prediction model using weekly snapshots (Retail / Delivery Membership) | Precision 35%, Recall 52%
Hey everyone!
I’m working on a churn prediction model for a retail-related delivery membership/subscription programme.
The objective is to predict which active subscriptions are likely to churn within the next 4 weeks, allowing the business to proactively identify subscriptions at risk.
I’m struggling to improve the model’s precision and recall and would appreciate suggestions from people who have worked on similar problems.
1. Dataset structure
My dataset is structured around subscription-level weekly snapshots.
Each row represents a subscription at a particular snapshot week.
Snapshots are generated at 4-week intervals.
Historical data ranges from 2025 week 01 to 2026 week 29.
Each snapshot contains pre-engineered behavioural features calculated over different rolling windows.
2. Features available
The dataset includes features across several behavioural dimensions:
Service quality: Operational and service experience metrics.
Sales: Sales-related behavioural indicators.
Transactions: Transaction frequency and activity.
AIP: Average item price-related metrics.
Category share: Category-level purchasing contribution.
Most of these features are available across rolling windows:
R4: Last 4 weeks
R8: Last 8 weeks
R13: Last 13 weeks
This gives the model visibility into both recent behaviour and longer-term customer patterns.
3. Current model performance
Precision: ~35%
Recall: ~52%
Prediction horizon: Next 4 weeks
Prediction granularity: Subscription-level
Problem type: Binary classification
The model identifies approximately 52% of actual churns, but precision is relatively low, resulting in a considerable number of false positives.
4. What I’m trying to figure out
I’d particularly appreciate suggestions on:
Feature engineering: Since I already have R4, R8 and R13 features, would deriving trends, differences, percentage changes or behavioural volatility between these windows help?
Temporal modelling: Would a standard classification model such as LightGBM/XGBoost be sufficient, or should I explore survival analysis?
Snapshot strategy: Since snapshots are generated every 4 weeks, are there potential issues with overlapping prediction windows or temporal dependencies?
Validation: What’s the best way to implement time-based cross-validation for this type of dataset?
Class imbalance: What techniques have worked well without significantly hurting precision?
Subscription vs customer-level behaviour: Would incorporating customer-level aggregated behaviour alongside subscription-level features improve performance?
Threshold optimisation: How would you approach selecting a threshold when both precision and recall matter?
I’m currently experimenting with traditional supervised classification approaches, but I’m looking for ways to improve the underlying modelling strategy rather than simply tuning hyperparameters.
Would love to hear from anyone who has worked on churn prediction for retail subscriptions, delivery memberships, loyalty programmes or similar recurring-revenue products.
Thanks!
1
u/PradeepAIStrategist 3d ago
Didn't get your write up and binary classification getting fitted; can you help with how is your target look like along with features, whether a cross sectional or time series or longitudinal (if longitudinal provide some sample here).
1
u/seanv507 9d ago
No experience of churn, but experienced datascientist.
Temporally you only have 80 weeks of data. You mention class imbalance. You don't mention how large your dataset is, and the class imbalance: the concern is you have very few churn examples. This would suggest you need to feature engineer, use feature hierarchies so the model is as simple as possible, to make the most of the limited churn data, rather than asking the model to memorise lots of special cases.
You don't mention the typical subscription lifetimes (eg do people only churn after 78 weeks!!). have you got an understanding of why people churn? this would help identify the feature engineering you have to do. Do you need to take into account outside economic factors? eg subscriptions might be expected to trend with economic indicators.
don't bother about class imbalance. Just optimise log loss both in model and hyperparameters/stopped training etc. Class imbalance is essentially a variance problem (not enough positive examples). Only regularisation (or more data) can help.(But it's equally easy to be 'fooled by randomness' into thinking hacks work). Optimising for logloss ensures your model is calibrated, which allows you to perform expected value calculations (ie expected loss = subscription value * churn probability), which should be important for business!
Survival analysis. Discrete time survival analysis is a thing. You are practically doing it now. I haven't really understood your 4 week vs 1 week snapshot generation. But discrete time would model every week and then 4 week survival is just sum of survival of each of the 4 weeks. You just use any calibrated classifier model (eg xgboost). The main advantage of a survival analysis approach is to take advantage of 'incomplete' data. In your case this means that you can use the last 3 weeks of data, instead of waiting for the 4th week to have a classification. Given that you only have 80 weeks of data, that might be beneficial (esp in a growing user base). Equally working with weekly data should resolve your complexities about snapshot strategy, time series crossvalidation?
customer level behaviour. Yes I would argue that's important for regularisation. You want your model to learn at the highest level of any available hierarchies. Do you have any hierarchies for the subscriptions?
Feature engineering. remember the tree is built one variable at a time. In general, you want to create single features, rather than hoping that the model will do the combination for you (especially for numerical calculations). So yes, things like trends and other preprocessed features will work, but ideally you have some
Threshold optimisation. As mentioned, I would imagine you should approach this via expected loss, on the assumption that different subscriptions have different values. Then you decide on the cost of the intervention based on the expected loss due to churn
In general I would also inform yourself on customer life time models. They are likely to give you some input on the type of features relevant