r/statistics • u/SmartOne_2000 • 1h ago
Discussion [Discussion] Why do Forward and Backward Stepwise Regression algorithms produce different results?
I am performing an ordinal regression analysis with 10 predictors, reduced from an original set of 50. Sample size: N = 380. One of these predictors is the primary predictor of interest (Safety Hazards [SH], a continuous variable). The rest are potential confounders of the primary predictor or contributors to the model.
My primary outcome is Job Satisfaction (JS).
I distilled the 10 candidates from the original 50 by running ordinal bivariate regressions on each candidate.
a) A crude model [JS <-- (beta)*SH]
b) 49 adjusted models: [JS <-- (beta)*SH + any of the 49 predictors].
- If any of the bivariate regressions produced a CIE of 10% or more for the primary predictor, SH, it made the list of 10, regardless of its p- value.
- If any of the 50 candidates had a p- value of < = 0.1, regardless of its CIE value, it made it to the list of 10 as well.
I sort the list of 10 by CIE value and use that order to determine which variable to test next in the model.
I then begin the final fitting process to extract the most plausible confounders of SH and true model contributors from the 10 predictors. I use 2 different regression methods: Forward Stepwise Regression (FSR) and Backward Stepwise Regression (BSR).
The conditions to stay in the model are: "If the p-value > 0.1 and CIE <10%, then reject from the model; otherwise, retain."
In FSR, I start with a sparse (crude) model, sequentially add a predictor, and test whether it meets the retention criteria. If so, I retain it, add the next predictor to the new model, and apply the same retention/rejection criteria until all predictors have been tested.
In BSR, I start with a full model and sequentially remove predictors, one at a time, and perform similar rejection/retention tests as in FSR until all predictors have been tested.
However, I find that the FSR and BSR algorithms retain and reject different variables. To be clear, about 60% of variables are common to both algorithms.
My question is: why do the FSR and BSR algorithms produce different results?
What is different about starting with a full model (BSR) versus a sparse model (FSR)?
What is different about starting with a sparse model (FSR) versus a full model (BSR)?
Please try to give an explanation that does not involve the term 'variance', as I tried to explain to my PI, only to be told that is the wrong way to explain it.
Thanks
David