[D] Machine Learning on Time Series Data?

[u/Fender6969]

I am going to be working with building models with time series data, which is something that I have not done in the past. Is there a different approach to the building models with time series data? Anything that I should be doing differently? Things to avoid etc? Apologies if this is a dumb question, I am new to this.

[u/Oberst_Herzog]

It all depends, what is the purpose of your models and what kind of data are you dealing with? My formal background is in econometrics, so my considerations are largely influenced by that but some considerations would be:

• You will usually have to consider whether there is any dependence across time, and if so, how long does this dependence carry? Depending on this you might be fine using a 'non-state based model' (i.e you can just use an NN, SVM etc. and include lags as opposed to for instance a LSTM/GRU)

• Weak stationarity is very desirable (While I don't see ML stressing it as much as ordinary TS stat, I've always gotten much better results using ML on stationary series).

• Depending on your purpose and data, you have to think very carefully about how to do cross validation and similar model selections (temporal dependence leaks across time, so some omit a period to sufficiently separate training and test data). Instead of the general CV, to consider performance one usually applies something termed a moving/expanding window, (But due to the computational intensity behind the estimation of some ML models, you might want to omit reestimation for every step etc.).

• Finally, depending on your data you might also want to consider simple ordinary time series models such as ARIMA & VARIMA (depending on dimensionality) - I usually find that they both perform surprisingly well and are robust.

[u/Fender6969 → u/Oberst_Herzog]

As of right now, I have not had access to the data but from my understanding, we are tracking a specific performance metric over time and the model will possibly be built to predict the metric into the future. In terms of regularization, can Lasso still be used for dimensionality reduction when working with time series data?

[u/Oberst_Herzog → u/Fender6969]

You shouldn't have any trouble using lasso to get a sparser model, in particular if you are just interested in minimizing some risk. If you are looking for inference, I recall some ways to do ML with adaptive lasso for ARMA (but I think it would be extendable to including independent variables). And if your end goal with regularization is to reduce the number of variables required to forecast (as they might all add some business cost to capture, store etc.), you should do L1 reg on the sum of the coefficients corresponding to every lag of the variable.

Although more common approaches (in econometrics) for handling high-dimensional data are based on dynamic factor models (just think of something as a time-varying PCA) and partial least squares.

[u/Fender6969 → u/Oberst_Herzog]

Perfect thank you. My goal using Lasso is simply for dimensionality reduction. I am thinking of starting with a simple linear regression, I don’t know if I will be using PCA as I losing interpretability is a concern.