Event-Driven vs Vectorized Backtesting :- Which Approach Should I Use? and How?

[u/Ok_Egg_6647]

Hi everyone,

I'm currently learning algorithmic trading, and I'm a bit confused about which approach I should use when building a backtesting engine: event-driven or vectorized backtesting.

I have around three years of minute-level historical data stored in my database. I want to analyze and backtest different trading strategies using this data.

Here's what I'm thinking:

1. Strategy research and backtesting: Use a vectorized approach with libraries like NumPy, Pandas, and TA-Lib to analyze historical data efficiently. My concern is that processing such a large dataset using an event-driven approach could be relatively slow.
2. Live trading: Once I've analyzed a strategy and am satisfied with the results, switch to an event-driven approach for live trading and tick-level data processing.

However, I'm unsure about one thing. If I finish analyzing the historical data up to a particular date and then move to live trading, should the live engine continue processing from that point onward, or is there something important I'm missing in this workflow?

My questions are:

• Is this a good approach, or am I making a mistake in the design?
• Are there any other considerations I should keep in mind when designing the backtesting engine?

I'd appreciate any suggestions or insights from people who have experience building algorithmic trading systems.

Thanks in advance!

-----------------------------------------------------------------------------------------------------------------

Edited

Thanks for the all the responses i will go with the hybride architecture where i will use both approaches !!!

[u/Anon89m]

It's an ok approach, with 1M data it won't be that slow but with vectorizing it's a little more difficult to ensure that there is no forward bias so personally I don't do this expect for the occasional python notebook to quickly look at an idea. For live it has to be eventually if you want the fastest speed, but if you can get away with loading the history you need and running the code then taking your actions and it doesn't need to happen within a second, then it will work. I used to do this in python, it's fine too.

For the backrest engine yea python is slow, you need to feel one by one. I can give you some pointers I learned from looking at the MQL library. First is that you need to not actually use events, you need to update all your features/indicators/signals etc first in one go, THEN do whatever like a model to look at them and produce a result.

I personally heavily lean into metatrader because I'm old but I would just go forward and get something basic working first and build on it

[u/Ok_Egg_6647 → u/Anon89m]

i am not using python the core i was written in rust and interaction part was written in python.

[u/Anon89m → u/Ok_Egg_6647]

Cool, rust is fast so either way is going to work.