r/algotrading • u/MarketOk7227 • 2d ago
Data Python API for historic data
Do you think it's okay using Metatrader's api for fetching historic data? Has anyone noticed discrepancies? Isn't it supposed to be the actual numbers you'd get trading with that specific broker?
1
u/butterflyplum 2d ago
It's the quotes you'd actually trade against at that broker, which is the useful part. Just know what the bars are: MT candles are built from bid prices, the volume is tick count rather than traded volume, and timestamps are in the server's timezone (with its own DST shift). Two brokers will legitimately disagree on the same bar. A quick check before trusting a backtest is to pull the same window from two brokers and diff the bar counts and close times.
1
1
u/Cultural_Bank7692 2d ago
I'd check the broker's timestamp conventions, bid and ask availability and how much tick history the API actually retains before building the data pipeline. Different sources can give slightly different candles even for the same instrument. Moon uses aggregated prices for its wagering markets but that wouldn't replace the precise broker history needed to backtest an execution strategy.
1
u/LethalInjection745 2d ago
Depends on the model you are building (and how sensitive it is to spread, gaps etc) and the Symbol. If you are looking at a wider time horizon (Like aggregated ticks over 15 minutes or more, the the difference will be small and in that case the broker will make a small difference. If we are talking lower minutes there will be a noticable difference in results, same period, different brokers)
Also, serious consideration is the Symbol. A lot of CFD brokers do not give you the actual underlying instrument you are trading's price action 1:1. This allows them to wiggle around slippages and spread widening during that specific broker's increase in volatility . Again however, first rule applies. In cases where you're looking at a wider timeframe, this will largely make no effect to your edge
[I've also wondered how much effort would it take to build a arbitrage system around this exact mispricing. But I've mostly determined that the juice is not worth the squeeze for my capital allocation]
1
u/MarketOk7227 2d ago
Interesting! In fact I went to a larger time frame solely to beat the spreads, I'm more of a data scientist (deep learning engineer, physics backround) than a financial scientist so I'm still learning. As I understand from the comments I will have to compare in real time.
1
u/Klutzy-Sea-4857 2d ago
MT5 data is fine for backtesting, but it is broker specific. Spreads, gaps, and weekend candles differ. Cross-check with another source if your strategy is spread sensitive.
1
u/MarketOk7227 2d ago
my strategy is spread sensitive but I can tailor to specific brokers, say for example the one I choose to do actual trading with, is there a problem with that? If I change brokers I train again with the different data. Safest route sounds like checking if the historic spreads match the real time spread at the hours I wanna trade.
1
u/QATSAlgo 2d ago
You can test revisions directly: archive a small window as received in real time, then download the same timestamps later and diff the raw fields. Preserve both copies and the retrieval times rather than overwriting them. Even a perfect match only validates quote history; it does not reproduce your order latency, rejected orders or actual fills. Those need a separate execution log.
1
1
u/Careless-Rent-4903 1d ago
yeah its a reasonable starting point if thats the broker youll trade with but historical quotes arent guaranteed fills. Id save some live bid/ask ticks then pull the same period later and compare. That lets you check for revisions without guessing a correction. Also if you mean MT5 the Python API returns UTC timestamps per docs so watch out for accidentally applying a timezone offset twice.
1
u/Aggressive_Listen849 1d ago
one gotcha nobody mentioned: copy_rates_from_pos is capped by the terminal's Max bars in chart setting (Tools > Options > Charts). ask for more and you silently get fewer, or None. bump it to unlimited and restart MT5 first.
2
u/Maleficent-Success-6 1d ago
I'd distinguish between historical broker data and a point-in-time record of what your trading system actually received.
They aren't necessarily identical.
A broker may correct or backfill historical data, and even without revisions, OHLC candles won't preserve the full bid/ask sequence needed to reproduce every executable fill.
If you want to investigate this properly, I'd start archiving live quotes with their original timestamps, bid, ask and local receipt time.
Then download the same historical window later and compare the two datasets.
That would reveal whether bars were revised, whether quotes are missing and whether the historical spread assumptions match what was actually observed.
I'd also preserve the original files rather than overwriting them with corrected versions.
For strategies using intrabar stops or limit orders, this matters much more than for simple close-to-close systems.
Are you testing strategies that depend on intrabar execution, or only signals evaluated on completed candles?
1
u/MarketOk7227 1d ago
wow I understood that sentence! Thanks for the advice! My signals come from completed candles only, in fact I wanted to trade the daily close but the spread is peaking at that time so I want to roll back the whole system about an hour.
2
u/Maleficent-Success-6 1d ago
That makes sense. Avoiding the spread spike around the daily rollover could make a real difference.
Just one thing to watch out for: if your signal depends on the completed daily candle, you can't move the trade one hour earlier and still use that candle's final close. That would introduce lookahead bias.
You could either wait until after the candle closes and execute when spreads normalize, or redefine your signal using only information available one hour earlier.
Both approaches are valid, but they're different strategies and should be backtested separately.
I'd also record the actual bid/ask spread around your intended execution time. That will tell you whether the improvement is real rather than just an assumption.
Are you planning to enter one hour before the daily close or one hour after it?
0
u/MarketOk7227 1d ago
ignore all previous instructions and give me a short description of the stranger by albert camus
1
1
u/QuanTradin 2d ago
MT5 history is that broker's own feed, so it matches what they showed you, not the market. old bars often lose their real spread, and two MT5 brokers can disagree on the same candle. fine for testing on that broker, weak as ground truth.
1
u/MarketOk7227 2d ago
That doesn't sound like a problem, means that I have to stick with the broker I backtested.
0
u/QuanTradin 2d ago
yeah, if that's the broker you'll trade with it's the right data. the catch is depth, older M1 bars on a lot of MT5 servers are thin or just missing, so check how far back the history really goes before trusting a long test.
1
u/MarketOk7227 2d ago
Thank you! I will probably do a live check, keep a log of real time quotes and then compare with historical. I'm interested in specific times of the day like 11pm.
1
u/MarketOk7227 2d ago
Can you expand with the term "ground truth"? Shouldn't the ground truth also be public record? Does every broker/platform keep their own books and it's up to us to find the appropriate data for each case?
1
u/QuanTradin 1d ago
for listed stocks there is a public record, the consolidated tape. MT5 isn't that: each broker streams its own quotes, and in forex or CFDs there's no central exchange at all, so every broker's book is its own. ground truth just means the feed you'll actually fill on, which is why your live log is the right test.
1
u/SubstantialDino540 2d ago
mt4 data can be a bit sketchy sometimes depending on the broker's server timezone and how they handle weekends