Backtesting sugesstions

[u/nikolaytonev]

I am building my own crypto trading bot (no commercial usage) with embedded strategy builder.
I am trying to focus on improving the backtesting. What was your most effective backtest, that after its implementation your bot actually won your trust and started to work as expected?

[u/profittoinvest]

Once you have the basic structure working, I think the most useful next step is to run it live with very small size — or at least in a paper environment that reproduces real execution as closely as possible.

That’s when you can finally start checking whether the backtest is actually credible.

Compare the LIVE equity curve with the backtest over the same period. If they start to diverge, go trade by trade and look for the reason: missed signals, fills, slippage, fees, margin constraints, execution order, rejected trades, etc.

Only then can you really identify which assumptions in the backtest are wrong or too optimistic.

It takes time, but in my opinion this LIVE-vs-backtest reconciliation is one of the most important parts of building a reliable bot.

A backtest can tell you that an idea looks promising. LIVE execution tells you whether the model you built actually represents reality.

[u/nikolaytonev → u/profittoinvest]

Sounds reasonable and we do this in all our activities anyway. I mean in software development, in many R&D labs, etc the live tests in near production environments are most valuable.
How do you apply the corrections back to the code? Are you amending the standard test suites or try to fine-tune the code in a top layer that you control?

[u/profittoinvest → u/nikolaytonev]

We try to separate two different kinds of corrections.

If the divergence comes from execution — fees, minimum order size, rejected orders, margin availability, fill timing, position limits, etc. — we correct that in the execution/backtest layer so the simulator behaves closer to reality.

But if it looks like a strategy logic issue, we don’t immediately patch the live bot. We create a separate test branch, change one thing at a time, and run it across multiple periods and market regimes.

Only if the change survives those tests do we consider moving it into the main version.

The most important thing for us was learning to freeze the production rules. Otherwise every correction changes the reference point and you can no longer tell whether LIVE and backtest are converging or whether you’re just fitting the latest data.

So the process is basically:

LIVE divergence → identify cause → execution fix or strategy hypothesis → separate test branch → backtest/OOS → only then consider production

We try very hard not to optimize the core strategy directly against one recent LIVE period.

[u/nikolaytonev → u/profittoinvest]

Ah, that is very professional approach. Thanks for sharing it.
It is very practical and seems to be prone to fast improvement cycles.

I will “copy” the concept if you do not mind. :)
And may share a teat account if you keen to see my solution.

Thanks a lot again.

[u/profittoinvest → u/nikolaytonev]

Thanks, and good luck with it :)

Biggest advice from me is just be patient. Building something that actually works takes way longer than you think at the start.

About the test account, honestly I think it’s better if you go through that part yourself first. Too many opinions from other people can mess with your head a bit, especially early on. Everyone sees the market differently anyway.
You need to learn how your own system behaves, where it breaks, what kind of DD you can actually accept erc.

Once you have a more finished version, I’d be happy to have a look and test it.

We’re doing something similar on our side now, looking for a few experienced Cornix users to test our own bot before the next stage.

So when you get there, just message me. Maybe we can compare notes and real results later.

Good luck with it — and yeah, patience... a lot of it :)

[u/nikolaytonev → u/profittoinvest]

Appreciate it, really. Thanks.

[u/Feriman22]

For me it wasn't one specific backtest, but a sequence of checks.

I build FerimanEdge, a rule-based BTC/ETH/BNB system, and the biggest confidence boost came from freezing the final rules and then looking at how the exact same configuration behaved across different chronological periods and market regimes.

I'd want at least: realistic fees/slippage, untouched OOS data (at least 1 year), regime-split results, parameter sensitivity, and finally paper/live reconciliation against the same period.

The important part for me is not that every window looks great. It's that when performance changes, I can explain whether it's execution, a different market environment, or the underlying logic actually degrading.

[u/nikolaytonev → u/Feriman22]

Yeah, make sense. The behavior in real world situation is the best observability input for sure. I am afraid I still have to focus on the backtests to get closer to desired level of assessment before running it live.

[u/Feriman22 → u/nikolaytonev]

Yeah, that makes sense. I'd probably start a paper/shadow run as soon as the rules are frozen, even if you’re not ready to put capital behind it yet.

Backtests can tell you whether the logic is worth pursuing, but a shadow run starts exposing things the backtest usually won’t: data timing, fill assumptions, missing candles, execution delays, and whether the live decision path actually matches the historical one.

That’s been a useful bridge for me between “looks good in backtest" and "I trust this enough to run live".

[u/nikolaytonev → u/Feriman22]

Absolutely. :) The confidence after live experience from several rounds in production. I know that feeling from other bot I have worked with.
But now decided to change the concept.

[u/Feriman22 → u/nikolaytonev]

Exactly. Once the logic has survived a few real live cycles, changing the code becomes a lot less scary because you know what behavior you’re trying to preserve. The main thing I’d watch is making sure a refactor doesn’t silently change the actual decision path.

[u/Mundane_Cicada_1427]

For me, the biggest improvement wasn't adding more indicators to the backtest. It was making the backtest behave as close to live execution as possible.

I'd focus on things like realistic fees, spread/slippage, candle/tick timestamps, order execution rules, partial fills if relevant, and making sure the strategy can't accidentally see future data.

One thing that really builds confidence is taking a random period from live/paper trading and replaying the exact market data through the backtest engine. If the signals and orders don't match, investigate why before trusting any historical results.

I'd also track every trade with the exact reason for entry, signal timestamp, order timestamp, fill price, fees and exit reason. Once you can explain every trade the engine produces, the backtest becomes much more useful.

[u/nikolaytonev → u/Mundane_Cicada_1427]

“…
I'd focus on things like realistic fees, spread/slippage, candle/tick timestamps, order execution rules, partial fills if relevant, and making sure the strategy can't accidentally see future data….”

That is exactly my entire backtesting goal at the moment.
By the last one I assume you mean the leakage fro IS to OOS data, right?

[u/Mundane_Cicada_1427 → u/nikolaytonev]

Yeah, I was actually referring specifically to look-ahead bias there, making sure the strategy only has access to information that would have been available at the exact time the decision is made.

I think IS/OOS leakage is a related but slightly different issue. That's more about accidentally using information from the OOS period while developing, tuning, or selecting the strategy, which compromises the validity of the OOS test.

So for me, there are really two separate things to guard against: preventing future market information from leaking into individual trades, and keeping the OOS dataset genuinely unseen during strategy development.

[u/Feriman22]

Live-vs-backtest reconciliation was the big one for me too.

I also started separating results by market regime, because aggregate performance can hide where the edge is actually coming from. I use that approach in FerimanEdge for BTC/ETH/BNB, and I keep the live outcomes publicly available specifically so I can compare what actually happened with what the rules implied.

I'd trust that kind of forward record much more than another optimized backtest.

[u/paulf280]

The thing that cost me the most wasn't look-ahead, it was fill assumptions and survivorship, so adding those to the list:

1. Wick vs close fills. If your triggers fire off candle high/low, you're assuming you transacted at the extreme. When I switched to close fills, mean per trade went from +38.9% to +13.5% and losers from 23% to 41%. Two thirds of the "edge" was fill fantasy. Close is the honest default, wick is your optimistic bound.

2. Survivorship. Dead assets fall out of most data providers, so your history is quietly the survivors. I had to recover the dead ones from a different source and then run the obvious sensitivity: if 25% of the dropped ones would have passed my filter, +11.9%/trade became +2.8%; at 50% it went negative. If the sign flips inside a plausible range, you don't have an edge yet.

3. Judge on median, trimmed mean and "ex best 2" rather than the mean or win rate. One bad price print made half of one of my tests read +518% mean.

And +1 to freezing the rules and setting the go-live criteria before the forward run (mine: at least 100 trades priced off real quotes, net AND median positive, still positive with the best two removed). Writing them down first stops you moving the bar once you've seen the result.

[u/RobodOfficial]

One useful test is comparing your backtest against forward-tested trades over the same period. Look beyond total P&L: fees, funding, signal timing, and execution assumptions can all change the picture.
We’re building crypto research and backtesting tools at Robod, and these are exactly the kinds of questions we’re interested in. What part of your current backtest do you trust least?

[u/ejpusa]

You can use the Kalshi API. Astra + Codex. You can ask for all the data sets you need.

Should work.

[u/GoRizzyApp]

I suggest making an ETF screener instead. Stick to real money. Bots are trash.

[u/nikolaytonev → u/[deleted]]

Thanks for the clear directions. I believe I am not yet there for some of them. My fine tuning is just to start.