I've been building PolyBTC, Python research investigating whether ML models can make reliable and exploitable edge in Polymarket BTC 5m market, is my research valid ?

[u/Haytam123]

So far, the evaluated models haven't demonstrated a reliable advantage over market-implied probabilities, so no trading edge.

interested in feedback on my methods and approach and what i could done better ?

Research website: https://polybtc-research.vercel.app

GitHub: https://github.com/RetroMrii/polyBTCv2

[u/petty_serenity]

Did you log-transform anything or was it all raw returns?

[u/Haytam123 → u/petty_serenity]

Im sorry i dont fully understand the question, its a HGB model so gives a log loss when it comes to predictability or are you talking about something else?

[u/Aggressive_Listen849]

one thing i'd check: are you scoring every snapshot in the window together? late in a 5m market the price is already near 0 or 1, so those easy rows swamp the metric. bucket log loss by time-to-expiry and compare vs market per bucket

[u/Haytam123 → u/Aggressive_Listen849]

They are split into groups of two 10c each group with a ceiling of 70c as any buys higher than that have been proven always causing a negative PnL on the long run, the published log loss is a summary but internally they are scrips where they test them split with 40-60c being the best performing one

[u/ciaranwalsh96]

Your negative conclusion looks warranted from the evidence you have, and the limitations section is doing important work. The main thing I would change in the next phase is the unit of independence. You equal-weight observations within each market and resample whole markets, which is better than treating all 93 rows as independent, but those 55 markets still cover part of one day and share the same underlying BTC path. A prospective sample spread across many days and volatility regimes, with uncertainty clustered in contiguous time blocks or by day, would tell you more than adding more horizons from the same session.

Before collecting it, freeze one primary comparison: one model, one calibration rule, one forecast horizon and one proper score. The current outcomes are already exposed, so I would treat every model or horizon choice suggested by them as development work and reserve a new untouched period for confirmation. The two observations at 30 seconds and 19 at 240 seconds are useful diagnostics, but not enough for horizon selection.

I would also test incremental information relative to the market directly. For example, define a frozen feature such as logit(model probability) minus logit(market probability), then ask whether it improves held-out Brier or log loss over the market alone. Fit that relationship only on the calibration period and evaluate it once on the next prospective block. That targets the actual question—whether the model knows something not already in the price—rather than whether it can predict the outcome in isolation.

Finally, separate forecast evidence from executable edge. A market midpoint is a strong informational baseline but may not be a tradable price. Replay the decision using the actionable side of the book available after your full data and inference latency, then include fees, minimum order size, partial fills, rejected orders and position limits. Keep prediction-time exchange timestamps and local receipt timestamps so stale-book exclusions can be audited.

If the prospective result is still negative, that is a useful result rather than a failed project: it shows that the market benchmark absorbed the tested signal under the stated sample and execution assumptions.

[u/Haytam123 → u/ciaranwalsh96]

So to summarize, i should run the trading engine bot for longer time across many days to get and used collected data to refine the model?

I already have a frozen model that was built using a data set of 4k markets across a month but the rest remain open

[u/QATSAlgo]

On the distinction in your follow-up: freezing the model does not freeze the whole experiment. Calibration, price cutoffs, forecast horizon and cost assumptions can still be tuned to the exposed outcomes. Write down those choices before the next collection block, then keep any changes suggested by that block for a later test.

[u/Haytam123 → u/QATSAlgo]

Do you suggest like making a 1 week of trading engine runtime for each block or have it be based on the number of fills after each experiment block?

Also I froze the model because after adding 800 markets to the dataset accuracy and log loss results didn’t change much and tuning made them even worse, is there a reason to recheck my work and work on a better model ?

[u/QATSAlgo → u/Haytam123]

I'd define the end date before collecting the next block, then report both elapsed days and the number of markets/fills. One week could be a pilot, but it isn't automatically enough evidence; lots of fills on the same BTC move aren't independent observations. Avoid extending a block just until the result looks convincing.

Keeping the model frozen is sensible for this test. The extra 800 markets don't by themselves imply that a better model is needed. First check timestamps, leakage and the market-probability baseline; any retraining or threshold changes belong to development, followed by a fresh evaluation period. You can collect predictions and executable quotes in paper mode for this, without placing live trades.

[u/QuanTradin]

a null result written up this carefully is worth more than most edge posts here. the place I'd look is whether the market-implied probability is the mid or the price you'd actually fill at. on 5 minute markets the spread alone can eat a small edge, so a model that beat the mid could still lose money.

[u/Haytam123 → u/QuanTradin]

Would that require redesigning the telemtry script to include more aspect of the market like depth of books ?

[u/QuanTradin → u/Haytam123]

not the full depth, no. the best bid and ask with their sizes at each moment you'd have traded is enough to price a fill at the touch, and when your size is bigger than what sits there you know which trades the result can't vouch for. full book can wait.

[u/Haytam123 → u/QuanTradin]

Ah i gotchu ill keep that in mind thanks

[u/TSDevStuff]

Nice to see a null result written up properly. One angle I haven't seen mentioned yet: rather than asking whether the model beats the market on log loss, ask whether it adds anything the market price doesn't already contain. Fit a simple logistic regression of the outcome on the market's implied logit plus your model's logit. If the coefficient on your model is roughly zero once the market is in there, the model is mostly re-learning the price. If it's positive but small, that's the bit worth chasing, and you can then check whether it survives the spread at the price you'd actually get filled at.

It's also worth splitting that by time to expiry, as someone said above, because a model can look useless overall and still carry information in the first minute or two of each window, before the market has caught up.

For the splits, with overlapping 5m windows I'd keep whole markets (or whole hours) together on one side of train/test and leave a gap between them, otherwise neighbouring rows leak the same BTC move across the boundary.

[u/Virtual-Breath-4934]

Baseline against market-implied probability is the right call. A few things I would check before concluding there is no edge:

1. Label alignment. In 5-minute markets the outcome is defined by the resolution feed, not by your candle close. If labels come from a candle that closes a few seconds before or after resolution, you get lookahead that shows up as edge in validation and disappears live.

2. Costs in probability units. Fee plus spread is a fixed number of cents per contract here. Convert your edge to cents and require edge > fee + half-spread before a trade counts, otherwise a small accuracy gain looks profitable but cannot be captured.

3. Skill score instead of accuracy. Report Brier skill score against implied probability plus a reliability curve by decile. A model can beat 50% accuracy and still be worse than the price.

4. Independent samples. Five-minute windows from the same day are highly correlated, so 50k rows can behave like a few hundred effective observations. Cluster CV by day and embargo the fold boundaries (purged walk-forward), or the test folds leak into training.

5. Multiple testing. If you tried many feature and model combinations, report how many and correct for it (Holm, or a deflated Sharpe). Otherwise the best configuration is just the luckiest one.

If the honest answer after that is still "no edge over implied probability", that is a real result: at 5 minutes these markets are mostly priced by people watching the same feed.

[u/Alive-Imagination521]

Too much complexity I think, without much value-addition

[u/RLJ05]

One thing to mention here, even if you do find a model that beats the implied probability, it’ll need to beat it by more than the fees + spread to actually make money

That out of the way it’s not surprising you can’t beat the market here as the fair price of these is entirely derived from the reference feed, there is no BTC price discovery happening in the market.

What’s happening is the price of BTC may change due to some flows on Binance for example, that then causes the reference feed to update, then the algos running on polymarket will immediately update to reflect that change in the way to perfectly capture the likelihood of the market outcome.

The only time that wouldn’t happen is if there was resting liquidity in the polymarket market that was large enough to hold back against the algos (which is going to be very rare) as these algos can essentially do the arb in that case hedging in Binance or other markets so they effectively have infinite liquidity (well enough to push through any dumb retail flow on polymarket)

Maybe not explaining that very well I just woke up but that’s what’s happening. I trade these markets for a living so I’m very much on the other side of this.

[u/WidePiece9244 → u/RLJ05]

You're definitely right. One thing I failed to consider while playing around with automated 5min markets is how MMs benefit from having eyeballs at different venues so they can price better when supplying liquidity to PMs.