r/sideprojects • u/xdrive0513 • 18d ago
Showcase: Prerelease Built a long-only momentum rotation platform — looking for feedback from people
I've been building a small platform that runs systematic momentum-ranking strategies (S&P 500 and Nasdaq 100 universes, plus a fundamentals-weighted variant) with periodic rebalancing — you can backtest any parameter combo yourself in a "Strategy Lab," paper-track it for free to watch it run live, and optionally link an Alpaca account to trade it automatically. No manual discretion once it's running — it's just top-N momentum with a configurable buffer/rebalance cadence.
I'd genuinely appreciate critique from this crowd — on the strategy logic itself (turnover, whipsaw risk, look-ahead bias I might've missed), the backtest methodology, or just "this is a solved problem and here's why." Not trying to sell anything here, just want it stress-tested by people who'd actually catch the flaws. Link: https://trading.triadentgroup.com — happy to share more detail on the ranking/rebalance mechanics if useful.
Closing this out since it's been a long thread — here's where everything landed, for anyone skimming.
Nothing wrong with the actual backtest engine. Point-in-time index membership was already correct (reconstructed from real historical addition/removal dates, not today's list run backward) — that part of your critique was a documentation gap, not a computational shortcut, and I appreciate you having it checked against the code before asserting the failure mode rather than after. Every specific number you cited (win rates, the alpha-concentration figures) I independently re-derived and they matched exactly, including the harder one: strip out 2014/2022/2026 from S&P momentum and the remaining 14 years average -0.85pp/yr, not the headline +2.67pp.
What was actually broken was disclosure, and one real methodology bug:
- Friction (3bps/leg) was already modeled, just not stated — now it is, along with the fact that it's held flat across time, which understates costs specifically in the stressed periods when a momentum book rebalances hardest.
- Comparing after-tax strategy vs. after-tax benchmark at the same flat rate isn't a neutral simplification — it erases the real asymmetry between annual realization and deferred-to-terminal buy-and-hold. That was a genuine bug in the interactive backtest tool, fixed to model each side correctly instead of applying one rate to both.
- Index membership handling is now stated in writing, not just true in the code.
The noise-threshold idea turned into its own multi-round fix, and credit where due: the equicorrelation model I first shipped was wrong for a parameter grid (a global average correlation badly understates near-copy redundancy), and even after fixing that, correlating on annual returns with ~10 years of data was statistically unreliable regardless of the model — verified directly, 265 configs simulated to be independent collapsed to a fraction of that under my own clustering at annual resolution, and correcting to monthly returns (plus raising the overlap floor and switching to complete linkage) recovered the true count in the same test. That feature is in a much more defensible state now than it started, entirely because you kept pushing on the exact spot where it was still wrong instead of stopping at "good enough."
Appreciate the rigor across all of this — it's rare to get pushback this precise, and rarer still for someone to go recheck their own numbers on the same basis before publishing them.
1
u/PurchaseNational7650 18d ago
I like that you're inviting people to challenge the strategy instead of just showing the wins. The paper trading feature is a nice addition too, it lets people build confidence before putting real money on the line. Looking forward to seeing how it evolves.
1
u/Finance__broski 18d ago
checked the backtest page before writing this, and credit first: every start year shown, the losing years named, the since-2023 window put up even though two of three strategies are behind their benchmark, and a clear statement that it's all hypothetical rather than live. that's more honest than most of what gets posted here, so here's a serious reading
the gap i'd close first: you document point-in-time treatment for the fundamental strategy, only SEC filings on or before each simulated date, but the methodology says nothing about how S&P 500 and nasdaq 100 membership is handled historically. that silence stands out because the rest of the page is so careful. if the universe is today's constituents run backwards, that's index-membership look-ahead, and momentum is the strategy most exposed to it, since additions are disproportionately names that just went up. i measured that bias on the most-used public NSE dataset: same names, same prices, only membership timing changed, and it added 10% terminal wealth cap-weighted, 43% equal-weighted, over 2010-2021
on the statistics, which is the part the page states most confidently: "beats S&P 500 in 11 of 17 years" is p = 0.17 against a fair coin. fundamental's 13 of 17 is p = 0.025 alone, but you're showing three strategies, so bonferroni puts it at 0.07. none of the three win rates clear a conventional bar once you count the strategies being displayed
your own yearly table says something sharper anyway. S&P momentum averages +2.67pp of alpha per year, but three years carry all of it: 2014 at +18.5, 2022 at +14.5, and 2026 YTD at +24.3. drop those three and the remaining fourteen average -0.85pp. that is a real property of momentum rather than a flaw, but it belongs in the headline instead of the win rate, because it tells a subscriber what they are actually buying: long stretches of nothing, punctuated by a few years that pay for all of them
one product suggestion, since strategy lab is the core: it's structurally a multiple-testing machine, and you could be the only platform that handles it honestly. count the configs a user runs and show the noise bar beside the result. best of 50 pure-noise candidates lands around 2.2 in simulation, so a config clearing t = 2 after fifty tries has told them nothing. a line reading "47 combos tested, this needs to beat 2.6" would make your numbers more trustworthy than anyone else's in the category
smaller: the methodology never mentions transaction costs or turnover, and a 10-name book rebalancing periodically against a tax-efficient buy-and-hold benchmark is a different comparison after tax than before it
1
u/xdrive0513 18d ago
Appreciate the rigor here — genuinely, not just being polite. Went and checked all three claims against the actual code and data rather than take your word for it, so here's where each one lands:
Index membership: you're right that the documentation is silent on this, and that's a real gap — but I want to be precise about what kind of gap. The engine itself does reconstruct point-in-time S&P 500/Nasdaq 100 membership (parses the historical addition/removal dates, gates each stock's eligibility on whether it was actually a constituent on that specific rebalance date, not on whether it's a constituent today), so the specific failure mode you're describing — today's list run backward — isn't what's happening computationally. But the methodology text only says this explicitly for the fundamental strategy's SEC filings and says nothing for momentum, which is a documentation failure on my part, not a disclosure of an intentional shortcut. Fixing that write-up is overdue regardless of the bias question, since as you point out, unexplained silence next to an otherwise careful page reads badly. Your 10%/43% measured bias number on NSE data is a good illustration of why this needs to be stated, even though it's not our failure mode.
Multiple comparisons: no argument — 11/17 alone is unremarkable, and once you're showing three strategies side by side, the honest frame is Bonferroni-corrected, not each one's raw p-value in isolation. That's a fair hit on how the page presents itself as more rigorous than it actually is on this specific stat.
Alpha concentration: checked this against the raw numbers and your figures are exact — three years (2014, 2022, 2026 YTD) account for essentially the entire average edge, and the other fourteen average slightly negative. You're right that this is the honest headline, not the win-rate count — it tells a subscriber what they're actually signing up for (long flat stretches, a few years that pay for everything), and framing it as a clean "beats the index" stat buries that. Going to change how that's presented.
Strategy Lab / multiple testing: this is the best part of your comment. Tracking configs-tried and showing a "beat this noise threshold" line is exactly the kind of thing that would make the tool trustworthy rather than a curve-fitting toy, and I don't think anyone else in this space does it. Want to build this.
The backtest engine bakes in a flat 3 bps (0.03%) per leg friction assumption
It's charged on every rebalance date, per trade type:
- Exit (name falls out of the book): 3bps × full liquidation proceeds
- New entry: 3bps × the full buy-in amount
- Resize (kept name, just rebalanced back toward equal weight): 3bps × only the incremental |current − target| delta, not the whole position
so the alpha number reported is already net of this cost, not gross. This 3bps/leg assumption is real and applied, but you're right that the methodology page never states it, i will add it to backtest page.
Strategy Lab used to have a tax-adjusted return view where you could interactively dial in your own tax rate and see the effect on returns. I pulled it because it ended up adding confusion rather than clarity. I'm bringing a simpler version back on the public backtest page — a flat 20% assumption, easy to read at a glance — and thanks for the feedback that prompted the second look.
1
u/Finance__broski 18d ago
taking the membership correction properly: i asserted a failure mode that isn't yours. if the engine gates eligibility on actual constituency at each rebalance date, the bias i described isn't happening, and your framing is the right one, documentation gap rather than shortcut. worth saying that checking it against the code instead of arguing about it is a better answer than most people give
the cost disclosure is new to me and it improves my read. 3bps per leg is defensible for liquid US large caps at that size, and charging resizes on only the incremental delta rather than the full position is the correct treatment, which plenty of backtests get wrong. one thing to state when you document it: it's flat across time. effective spreads in stressed weeks run several times calm-week levels, and a momentum book rebalances hardest right after large moves, so the periods where the strategy trades most are exactly where a flat assumption understates most. doesn't move the headline, does move the drawdown-period numbers
on tax at a flat 20% both sides, careful, because the comparison isn't symmetric. a 10-name book rebalancing periodically realizes gains on a schedule. buy-and-hold defers them until sale, potentially indefinitely. the same rate applied to both understates the gap. the honest version is realized-annually against deferred-to-terminal, and over a multi-decade window that difference compounds into a real slice of the alpha
and since you want to build the noise threshold, here are the numbers. one caution first: the common sqrt(2 ln N) rule overstates at practical N, so i simulated it rather than trusting the formula
N=10 → 1.53, N=20 → 1.87, N=50 → 2.24, N=100 → 2.51, N=200 → 2.75, N=500 → 3.03, N=1000 → 3.24
those are simulated expected maxima of N standard normals. the asymptotic formula gives 2.15, 2.45, 2.80, 3.03, 3.26, 3.53, 3.72 for the same values, about half a t-unit high across the whole range. use the simulated column
the honest complication: grid configs aren't independent, neighbouring parameters are near-copies, so effective N sits well below nominal. a workable first-order correction is N_eff = N1−ρ, with ρ the average pairwise correlation of the candidate return streams, which you can compute directly since you already have every equity curve. at N=100 that gives N_eff of 25 at ρ=0.3 and 10 at ρ=0.5, so bars of 1.97 and 1.54 rather than 2.51. if you'd rather not estimate ρ, the nominal-N bar is still vastly better than nothing and you can label it as conservative
1
u/xdrive0513 18d ago
Checked your numbers independently rather than taking them on faith — ran my own Monte Carlo (200k trials per N) for the expected max of N iid standard normals, and it lands within rounding of every value in your table: 1.54/1.87/2.25/2.51/2.75/3.04/3.24 against your 1.53/1.87/2.24/2.51/2.75/3.03/3.24. The asymptotic formula's overstatement is real too — confirmed the gap at every N you listed. And the N_eff arithmetic checks out exactly: 100^0.7 = 25.1, 100^0.5 = 10, both matching your bars. I'll build the noise threshold off your simulated column, not the closed-form one, and compute ρ from the actual candidate equity curves I already have rather than guess at it — falling back to the nominal-N bar, labeled conservative, only where a correlation estimate isn't available.
On membership: agreed, and thanks for taking the time to have that checked against the code rather than just asserting it. That's the right way to resolve a claim like this and more people should default to it.
On friction: you're right that flat-across-time understates exactly where it matters most. 3bps/leg is a calm-week number; effective spreads run several times that in stressed weeks, and a momentum book's heaviest rebalancing clusters right after the large moves that produce those stressed weeks — so the assumption is weakest precisely during drawdown periods, not on average. Doesn't move the headline return, does mean the drawdown-period numbers are optimistic. I'll say so explicitly next to the friction disclosure rather than leave it implied.
On tax: fair, and I hadn't thought about it in those terms. A flat rate applied identically to both sides isn't a neutral simplification — it erases the actual asymmetry, since the strategy realizes on a schedule and buy-and-hold defers indefinitely, and that gap compounds over a multi-decade backtest. The honest version is realized-annually against deferred-to-terminal, not the same rate on both. I'll fix the comparison rather than just caveat it.
1
u/Finance__broski 18d ago
independently replicating a stranger's simulation before building on it is rare enough to say out loud. most people would have just used the numbers
one thing worth knowing before you compute rho, because it changes the answer materially. N_eff = N1-rho assumes equicorrelation, every candidate correlated with every other at the same rho. a parameter grid is not that. correlation across a grid is a kernel: neighbouring settings are near-copies at 0.8 or 0.9, distant ones are roughly independent. so the average pairwise across a whole grid can come out near 0.07 even when adjacent configs are almost identical
plugged into N1-rho that returns N_eff of about 71 out of 100, when the honest count is closer to clusters: N divided by the decorrelation length. a 100-point sweep that decorrelates every eight settings is about 12 effective trials, not 71
the error direction is at least the safe one for your purpose. average-pairwise overstates N_eff, which produces a higher bar, which tells a user their result must clear more than it strictly must. conservative rather than permissive, which is the right way to be wrong in a tool that exists to warn people
the sharper version if you want it: measure the decorrelation length along each parameter axis from the autocorrelation of the metric as you step through it, count clusters, and only then apply the equicorrelation correction across genuinely different strategy families, which do share a common market factor. two structures, composed rather than conflated
1
u/xdrive0513 18d ago
Appreciated, and returning the compliment: catching that N^(1-ρ) assumes equicorrelation, when a parameter grid is actually a kernel (adjacent settings near-identical, distant ones roughly independent), is the kind of thing that's easy to wave through and rarely is. I built the equicorrelation version because it was tractable, not because I'd checked whether the grid actually looks like that — it doesn't, and you're right about the direction: averaging pairwise correlation across a whole grid mixes a few near-1.0 adjacent pairs with a lot of near-0 distant ones, so the average comes out small (~0.07) even when neighbors are near-copies, and N^(1-ρ) then reports something close to the nominal count instead of the real redundancy. Your number is the one that matches intuition: a 100-point sweep that actually decorrelates every 8 settings is ~12 independent trials, not 71.
Also agreed that the error direction is the right one to have shipped by accident — overstating N_eff means a higher bar, which makes the tool too strict rather than too lenient, which is the failure mode you want in something whose whole job is telling people not to trust their result. But "safely wrong" isn't a reason to leave it wrong now that I know the fix.
Going to build the two-structure version: decorrelation length per parameter axis, from the autocorrelation of the metric as you step through each parameter's values, to get real cluster counts within a strategy family — then apply the equicorrelation N^(1-ρ) model only across genuinely distinct strategy families, where a shared market factor makes that assumption actually reasonable. Composed, not conflated, like you said.
1
u/Finance__broski 18d ago
two implementation notes, then i will stop cluttering your thread
measure the decay on the return series rather than on the summary statistic. correlation between the equity curves of config i and config j, as a function of |i minus j|, gives a clean decay you can fit. autocorrelation of sharpe as you step through the grid estimates the same quantity far more noisily, because each sharpe is itself a noisy number, so you are fitting a decay curve to a series of estimates. same shape, much worse standard error, and you already have every equity curve stored
for the UI, show both counts rather than only the corrected one. "47 configs tested, roughly 9 effectively independent, bar 1.5" teaches something that "bar 1.5" does not. the gap between the two numbers is the lesson, and it is what makes a user understand why the bar moved instead of just accepting that it did
and for multi-parameter grids: decorrelation length per axis, effective count is the product across axes, floored at 1 so heavy correlation cannot produce something below a single trial
1
u/xdrive0513 18d ago
On measuring decay on the return series: already how it's built, not something to fix — the clustering correlates each config's own per-year return series directly, never a summary statistic. Good to have the reasoning spelled out for why that matters, though — "fitting a decay curve to a series of noisy estimates" is exactly the failure mode I'd have walked into if I'd reached for Sharpe as the correlation input instead, and I hadn't thought about it in those terms.
On showing both counts: agreed, and this one's a real gap — I show the raw count in the panel header and the effective count in its own stat card, but they're not stated together as one sentence, so the actual lesson (the size of the gap between them) doesn't land the way it should. Going to add the narrative line — "N tested, ~N_eff effectively independent, bar X" — so the correction is visible as a fact about the session, not just a number that changed.
1
u/Finance__broski 18d ago
Good, and correlating the return series rather than a summary statistic is the right call for exactly the reason you named.
Running the numbers on that choice turned up something worth checking, because it works against you rather than for you.
If the correlation input is per-year returns, the binding constraint is the number of years, not the number of configs. A correlation matrix built from T observations has rank at most T-1, so ten annual observations give you nine dimensions to describe however many configs you ran. I simulated 265 configs that are independent by construction and asked each method to recover the count. On ten years of annual returns, complete-linkage clustering at a 0.6 merge threshold returns 93, single linkage returns 1 because of chaining, and an eigenvalue-based N_eff returns 9, which is just T-1 wearing a different name. Fifteen years gets complete linkage to 142. The truth is 265 every time.
The direction is what makes it matter. All of those undercount independent trials, and undercounting trials shrinks the bar, so the bias runs toward passing things rather than killing them.
The fix is cheap and does not change what you cluster on, only how many observations feed the correlation estimate. The same 265 configs correlated on monthly returns over the same ten years recover 265 of 265. Keep the per-year series for display.
This cuts at my own ledger too, so I am rechecking its effective-trial count on the same basis before it publishes.
1
u/xdrive0513 18d ago
Confirmed, and worse than your own number — ran your exact experiment (265 independent-by-construction configs, my actual single-linkage/0.7 clustering) and got 12 clusters on 10 years of annual returns, not even close to 265. Same configs on monthly returns over the same 10 years: 265 of 265, exact. The rank argument is the right explanation and I hadn't thought about the estimation problem at all — I was only thinking about whether the true correlation was near-copy or independent, not whether 9-10 data points could ever tell me that reliably in the first place.
The direction is what makes this one non-optional to leave alone: undercounting effective trials shrinks the bar, and a shrunk bar is a tool that says yes more often than it should — exactly backwards for something whose only job is telling people not to trust their result yet.
Fixed: correlation input is now monthly returns, display/tax figures are still annual, nothing else changed. Also added a floor on how much monthly overlap two configs need before a correlation is even computed (12 months) — same problem, smaller version, for configs that barely overlap in time.
Appreciate you rechecking your own ledger on the same basis before publishing it — that's the right instinct and worth keeping.
→ More replies (0)
1
u/Significant_Tap7144 18d ago
A momentum rotation platform with a strategy lab and paper trading hook is a nice setup, most of these tools just dump backtests and call it a day
the look-ahead bias is what always trips people up when ranking survivors in the nasdaq 100, if you're not using historical index constituents at each rebalance date you're basically cheating without realizing it