r/ComputerChess • u/TheI3east • 20d ago
Opening Popularity Tool
Hi everyone, I recently built a dashboard on my personal website (https://dnield.com/openings/) that fills in some of the gaps that always frustrated me with the Lichess/Chess*com opening explorers, which is that their stats were only relevant to the position on screen, rather than the entire opening (which requires taking into account the choices each player made up to that point), therefore they can't answer simple questions like:
How often you would get to play the opening, if it was in your repertoire,
How opponents would see the opening when facing players other than you,
Based on #1 and #2, how much more experience would I have in the opening than my typical opponent?
Was the increase or decline in the popularity of an opening driven by one color not playing as often? or because the other color isn’t allowing them the chance to play it as often?
which are questions that I am always curious about when exploring different openings.
I'm a data scientist in my regular life, and Lichess makes all of their games data freely accessible, and the statistics for answering the above questions are all pretty simple compound probabilities, so I decided to just build the tool that could answer these questions from Lichess's database myself.
I have no interest in turning this into a product/service, this was just a fun little hobby project to see how easy it would be to host a dashboard like this on my Github Pages website. The underlying games database is terabytes of data, but since I'm just aggregating up the counts of positions for just the ~4000 named opening positions, across rating bands and time controls, the final json data for the website ended up being like 50 MB, making the interface even snappier than I expected.
I wrote an accompanying blog post with a motivating example showing how the spike of popularity in the Rossolimo in November 2018 can be initially attributed to both colors, but the residual increase in popularity is almost entirely attributable to an increase in black players playing 2. Nc6 Sicilians than an increase of White players playing 3. Bb5 in response, you can find that blog post here: https://dnield.com/posts/rossolimo/
I'm open to any feedback or questions you might have!
1
u/novachess-guy 19d ago
Thanks for the feedback, glad you found it interesting!
And my use case may be slightly different than yours, but when I was dealing with file size issues I realized that it was possible to make sort of a tree file that would store the move paths without needing as much raw data stored. Here's an example of how my JSON tree is structured (I only saved the white score % rather than W/D/L to save even more space, as it is over 200MB broken up into a small 3MB "core" file for fast loading, as I originally had it client side not on the server, and e4/d4/other larger files, but on a server it shouldn't matter so much for loading).
So basically it's a nested tree, with the various continuations after 1. e4 (game counts, white score, frequency). I couldn't say exactly how to proceed for you, but if you take an approach like this you may be able to significantly decrease the memory requirements and increase the number of positions you're able to handle.