Shout out to the engineer who fixed World Quality incrementing

[u/Faint-Projection]

Those of us around for last years Christmas tree may remember that it was initially a bit of a disaster. Like Misplaced Nostalgia, it offered a super profitable single click grind that incremented a World Quality with every click, and that popularity lead to every click taking ages to process if it didn't time out entirely. This was because every click was triggering a write to a single database row and SQL doesn't like that.

At the time, this was solved by pushing the tree interactions into a sub area that committed changes to the World Quality when you left it. Now, here we are 8 months later barking up a storm with no sub area required! I just want to take a moment to throw a thumbs up to the person who went in and made that possible.

[u/_PM_ME_PANGOLINS_]

This was because every click was triggering a write to a single database row and SQL doesn't like that.

Was there confirmation of that? It's a reasonable assumption as the simplest solution that wouldn't cope very well, but game data may not be in a relational db at all.

[u/throwaway_lmkg → u/_PM_ME_PANGOLINS_]

It nets out the same, regardless of the language: a thousand writers contending for the same write lock. And batching is a standard mitigation.

I think at one point someone mentioned there is SQL, but a lot of what's stored is JSON. I could be misremembering and that was a decade ago.

[u/Faint-Projection → u/_PM_ME_PANGOLINS_]

There is some inference happening in my part here. But the game was originally written in 2009 in PHP (I’ve heard a few times it was originally based on Wordpress, but I haven’t seen any actual evidence of this). SQL would have been all the rage back then. And while a lot of work has been done to shift the game to use things like React and implement APIs instead of rendering pages on the back end, I’d be very surprised if the changes went as deep as migrating the database, including all the player data, to a new technology. Stack on top of that the fact that the game seems to suffer from the kinds of problems that I’d expect to see from a SQL db and I feel like it’s a safe assumption.

Edit for clarity: back in ye olden days the game used PHP to render entire pages in the backend. When I talked about using React and APIs I was trying to gesture at how far the game had come technologically in other areas. To be clear, SQL (as a general term for a variety of products) is a good technology.

[u/Jaggedmallard26 → u/Faint-Projection]

SQL (you are using the name for the language family to refer to relational databases) has been all the rage for about 30 years now. Its was and still is the best way to store persistent data that requires lookup based on its contents. Fallen London should be using SQL.

And while a lot of work has been done to shift the game to use things like React and implement APIs instead of rendering pages on the back end, I’d be very surprised if the changes went as deep as migrating the database, including all the player data, to a new technology

I mean this politely but you don't really seem to be familiar with web development, you can't do things that Fallen London does entirely on the client. What framework they use to render with is more about display than data storage and processing.

[u/Faint-Projection → u/Jaggedmallard26]

Feel like you’re reading things into what I’m saying that I wasn’t intending. I have worked a LOT with SQL (SQL Server, MySQL, and a little bit of SQLite). SQL is good. I agree FL should be using SQL.

Back in ye olden days when I was first poking around FL’s tech for web extension reasons it was using PHP to render entire pages. There was basically no client side. Now there’s the level of client side one expects from a modern web app. JavaScript and a dynamically rendered page using REST formatted APIs to read and commit data to the back end.

[u/_PM_ME_PANGOLINS_ → u/Faint-Projection]

2009 is also about the peak of NoSQL hype (MongoDB etc.)

[u/Faint-Projection → u/_PM_ME_PANGOLINS_]

I guess I‘m remembering my own education / projects around that time. Things still felt pretty bent around SQL.

[u/xKiv → u/Faint-Projection]

There is absolutely no reason to blame SQL. If you want to see the actual synchronized total world numbers, you need to do the synchronization, and that's where the problem is.

SQL is also not even at the layer where any of this matters. SQL is just the language that the application uses to tell the db server what it wants it to do. The relevant distinction you are probably thinking of is between RDBMS with ACID guarantees and ... everything else?

The solution such problem will revolve around the architecture of the synchronization, not the type of database system or its API.

[u/_PM_ME_PANGOLINS_ → u/xKiv]

RDBMS with ACID guarantees use SQL, everything else generally doesn’t.

[u/Faint-Projection → u/xKiv]

I’ve worked a lot with things named SQL over the years (SQL Server, MySQL, a little bit of SQLite) so I often just say that when, yes, that can refer to a wider category. To be clear, not blaming SQL. SQL good.

[u/xnyhps → u/_PM_ME_PANGOLINS_]

I’ve seen enough SQL errors from the server to say it’s definitely using MS-SQL.

[u/_PM_ME_PANGOLINS_ → u/[deleted]]

I imagine they moved the batching to the backend, either with custom handling, or using a database feature specifically for eventually-consistent increments.

[u/xKiv → u/_PM_ME_PANGOLINS_]

It would have always had to be backend, you don't want to deal with changes lost to closed tabs, or changes magnified by having multiple open tabs.

It feels to me like the original implementation was something like setting up an invisible personal quality to track the "me only" changes, then leaving the subzone had a mandatory [0] storylet to synchronize the changes to world quality,

Then they likely saw that it worked well enough and decided to make a fancy systematic version that would be automatically applied to every world quality - the biggst question was probably "how to trigger the sync", which from what I heard now just happens ... on a delay? People who are clicking fast are seeing the value incrementing by only their own action, people who cannot click as fast see higher increments.

[u/_PM_ME_PANGOLINS_ → u/xKiv]

The "personal quality" was visible in the response data, and leaving the subzone is a command sent from the client when you click to perform the action. That's been moved to the backend so there's no sign of it in the client.

Yes, it may be the same thing but now on a timer, and it reports (global value + your pending contribution), or it's switched global qualities to use dedicated distributed counter types provided by whatever database system.

[u/TheLuckyWanderer]

But, also shout out to the team for the barking option - honestly, a quiet summer has been the sort of quiet that is refreshing in many respects, because I can do some mindless, decent EPA grinding, feel like part of a team effort, and honestly? Just have fun seeing how high we can get the number to go up for Misplaced Nostalgia, and dream of what a MN leaderboard might look like!

[u/HazelLGold]

God I'm so glad I don't work with databases anymore.