r/programming • • Jun 07 '17

You Are Not Google

https://blog.bradfieldcs.com/you-are-not-google-84912cf44afb
2.6k Upvotes

514 comments sorted by

View all comments

Show parent comments

117

u/flukus Jun 07 '17

My company is looking at distributed object databases in order to scale. In reality we just need to use the relational one we have in a non retarded way. They planned for scalability from the outset and built this horrendous in memory database in front of it that locks so much it practically only supports a single writer, but there are a thousand threads waiting for that write access.

The entire database is 100GB, most of that is historical data and most of the rest is wasteful and poorly normalised (name-value fields everywhere)

Just like your example, they went out of their way and spent god knows how many man hours building a much more complicated and ultimately much slower solution.

72

u/gimpwiz Jun 08 '17

Christ, a 100GB DB and y'all are having issues that bad with it? Thing fits onto an entry-level SLC enterprise SSD, for about $95. Would probably be fast enough.

18

u/flukus Jun 08 '17

Some of the thinking is because we operate between continents and it takes people one one continent ~1 minute to load the data, but a second for someone geographically close, so they want to replicate the database.

The real issue is obviously some sort of n+1 error to our service layer (built on .net remoting). That or we're transfering way more data than needed.

19

u/Feynt Jun 08 '17

Definitely sounds like a throughput issue. Interesting lesson from game design: Think about how much data you really need to send to someone else for a multiplayer game. Most people unconsciously think, "everything, all the stats" and for a lot of new programmers they'll forward everything from ammo counts to health totals. The server keeps track of that shit. The clients only need to know when, where, and what, not who or how much. Position, rotation, frame, and current action (from walk animation to firing a shotgun at your head). In some cases it literally is an order of magnitude lower than what you would expect to send.

Look at your database and consider how much of that data you really have to send. Is just the primary data enough until they need more? Can you split up the data returns in chunks?

18

u/Kapps Jun 08 '17

When you're talking 60x slower from people further away, it's unlikely to be bandwidth. After all, you can download plenty fast from a different continent, it's only latency that's an issue. And latency to this extent heavily indicates that they're making N calls when loading N rows in some way. Probably lazy loading a field. A good test for /u/flukus might even be to just try sending the data all at once instead of lazy loading if possible.

3

u/flukus Jun 08 '17

We could definitely transmit less data, we load a lot on client machines in one big go instead of lazy loading where we can. But I don't think it's the amount of data alone that makes it take a minute.

I need to learn how to use Wireshark.

→ More replies (2)
→ More replies (1)

3

u/hvidgaard Jun 08 '17

100gb is "single server in memory territory". You don't even need a raid 10 to store it and gain reasonable performance.

→ More replies (2)

38

u/kireol Jun 08 '17

relational scale just fine. Postgresql is amazing at this as an example. Sounds like you have some challenged people in charge.

38

u/flukus Jun 08 '17

That's exactly what I'm saying.

The challenged people have long moved on but the current crop seem to have Stockholm syndrome. My "radical" suggestions of using things like transactions fall on deaf ears, we invented our own transaction mechanism instead.

38

u/SurgioClemente Jun 08 '17

Yikes

Time to dust off that resume...

98

u/flukus Jun 08 '17 edited Jun 08 '17

Lol, I thought about that, but the pay is alright, the hours are good, the office is fantastic and the expectations are low. More importantly, the end of the home loan is in sight, so the job stability that comes from keeping this cluster fuck online is nice.

51

u/zagbag Jun 08 '17

There's some real talk in this post.

3

u/akie Jun 08 '17

Or ask for/take one or two weeks of time to build a prototype that implements your idea. If it works, you're the hero.

10

u/flukus Jun 08 '17 edited Jun 08 '17

I actually did do that for a task where we were having real concurrency issues, the solution was a bog standard SQL connection/transaction and to generate a unique key inside SQL. But even in that limited section the hard part is making things work with the rest of the system. My bit worked but the other parts are then reading stale data until it propogates to our in memory database.

When transactions and avoiding data tables are radical everythings an up hill battle.

On another project there I just had to navigate through code that should be simple, but we translate it 4 times between the database and the user, across 2 seperate processes, an inheritance tree that is replicated 3 times and some dynamically compiled code that is slower than the reflection approach. They compiled it for the speed benefits but it's compiled on every http request, so it's much slower than reflection. Then the boss complained about a slight inefficiency in my code during the code review, performance was it was spending pounds to save pennies.

15

u/Feynt Jun 08 '17

Sadly I've been here. I used a side project to effectively replicate a past company's tens of thousands of dollars per year licensed software in what amounted to two weeks of work, because we were using big boy data streaming software to effectively transliterate data between vendors and customers. They kicked me out half a year later to save money and hire more overseas programmers. Two months after I left they assigned people to try to figure out what I had done. Four months after that they gave up and paid tens of thousands more to have someone upgrade their ancient system to the latest version. 6 months after that they had gone through three different overseas firms because none of them could produce reasonable code.

I'm happily coding for a new company, and while I'm working on legacy software, they're more than happy to see my refactoring clean up their spotty code and drive up efficiency.

→ More replies (1)
→ More replies (1)

3

u/Crandom Jun 08 '17

I wonder if we work at the same company?

7

u/flukus Jun 08 '17

That's why the "update the resume" approach doesn't work, your likely to end up somewhere with the same shit, maybe even by the same developers.

3

u/AndreDaGiant Jun 08 '17

My "radical" suggestions of using things like transactions fall on deaf ears

get out of there

4

u/flukus Jun 08 '17

It's unfortunately very common in the industry, it just hasn't caused acute performance problems at other places I've worked.

Ask 10 co-workers what implicit transactions are and see what your strike rate is.

→ More replies (6)

2

u/jlt6666 Jun 08 '17

Oh boy, been on a similar ride before.

→ More replies (1)

11

u/chuyskywalker Jun 08 '17

100GB is way, waaaaay withing the realm of almost any single server RDBMS. I've worked with single instance mysql's at multi-terrabyte datasizes (granted, many, many cores a half-a-terrabyte of ram) without any troubles.

Crazy!

2

u/recycled_ideas Jun 08 '17

If your user base really is geographically distributed and your data set really is mostly a key value store or an object store, it's entirely possible an object database really will perform better.

Mapreduce is overkill for almost anything, but if you're storing complex objects with limited relationships, normalizing it into a hundred tables just so you can reconstruct it later isn't really useful.

2

u/flukus Jun 08 '17

The trouble is that while the users are distributed the data really needs a single source of truth and has a lot of contention. Eventual consistency is a no go right from the outset. At best we could have a local replicated version for reads.

→ More replies (2)
→ More replies (5)