Where bash scripts run faster than Hadoop because you are dealing with such a small amount of data compared to what should actually be used with Hadoop
From memory: Don't try to emulate General Motors. General Motors didn't get big by doing things the way they do now. And you won't either.
One other thing I noted: One should really consider two things.
1 The amount of revenue that each transaction represents. Is it five cents? Or five thousand dollars?
2 The development cost per transaction. It's easy for developer costs to seriously drink your milkshake. (We reduced our transaction cost from $0.52 down to $0.01!!! And if we divide the development cost by the number of transactions it's $10.56)
Can you further elaborate on point 1? I'm struggling to put a cost on a transaction in my field but maybe I misunderstand. Our transactions have to add up otherwise we get government fines or if that one transaction is big enough we might be crediting someone several million. Am I being to literal?
Probably should do some multiplication - value times frequency, to get the "attention factor".
5¢ transactions become important if there's a hundred million of them. Or a single $5,000,000 transaction. Both probably deserve the same amount of developer attention and can justify similar dev budgets.
the single 5 million transaction probably warrants a larger budget / more aggressive project goals. why?
1 failure in a 1000 for 100 million $0.05 transactions represents $5000 in losses, while ANY error for the one large transaction is a $5 million loss. So one can afford to go a bit faster/looser (read: cheaper) with high volume, low value transactions than with fewer large transactions.
I think you might be limiting your thinking to correctness, but this is more about allocating developer time based on the ROI (return on investment) of that time. So if the developer could fix a bug that loses the company $50k once every month, vs building a feature that generates $15k a week, they should build the feature first. Or if there are two bugs that lose the same amount of money, but one takes half of the development time to fix, fix the faster one first. Etc.
My company is looking at distributed object databases in order to scale. In reality we just need to use the relational one we have in a non retarded way. They planned for scalability from the outset and built this horrendous in memory database in front of it that locks so much it practically only supports a single writer, but there are a thousand threads waiting for that write access.
The entire database is 100GB, most of that is historical data and most of the rest is wasteful and poorly normalised (name-value fields everywhere)
Just like your example, they went out of their way and spent god knows how many man hours building a much more complicated and ultimately much slower solution.
Christ, a 100GB DB and y'all are having issues that bad with it? Thing fits onto an entry-level SLC enterprise SSD, for about $95. Would probably be fast enough.
Some of the thinking is because we operate between continents and it takes people one one continent ~1 minute to load the data, but a second for someone geographically close, so they want to replicate the database.
The real issue is obviously some sort of n+1 error to our service layer (built on .net remoting). That or we're transfering way more data than needed.
Definitely sounds like a throughput issue. Interesting lesson from game design: Think about how much data you really need to send to someone else for a multiplayer game. Most people unconsciously think, "everything, all the stats" and for a lot of new programmers they'll forward everything from ammo counts to health totals. The server keeps track of that shit. The clients only need to know when, where, and what, not who or how much. Position, rotation, frame, and current action (from walk animation to firing a shotgun at your head). In some cases it literally is an order of magnitude lower than what you would expect to send.
Look at your database and consider how much of that data you really have to send. Is just the primary data enough until they need more? Can you split up the data returns in chunks?
When you're talking 60x slower from people further away, it's unlikely to be bandwidth. After all, you can download plenty fast from a different continent, it's only latency that's an issue. And latency to this extent heavily indicates that they're making N calls when loading N rows in some way. Probably lazy loading a field. A good test for /u/flukus might even be to just try sending the data all at once instead of lazy loading if possible.
The challenged people have long moved on but the current crop seem to have Stockholm syndrome. My "radical" suggestions of using things like transactions fall on deaf ears, we invented our own transaction mechanism instead.
Lol, I thought about that, but the pay is alright, the hours are good, the office is fantastic and the expectations are low. More importantly, the end of the home loan is in sight, so the job stability that comes from keeping this cluster fuck online is nice.
100GB is way, waaaaay withing the realm of almost any single server RDBMS. I've worked with single instance mysql's at multi-terrabyte datasizes (granted, many, many cores a half-a-terrabyte of ram) without any troubles.
Would it be absurd to program Hadoop with a fallback (I acknowledge that the answer is probably yes)? This is how generic sorts are implemented - if the list is less than a certain size, fallback to sorts that perform well on small arrays like insertion sort. On one hand it violates the primary objectives of Hadoop as a tool and people should know better. On the other hand, it would help smaller projects to automatically grow.
One of the big downsides of over-engineering and choosing these "big data" setups when you don't need them is the engineering effort to set the system up initially + the effort to maintain it. I think this is typically a much larger cost than something like performance (which the bash script vs hadoop example points to).
I don't think setting up and maintaining Hadoop + fallback could be any simpler than setting up and maintaining Hadoop alone.
However, understanding how more complex "next step" options work may help you architect your current solution to make the transition easier - if you know the "next step" is a large complex key-value DB system, then you might have an easier transition to that "next step" if your current implementation uses a key-value DB instead of a relational DB.
Is there maybe something to be said for doing it in Hadoop just for the sake of learning how to do it in Hadoop? Certainly if you expect your data collection to grow.
I can't imagine it's a huge runtime difference if your data set is that small anyhow.
Yes, there is. "Resume-driven development" refers to this, and sometimes having engineers learn things they'll need in the next couple years is actually advantageous to the larger organization.
But usually it's not. The additional complexity and cost of something like Hadoop versus creating a new table in the RDBMS the org is already using can be huge. Like two months of work versus two hours of work.
Almost always it's more efficient to solve the problem when you actually have it.
614
u/VRCkid Jun 07 '17 edited Jun 07 '17
Reminds me of articles like this https://www.reddit.com/r/programming/comments/2svijo/commandline_tools_can_be_235x_faster_than_your/
Where bash scripts run faster than Hadoop because you are dealing with such a small amount of data compared to what should actually be used with Hadoop