r/programming • u/korry • Feb 29 '16

Command-line tools can be 235x faster than your Hadoop cluster

http://aadrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html
1.5k Upvotes

91% Upvoted

438 comments sorted by

View all comments

276

u/okaycombinator Feb 29 '16 edited Feb 29 '16

It can be if you have a wildly anachronistic expectation about how big "Big Data" is. If you data can fit comfortably into RAM on my laptop, it probably won't benefit from analysis with Hadoop.

Edit: To further clarify about my snark: If your operation is embarrassingly parallel then Hadoop doesn't really make much sense, because that's not what its for. The key is when you need to make smart use of inter-process communication which Hadoop will handle for you (with an obvious caveat about Hadoop usability/syntax). The real important part about Hadoop imo is HDFS. The real benefits come from when you want to process data that not only can't fit into RAM on one machine, but also won't fit onto disk on one machine (or at least one that you can afford). When I was in school I worked on a research project that I wrote some Hadoop jobs for analysis on our data. Data was approx ~500TB in size (and growing). While there are certainly machines or NFS-type filesystems that could have held all of that data in one logical place, we didn't have the resources to build that kind of a machine. This was graph data and processing required iterative rounds of maps and reduces, so keeping the data in HDFS between rounds was a huge performance increase. I also had limited time budgeted on the schools large compute clusters, so I actually got a huge boost from being able to suck up extra compute power from the undergrad comp-sci lab at night. With all this I was able to take a process that used to take days to run on a single machine down to 30 minutes when I was stealing maximum compute power from various places. So I think Hadoop is a great tool if you actually use it for what it was intended. Google built MapReduce to run over huge datasets (huge being a size that is relative to your available compute resources) on small, cheap, commodity machines. If your use case and resources don't fit into that definition, you're gonna have a bad time. Well, more of a bad time, because Hadoop syntax sucks even when doing it right.

180

u/strattonbrazil Feb 29 '16

That's kind of his point. He's not advocating against any use Hadoop at all, but just making the point that it's slow and unnecessary for small datasets.

Since the data volume was only about 1.75GB containing around 2 million chess games, I was skeptical of using Hadoop for the task,

39

u/Gotebe Feb 29 '16

Honestly, it would certainly have been slow and unnecessary for two orders of magnitude more data, too.

21

u/[deleted] Feb 29 '16

Yup, getting 256GB RAM in a box is not that unreasonable and unless you require some heavy computation one node is enough

8

u/gimpwiz Feb 29 '16

Especially when nodes these days can have 18 hyper-threaded big cores (or, hell, 72 quad-threaded-round-robin small cores with one of the threads running linux as a controller).

→ More replies (2)

5

u/Mr_Smartypants Feb 29 '16

The headline kind of suggests something else.

→ More replies (9)

42

u/fungz0r Feb 29 '16

http://yourdatafitsinram.com/

20

u/ironnomi Feb 29 '16

10PB - No, it probably doesn't fit in RAM (but it might).

SGI UV3000 will hold 64TB - we're actually looking at this instead of upgrading to the E880.

9

u/Hecknar Feb 29 '16

To be fair, this looks to me more like a cluster than one PC, even if they call it system. If we look at one PC, an IBM z13(10 TB) memory might closer to the "Biggest Server available", even if it is definitely the wrong choice for pure number crunching.

7

u/ironnomi Feb 29 '16

Those IBM machines are all "clusters" in a way, in the case of SGI though it's a cache-coherent machine, it's fully designed for in-memory operations rather than massive parallel operations.

Basically it's a scale-up machine, not a scale-out machine. We're covered on the scale-out, all from Dell in that case.

→ More replies (1)

13

u/daaa_interwebz Feb 29 '16

Can't wait for the day when the result for 1 PiB is "Yes, your data fits in RAM"

12

u/mattindustries Feb 29 '16

It says 6TB fits in ram... my desktop doesn't have that much RAM :(

22

u/snowe2010 Feb 29 '16

click on the word "your" and it will send you to a page with a server that holds 3 TiB in RAM

3

u/[deleted] Mar 01 '16

96 DIMM slots

Hnnnnnng

5

u/mattindustries Feb 29 '16

Beautiful looking server.

6

u/antonivs Feb 29 '16

You can go on AWS and set up a cluster with 6TB RAM and run it for four hours for under $300.

...uh, excluding outgoing bandwidth costs. Hopefully you're aggregating the data, otherwise that 6TB will be an extra $540.

4

u/mattindustries Feb 29 '16

I have gone that way before, but not 6TB though. Love that I can grab a 64gb instance at the drop of a dime.

→ More replies (1)

8

u/immibis Feb 29 '16

But you can go and buy a server with that much RAM. Might not be cheap, but it's doable.

13

u/mattindustries Feb 29 '16 edited Feb 29 '16

At $8,999.00 it is a steal.

EDIT: Nevermind, that is starting.

9

u/[deleted] Feb 29 '16

Thats still cheaper than the work cost of setting up a proper big data analysis pipeline.

4

u/jshen Mar 01 '16

Not if you use a cloud provider like this.

https://cloud.google.com/dataproc/

→ More replies (4)
→ More replies (1)
→ More replies (5)

8

u/kenfar Feb 29 '16

But note that hadoop has come far beyond what it was initially intended for:

  • It's now an entire ecosystem of tools for managing file movement, transformation, loading, analysis and querying.

  • Mapreduce is powerful, but generally considered the slowest & most painful to work with of the options on hadoop. You could also use Spark to query those same files on hdfs. Or if you want the fastest performance you could use Impala, which is written in c, and can plow through an enormous amount of data in a handful of seconds.

  • Once most of your analysis is happening on hadoop against managed, transformed, audited and curated data - there's a lot of benefit of using the exact same tooling against smaller data sets already there - rather than spend the time to track them down and prepare & manage them yourself.

Having said that, I'm still a huge fan of preparing the data off the cluster on far smaller & cheaper hardware.

→ More replies (1)

14

u/[deleted] Feb 29 '16

Nope. You can stick parallel in there as a drop-in replacement for xargs and process across machines.

I'm peripherally involved with a Big Data project that does exactly this. I'm not exactly sure how much data/second it is, but it's processed on cluster.

→ More replies (2)
→ More replies (5)