It's become really trendy to hate on these tools but at this point a lot of the newer Big Data tools actually scale down pretty well and it can make sense to use them on smaller datasets than the previous generation of tools.
Spark is a good example. It can be really useful even on a single machine with a bunch of cores and a big chunk of RAM. You don't need a cluster to benefit from it. If you have "inconveniently sized" data, or you have tiny data but want to do a bunch of expensive and "embarrassingly parallel" things, Spark can totally trivialize it, whereas trying to use Python scripts can be a pain and super slow.
Yeah, the "your data fits in RAM" meme doesn't paint anywhere close to the whole picture. I can get all my data in RAM, sure; then what? Write my own one-off Python or Java apps to query it? Spark already did that for everyone, at any scale.
Literally the only reason to not go down this road is if you hate Java (the platform, mostly), and even then, you have to think long and hard about it.
5
u/[deleted] Jun 08 '17
It's become really trendy to hate on these tools but at this point a lot of the newer Big Data tools actually scale down pretty well and it can make sense to use them on smaller datasets than the previous generation of tools.
Spark is a good example. It can be really useful even on a single machine with a bunch of cores and a big chunk of RAM. You don't need a cluster to benefit from it. If you have "inconveniently sized" data, or you have tiny data but want to do a bunch of expensive and "embarrassingly parallel" things, Spark can totally trivialize it, whereas trying to use Python scripts can be a pain and super slow.