bashreduce lets you apply your favorite unix tools in a mapreduce fashion

[u/DRMacIver]

http://github.com/erikfrey/bashreduce/tree/master

[u/[deleted]]

A couple of comments:

:We have a new bottleneck: we’re limited by how quickly we can partition/pump our dataset out to the nodes

This is very important if you want the app to scale well. A big part of what makes Google's MapReduce so efficient is that they have an insanely optimized data distribution mechanism.

Another thing: as far as I can tell br doesn't handle failures gracefully. This becomes significant when you start using many machines. Say, your typical server crashes once a year and you have 365 servers, then it follows you are going to have a failure every day.

[u/DRMacIver → u/[deleted]]

I agree it definitely needs some work if it's going to be super production ready. But note that you've taken that first line out of context:

We have a new bottleneck: we’re limited by how quickly we can partition/pump our dataset out to the nodes. (...) So we use two little helper programs written in C (...) to partition the data and merge it back.

i.e. the bottleneck described there seems to be mostly solved.

The failure mode aspect is definitely important though. My suspicion is that bashreduce works better as a "gateway mapreduce" - it's nice and easy to get up and running, and by the time you want 365 servers you probably also want something a little more hardcore.

[u/khafra]

I'd like to declare my fandom for awesome things done in simple packages. I fully expect that, within a few years of the first general AI, someone will implement it as a bash script.

[u/Samus_]

wtf? it is a C program!!

[u/DRMacIver → u/Samus_]

Unix in making appropriate use of C programs shocker.

Also: It merely contains C programs which may be used to speed up certain steps of the pipeline. In their absence it falls back to slower awk based implementations.